Adding an OCI Data Sink

You must have Root scope to use this feature.

You can add an OCI Data Sink from the System | Data Processor | Data Sinks page using the instructions in this topic.

Adding an OCI Data Sink requires you to retrieve some information from your account in Oracle Cloud Infrastructure and then supply that information in the Data Sinks interface in Stellar Cyber.  The easiest way to do this is to open two browser windows with one connected to Stellar Cyber and the other to the OCI Console. That way, you can just copy from OCI and paste into Stellar Cyber. The example below shows how to do this.

Use our example as a guideline, as you might be using a different software version.

Summary of Required Information from OCI

You will need the following items from OCI to add the data sink in Stellar Cyber:

  • RSA key pair (PEM format)

  • Fingerprint of the public key

  • OCID for both the Tenancy and the User

  • Identifying information for the bucket

You can refer to the following article in the OCI documentation for descriptions of how to generate and retrieve all of this information. The procedure below also describes how to gather the necessary details.

Open the Browser Windows

Start by opening two browser windows, one connected to Stellar Cyber's Data Sink interface and the other to Oracle Cloud Console:

Start to create the OCI Data Sink in Stellar Cyber:

  1. Log in to Stellar Cyber and navigate to System | Data Processor | Data Sinks. The Data Sink list appears.

  2. Click Create. The Setup Data Sink screen appears.

  3. Enter the Name of your new Data Sink. This field does not support multibyte characters.

  4. Choose OCIfor the Type.

    Additional fields appear in the Setup Data Sink screen:

    Now, we'll open a second browser window so we can paste the necessary information for the OCI data sink in Stellar Cyber directly from the Oracle Cloud Console.

Log in to the Oracle Cloud Console:

  1. Open a second browser window.

  2. Log in to Oracle Cloud Console at https://cloud.oracle.com/.

Add Information from OCI to Stellar Cyber

Now that we have windows open for both the Data Sink in Stellar Cyber and our account in Oracle Cloud Console, we can just paste the necessary information from OCI to Stellar. Use the following procedure:

  1. Click the main menu icon at the top left of the Oracle Cloud Console.

  2. Navigate to Governance & Administration | Account Management | Tenancy details and use the Copy link to copy the OCID shown in the Tenancy information tab, as shown below.

  3. Paste the OCID of the tenancy into the OCI Tenant ID field in Stellar Cyber:

  4. Next, you need the OCID of the user account for the sink:

    • If you're signed in as the user whose OCID you need, you can get it from the Profile | My Profile view.

    • If you're admin getting the OCID for another user, you can get it from Identity & Security | Identity | Users.

  5. Paste the OCID of the user into the User ID field in Stellar Cyber:

  6. Click the main menu icon at the top left of the Oracle Cloud Console and navigate to Storage | Object Storage & Archive Storage | Buckets.

  7. If you haven't already created the bucket you want to use for the data sink, you can use the Create Bucket button to create it now.

  8. Click the entry for the bucket you want to use for the data sink in the list. We're using a bucket called oci-data-sink in this example.

  9. Copy the bucket's namespace.

  10. Paste the bucket's namespace into the Namespace field in Stellar Cyber.

  11. Return to Oracle Cloud Console and copy the bucket's name.

  12. Paste the bucket's name into the Bucket Name field in Stellar Cyber.

  13. Take note of the region in which you created the bucket in OCI (it's shown at the top right of the Oracle Cloud Console. Then, use the Region dropdown in Stellar Cyber to select the matching region. We're using the US_PHOENIX_1 region in our example.

  14. Next, you'll need both an API Private Key and its associated Fingerprint for the user account used to sink data to this bucket. Start by using these instructions on how to generate an API Signing Key in OCI.

  15. Download the private key in PEM format and upload it to Stellar Cyber in the API Private Key field.

  16. Use these instructions to get the fingerprint for the key you just uploaded.

  17. Paste the fingerprint into the Fingerprint field in Stellar Cyber.

  18. Select the types of data to send to the Data Sink by toggling the following checkboxes:

    • Raw Data – Raw data received from sensors, log analysis, and connectors after normalization and enrichment has occurred and before the data is stored in the Data Lake.

    • Alerts – Security anomalies identified by Stellar Cyber using machine learning and third-party threat-intelligence feeds, reported in the Alerts interface, and stored in the aella-ser-* index.

    • Assets – MAC addresses, IP addresses, and routers identified by Stellar Cyber based on network traffic, log analysis, and imported asset feeds and stored in the aella-assets-* index.

    • Users – Users identified by Stellar Cyber based on network traffic and log analysis and stored in the aella-users-* index.

    Alerts, assets, and users are also known as derived data because Stellar Cyber extrapolates them from raw data.

  19. Click Next.

    At this point, Stellar Cyber attempts to reach the bucket for the data sink using the settings you have specified in the previous steps:

    • If Stellar Cyber is not able to reach the bucket, an error message appears and you must check and correct the settings as necessary.

    • If the settings are validated successfully, the Advanced (Optional) page appears, as illustrated below:

    Stellar Cyber can detect and alert you to the following errors in your data sink configuration:

    • Missing required parameters

    • Failed to connect to bucket (for example, due to an incorrect region or endpoint URL)

    • Incorrect access key

    • Incorrect secret key

  20. Specify whether to partition records into files based on their write_time (the default) or timestamp.

    Every interflow record includes both of these fields:

    • write_time indicates the time at which the Interflow record was actually created.

    • timestamp indicates the time at which the action documented by the Interflow record took place (for example, the start of a session, the time of an update, and so on).

    When files are written to the Data Sink they are stored at a path like the following, with separate files for each minute:

    In this example, we see the path for November 9, 2021 at 00:23. The records appearing in this file would be different depending on the setting of the Partition time by setting as follows:

    • If write_time is enabled, then all records stored under this path would have a write_time value falling into the minute of UTC 2021.11.09 - 00:23.

    • If timestamp is enabled, then all records stored under this path would have a timestamp value falling into the minute of UTC 2021.11.09 - 00:23.

    In most cases, you will want to use the default of write_time. It tends to result in a more cost-efficient use of resources and is also compatible with future use cases of data backups and cold storage using a data sink as a target.

  21. Enable the Compression option to specify that records be written to the Data Sink in compressed (gzip) format.

    For most use cases, Stellar Cyber recommends enabling the compression option to save on storage costs. Compression results in file sizes roughly 1/10th the size of uncompressed files.

  22. You can use the Retrieve starting from field to specify a date and time from which Stellar Cyber should attempt to write alert, asset, and user records to a newly created Data Sink. You can click in the field to use a handy calendar to set the time/date

    Note the following:

    • If you do not set this option, Stellar Cyber simply writes data from the time at which the sink is created.

    • This option only affects alert, asset, and user records. Raw data is written from the time at which the sink is created regardless of the time/date specified here.

    • If you set a time/date earlier than available data, Stellar Cyber silently skips the time without any available records.

  23. Use the Time Format, Batch Window and Batch Size fields to specify how often data is written to the sink. The frequency with which data is written to the sink also affects the size of the files – the longer you wait to send files, the larger they will be.

    • Start by setting the Time Format option. This specifies the units for the Batch Window and gives you access to different granularities for the files written to the Data Sink. You can specify Seconds, Minutes, or Hours (the default).

    • The Batch Window specifies the maximum amount of time that can elapse before data is written to the Data Sink. By default, this is set to 6 hours. The values available depend on the units selected for Time Format:

      • If Time Format is set to Hours, you can select from 1, 4, 6, 12, or 24 hours. The 24 hour setting is the maximum granularity; after that, files start to become too large for efficient storage.

      • If Time Format is set to Minutes, you can select from 5, 10, 20, or 30 minutes. As you can see, each of these values divides evenly into an hour, giving you a precise idea of the number of files stored to the sink per hour by each worker. For example, with a setting of 30 minutes, there will be two files per hour for each worker in the sink.

      • If Time Format is set to Seconds, you can specify any value up to 60. This was the granularity supported in Data Sink version prior to 4.3.7.

      The Batch Window helps you balance granularity with costs when storing data in external cloud storage, where vendors often charge you by the API call.
      For example, writing to a data sink with a fine granularity expressed in seconds may result in excessive files and folders written to your external cloud storage and require you to incur the costs of a more expensive storage tier. By using a coarser granularity, you can ensure that the data files written to the cloud are larger and written less frequently. For example, the default granularity of six hours typically ensures that data files are larger than 128 KB, allowing you to take advantage of, for example, the less costly AWS S3 Intelligent Storage Tier. Contact Customer Success if you are interested in moving your Data Sink to a different storage tier.

      Note that after upgrading to 4.3.7, existing AWS and OCI data sinks with a Batch Window greater than 60 seconds are converted to the nearest available selection expressed in minutes or hours.

    • The Batch Size specifies the maximum number of records that can accumulate before they are sent to the Data Sink. You can specify either 0 (disabled) or a number of records between 100 and 10,000.

      The Batch Size option is only available when the Batch Window is set to a per-second interval and Size Format is set to Records. This is the batching implementation used in versions prior to 4.3.7. For all other Batch Windows, the Batch Size is set to Unlimited and cannot be changed.

      If you use the per-second interval together with the Batch Size option, Stellar Cyber batches data to the Data Sink depending on whichever of these parameters is reached first. Consider a Data Sink with a Batch Window of 30 seconds and a Batch Size of 300 records:

      • If at the end of the Batch Window of 30 seconds, Stellar Cyber has 125 records, it sends them to the data sink. The Batch Window was reached before the Batch Size.

      • If at the end of 10 seconds, Stellar Cyber has 300 records, it send the 300 records to the Data Sink. The Batch Size was reached before the Batch Window.

  24. You can use the Filter options to Exclude or Include specific Message Classes for the Data Sink. By default, Filter is set to None. If you check either Exclude or Include, an additional Message Class field appears where you can specify the message classes to use as part of the filter. For example:

    You can find the available message classes to use as filters by searching your Interflow in the Investigate | Threat Hunting | Interflow Search page. Search for the msg_class field in any index to see the prominent message classes in your data.

  25. Click Next to review the Data Sink configuration. Use the Back button to correct any errors you notice. When you are satisfied with the sink's configuration, click Submit to add it to the DP.

  26. Click Submit.

The new Data Sink is added to the list.