System Status

There are two ways to see the System Status:

  • From Dashboards | PREDEFINED | System Status

  • By selecting in the upper right corner of the Stellar Cyber UI

The provides a quick indication of the health of the system:

  • indicates that all components are healthy

  • indicates that at least one component is yellow and should be monitored

  • indicates that at least one component is red and needs immediate attention

This page gives you an overview of the health of the system. You cannot take any actions on this page.

Sensor Health

The nine sensor-related components are:

  • Sensor Health – The count of all sensors, indicating which are healthy, need attention, or are disconnected.

  • Sensor Authorization – The count of sensors and how many need authorization

  • Sensor Need Upgrade – The count of sensors and how many require a software upgrade.

  • Average Reduction Ratio—the amount of data in bytes that the sensors ingest divided by the amount they transmit in bytes. This figure is an average; many sensors will have much higher reduction ratios individually.

  • Average Packet Bytes Input – The average amount of data seen by sensors (total bytes received divided by the count of sensors).

  • Average Metadata Output – The average amount of data sent by sensors (total bytes transmitted, divided by the count of sensors).

  • Input/Output Rate – The input and output rate of sensors in bits per second.

  • Sensor Activity Status – The most recent activity status for individual sensors.

  • Input/Output Bytes – The total input and output byte count for each sensor for the given.

For operations related to sensors, see the Sensor Overview page.

Data Analyzer Health

If the chart is green, the Data Analyzer (DA) is healthy.

If the chart is yellow, the DA is either misconfigured or stressed, with one or more of the following conditions:

  • A node is using more than 91% of its memory.

  • A node is using more than 79% of its CPU.

  • Batch processing time is taking more than 10 seconds.

If the chart is red contact Stellar Cyber support. The DA cannot process data, is completely overloaded, or is disconnected from the Data Lake (DL), and is in one or more of the following conditions:

  • A node is using more than 97% of its available memory.

  • A node is using more than 95% of its CPU.

  • Batch processing times have repeatedly exceeded 10 seconds for a period of over 5 minutes.

  • There haven't been any connections between the DA and DL for more than 5 minutes.

  • A node is using 75% of its available disk space.

Data Lake Health

If the chart is green, the Data Lake (DL) is healthy.

If the chart is yellow, the DL is either misconfigured or stressed, with one or more of the following conditions:

  • A node is using more than 91% of its available memory.

  • A node is using more than 79% of its CPU.

  • An Elasticsearch node is using more than 95% of its available memory.

  • The number of bulk rejections has increased over a period of 15 minutes.

  • One or more indices are using more than 90% of their allowed number of mapped fields.

  • 80% of available disk space is in use.

  • Data cannot be written to an index.

  • Shards are slightly to heavily imbalanced in their distribution across DL worker nodes.

  • An unexpected index is found in tomorrow's data, including a date format instead of just a timestamp.

  • No aliases were created for a specified index type in tomorrow's data.

If the chart is red contact Stellar Cyber support. The DL cannot process data, is losing data, or has a component completely offline, with one or more of the following:

  • A node is using more than 96% of its available memory.

  • A node is using more than 94% of its CPU.

  • An Elasticsearch node is out of memory.

  • At least one index mapping has reached the field limit.

  • 90% of available disk space is in use.

  • Current data cannot be written to an index.

  • Elasticsearch is unresponsive.

  • Cold storage is offline.

  • The system partition has exceeded 75% capacity. This applies when the system and data partitions are separate.

  • Shards are heavily imbalanced across DL worker nodes.

  • An unexpected index appears in today's data, using a date format instead of a timestamp.

  • Aliases were not created for a specified index type in today's data.

Contact Stellar Cyber support for assistance in addressing shard/index imbalances across DL worker nodes. You may be able to enable automatic shard rebalancing from the CLI to address the issue (set mode shard_balance enable/disable). Note that if shard rebalancing is enabled, the DL appears in yellow in the System Health page.

Additional Information in Data Lake Health

The Data Lake Health panel also reports any of the following conditions detected on the data lake:

  • Lost nodes

  • Excessive shards per node

  • Duplicated aliases in specific indices

  • Excessive field mappings in specific indices

These situations are only reported when detected and appear directly below the Data Lake heading.