The IANN Monitor Anomalies system follows a structured pipeline to process metrics, detect anomalies, and present results. This pipeline is fully automated and runs at scheduled intervals defined by the user.

4.1 Threshold Fetching

·       Initiation: Triggered from the UI when users preview or save a threshold.

·       Process:

1.    The selected threshold value, along with the datapoint name, index, and environment, is written into the index_UI index in Elasticsearch.

2.    These thresholds act as references for model training and anomaly scoring. 

·       Record Example:

json

·       Benefit: Decouples threshold configuration from code changes, making the system user-friendly and flexible.

4.2 Model Training

·       Trigger Conditions:

1.    A new threshold has been set.

2.    The corresponding model does not exist yet.

·       Actions Performed:

1.    Compares the current threshold with historical data in threshold_tracker.csv.

2.    If changes are detected, retrain the Isolation Forest model for that datapoint.

3.    Saves two artifacts:

§  datapoint_indexname_anomaly_model.pkl: Trained anomaly detection model.

§  datapoint_indexname_baseline_stats.pkl: Baseline stats such as mean, median, standard deviation.

·       Automation: Managed via model_management.py and train_and_save_model.py.

4.3 Anomaly Detection

·       Input: Live metric data is pulled from tyson_* indices in Elasticsearch.

·       Workflow:

1.    Loads the relevant .pkl model for the datapoint.

2.    Scores each incoming data point to determine whether it’s anomalous.

3.    Calculations:

§  Mean and median deviation.

§  Deviation label (e.g., close, moderate, far).

4.    Enhances the result using llm.py, which adds a natural language explanation for the anomaly.

Anomaly detected: Heap usage increased significantly from the median baseline. Likely memory leak.

4.4 Output

·       Storage:

1.    Locally: anomalies.csv

2.    Elasticsearch: Output index defined in config[‘elasticsearch’][‘index_name’]

·       Data Format:
 Each anomaly record includes:

1.    Timestamp

2.    Metric value

3.    Threshold

4.    Deviation score and label

5.    Explanation

6.    Source index and environment 

·       Purpose:

1.    Enables dashboard rendering.

2.    Facilitates long-term tracking and alerting.

3.    Allows integration with downstream notification systems. 

Index Structure

The IANN system utilizes Elasticsearch indices for storing and retrieving different types of data involved in the anomaly detection lifecycle. Each index serves a dedicated purpose, ensuring organized and efficient access to time-series metrics, thresholds, and anomaly results.

1. Raw Metric Data

·       Index Pattern: client_*

·       Purpose: Serves as the primary input source for the system, containing historical and real-time metrics collected from client systems or applications.

·       Data Contents:

1.    Timestamps

2.    Metric values (e.g., CPU, memory, heap usage)

3.    Metadata (e.g., environment, server ID, application name)

·       Usage: This index is queried during anomaly detection to fetch the latest datapoints to be analyzed.

2. Threshold Configurations

·       Index Name: index_UI

·       Purpose: Stores threshold settings submitted by users via the web UI.

·       Data Contents:

1.    dataPoint: Name of the metric (e.g., memory_utilization)

2.    indexName: Source index to which the threshold applies

3.    threshold: Numerical threshold value

4.    environment: Context such as dev, uat, or prod

·       Usage: Used during model training to determine whether to retrain and which threshold to apply for scoring anomalies.

3. Anomaly Output

·       Index Reference: config[‘elasticsearch’][‘index_name’]

·       Purpose: Acts as the final output repository for all detected anomalies.

·       Data Contents:

1.    Anomaly timestamp and metric value

2.    Threshold and deviation scores

3.    Severity label (close, moderate, far)

4.    Natural language explanation

5.    Contextual metadata (environment, indexName, dataPoint)

·       Usage: Queried by the UI dashboard for visualization, filtering, and alerting purposes.

Note: These indices can be managed or scaled independently. Data retention policies, backups, and index lifecycle management (ILM) should be considered for long-term monitoring in production environments.

Scheduling & Execution

The scheduling and execution mechanism in the IANN Monitor Anomalies system is designed for simplicity, reliability, and low overhead. It ensures that anomaly detection runs consistently at user-defined intervals, without requiring external tools like cron jobs or third-party orchestrators.

1. Scheduling Logic

·       Technology Used: Python’s built-in sched module.

·       Why sched?

1.    Lightweight and Python-native.

2.    Avoids the need for system-level cron services or external schedulers like Airflow.

3.    Simple to configure and maintain in self-contained deployments (e.g., via MobaXterm).

2. Configuration

·       The execution interval is defined in the config.ini file under the scheduler section.

·       Example configuration:

[scheduler]
scheduler_time = 30

·       The interval determines how often the anomaly detection process runs (e.g., every 30 minutes).

3. Execution Instructions

·       Once initiated, the script:

1.    Loads all necessary configurations and thresholds.

2.    Triggers model training if needed.

3.    Performs anomaly detection based on the defined schedule.

4.    Saves output to CSV and Elasticsearch.

4. Runtime Behaviour

·       All executions occur within the same terminal session (e.g., MobaXterm).

·       If the terminal or session is closed, the scheduled job stops unless re-launched.

·       Logs and progress indicators are printed to the console for monitoring.