Operator Handbook — 4 Components (Frontend + Backend + ML)

1) Architecture & Data Flow

The system is a React dashboard talking to a Flask ML API over /api/ml/. Models train, persist to .pkl, and generate graphs under reports/.

1.1) Project File Structure

Physical layout of the monorepo mapping code to components.

/var/www/forensic-portal/
├── backend/
│   ├── api/ml_api.py              # Main Flask entry point (All Routes)
│   ├── data/cicd_state.json       # JSON Database for Evidence & Logs
│   ├── ecosystem.config.js        # PM2 process config
│   └── ml_models/                 # Logic per domain
│       ├── anomaly_detection/     # Commit patterns, dependency checks, pipeline tampering
│       ├── cicd_agents/           # HDFS/Hadoop log analysis models
│       ├── integrity_verification/# Malware classifier, Ed25519, Merkle, hashing
│       └── reporting_portal/      # Attack/Breach classifiers & report generation
└── src/
    ├── api/mlApi.js               # Frontend-Backend bridge (fetch wrapper)
    ├── App.js                     # Main React layout & Tab Controller
    └── components/                # UI Views matching backend domains
        ├── AnomalyDetection/      # Graphs, Timelines, Training UI
        ├── ForensicReadyCICDAgents/ # Agent Status, Evidence Collection UI
        ├── ForensicReportingPortal/ # Final Reports, Chain of Custody UI
        └── IntegrityVerification/ # Crypto tools & Malware Verification UI
            

2) Flow Charts

These diagrams are “shapes” (SVG) so they render everywhere (no external assets). Use them as a mental model for how every button maps to API calls, persisted artifacts, and UI rendering.

Overall system flow

Model lifecycle (generic train → infer → reports)

3) Dashboard Layout (Tabs)

The top-level tabs are implemented in src/App.js. Each tab renders a component:

The header status indicator (“Connected/Disconnected”) calls GET /api/ml/health.

GET /api/ml/health
Response:
{
  "success": true
}
                

4) Component 1 — Forensic-Ready CI/CD Agents

Frontend folder: src/components/ForensicReadyCICDAgents/

This tab is designed to: show connected CI/CD agents, collect evidence from datasets/models, let you set evidence capture configuration, and run two ML models (HDFS anomaly, Hadoop failure classification).

UI sections

Buttons and what they do

POST /api/ml/cicd-agents/collect
Body:
{
  "mode": "dataset"
}
Response:
{
  "success": true,
  "added": 2
}
                

GET /api/ml/cicd-agents/evidence?limit=200
Response:
{
  "success": true,
  "evidence": [
    {
      "timestamp": "2025-12-13T02:32:20Z",
      "type": "build-logs",
      "description": "Hadoop job failure predicted",
      "metadata": {
        "app_id": "application_...",
        "confidence": 80.32,
        "label": 1
      }
    }
  ]
}
                

GET /api/ml/cicd-agents/config
Response:
{
  "success": true,
  "config": {
    "capture_git_diffs": true,
    "capture_build_logs": true,
    "capture_env_vars": true,
    "capture_secrets_access": true,
    "capture_artifacts": true,
    "capture_pipeline_configs": true,
    "auto_collect": false,
    "collection_interval_seconds": 30,
    "max_evidence_size_mb": 100,
    "encrypt_evidence": false
  }
}

POST /api/ml/cicd-agents/config
Body:
{
  "config": {
    "auto_collect": true,
    "collection_interval_seconds": 30
  }
}
Response:
{
  "success": true,
  "config": { "...merged_config" : true }
}
                

How Training Works (Step-by-Step)

Data sources

Advanced / Internal Endpoints

5) Component 2 — Automated Forensic Anomaly Detection Engine

Frontend folder: src/components/AnomalyDetection/

Tabs

Buttons and what they do

POST /api/ml/anomaly-detection/infer
Body:
{
  "model": "commit_pattern",
  "limit": 500
}
Response:
{
  "success": true,
  "anomalies": [
    {
      "id": "commit_0",
      "category": "commit-patterns",
      "confidence": 94.15,
      "details": { "is_off_hours": 1, "files_changed": 0 }
    }
  ]
}
                

How Training Works (Step-by-Step)

Reports/graphs displayed

Training graphs are requested as images from the backend. Example: GET /api/ml/anomaly-detection/reports/commit_confusion_matrix.png

Data sources

Advanced / Internal Endpoints

6) Component 3 — Integrity Verification (SBOM / Hash / Sign / Merkle / Malware)

Frontend folder: src/components/IntegrityVerification/

Tabs

Buttons and what they do

POST /api/ml/integrity-verification/keys/ed25519
Response:
{
  "success": true,
  "private_key_pem": "-----BEGIN PRIVATE KEY----- ...",
  "public_key_pem": "-----BEGIN PUBLIC KEY----- ..."
}

POST /api/ml/integrity-verification/sign/ed25519
Body:
{
  "payload": "hello",
  "private_key_pem": "-----BEGIN PRIVATE KEY----- ..."
}
Response:
{
  "success": true,
  "signature_b64": "..."
}

POST /api/ml/integrity-verification/verify/ed25519
Body:
{
  "payload": "hello",
  "public_key_pem": "-----BEGIN PUBLIC KEY----- ...",
  "signature_b64": "..."
}
Response:
{
  "success": true,
  "valid": true
}
                

How Training Works (Step-by-Step)

Advanced / Internal Endpoints

7) Component 4 — Forensic Chain-of-Custody & Reporting Portal

Frontend folder: src/components/ForensicReportingPortal/

This component consumes real evidence from the backend (CI/CD evidence state + anomaly latest results), constructs a timeline, provides custody logs, verifies integrity proofs via the hash API, generates reports, and includes an ML Models sub-tab using the Reporting Portal datasets.

Tabs

Buttons and what they do

POST /api/ml/reporting-portal/infer-sample
Body:
{
  "model": "attack_type",
  "sample": {
    "Attack Name": "Phishing",
    "Description": "Credential harvesting email campaign targeting employees"
  }
}
Response:
{
  "success": true,
  "prediction": "Phishing",
  "confidence": 0.81
}
                

POST /api/ml/reporting-portal/infer-sample
Body:
{
  "model": "breach_type",
  "sample": {
    "Organization": "Example Corp",
    "Type_of_Breach": "Hacking",
    "Summary": "Unauthorized access and data exfiltration detected",
    "Records_Lost": 120000
  }
}
                

How Training Works (Step-by-Step)

Reporting Portal ML datasets

Advanced / Internal Endpoints

8) API Index (Endpoints)

All endpoints are served by Flask in backend/api/ml_api.py.

GET  /api/ml/health

GET  /api/ml/cicd-agents/status
GET  /api/ml/cicd-agents/evidence?limit=200
POST /api/ml/cicd-agents/collect
GET  /api/ml/cicd-agents/config
POST /api/ml/cicd-agents/config
POST /api/ml/cicd-agents/config/reset
POST /api/ml/cicd-agents/register

GET  /api/ml/cicd-agents/hdfs/status
POST /api/ml/cicd-agents/hdfs/train
POST /api/ml/cicd-agents/hdfs/infer-dataset
GET  /api/ml/cicd-agents/hdfs/latest
GET  /api/ml/cicd-agents/hdfs/reports/<filename>

GET  /api/ml/cicd-agents/hadoop/status
POST /api/ml/cicd-agents/hadoop/train
POST /api/ml/cicd-agents/hadoop/infer-dataset
GET  /api/ml/cicd-agents/hadoop/latest
GET  /api/ml/cicd-agents/hadoop/reports/<filename>

GET  /api/ml/anomaly-detection/status
POST /api/ml/anomaly-detection/train
POST /api/ml/anomaly-detection/infer
POST /api/ml/anomaly-detection/infer-sample
GET  /api/ml/anomaly-detection/latest?model=commit_pattern
GET  /api/ml/anomaly-detection/reports/<filename>

POST /api/ml/integrity-verification/hash
POST /api/ml/integrity-verification/hash/verify
POST /api/ml/integrity-verification/sbom/hash
POST /api/ml/integrity-verification/merkle/root
POST /api/ml/integrity-verification/keys/ed25519
POST /api/ml/integrity-verification/sign/ed25519
POST /api/ml/integrity-verification/verify/ed25519

GET  /api/ml/integrity-verification/microsoft-malware/status
POST /api/ml/integrity-verification/microsoft-malware/train
POST /api/ml/integrity-verification/microsoft-malware/infer-dataset
GET  /api/ml/integrity-verification/microsoft-malware/latest
GET  /api/ml/integrity-verification/microsoft-malware/reports/<filename>

GET  /api/ml/reporting-portal/status
POST /api/ml/reporting-portal/train
POST /api/ml/reporting-portal/infer-sample
POST /api/ml/reporting-portal/infer-dataset
GET  /api/ml/reporting-portal/latest?model=attack_type
GET  /api/ml/reporting-portal/reports/<model>/<filename>
GET  /api/ml/reporting-portal/forensic-data
GET  /api/ml/reporting-portal/custody
POST /api/ml/reporting-portal/report/generate
            

9) PM2 Runtime

PM2 config is backend/ecosystem.config.js. The ML API process: forensic-ml-api runs backend/api/ml_api.py using the venv interpreter backend/venv/bin/python3, port 5000.

cd /var/www/forensic-portal/backend
pm2 start ecosystem.config.js
pm2 restart ecosystem.config.js --update-env
pm2 logs forensic-ml-api
pm2 save
            

10) curl Test Commands (Full Catalog)

Set BASE once and run any command. These are copy-pasteable smoke tests for every model and non-ML integrity function. All API routes are under /api/ml/.

BASE=https://monitoringsystem.space
HJSON='Content-Type: application/json'
            

Health

# Check if ML API is reachable; returns {"success": true}
curl -sS $BASE/api/ml/health
            

CI/CD Agents

# Get total agents and their statuses
curl -sS $BASE/api/ml/cicd-agents/status

# List recently collected evidence items (e.g., git-diffs, logs)
curl -sS $BASE/api/ml/cicd-agents/evidence?limit=50

# View current agent configuration (what to capture)
curl -sS $BASE/api/ml/cicd-agents/config

# Register a new agent (simulated registration)
curl -sS -X POST $BASE/api/ml/cicd-agents/register -H "$HJSON" -d '{"name":"curl-agent-1","type":"Jenkins"}'

# Update configuration (disable auto_collect, set interval)
curl -sS -X POST $BASE/api/ml/cicd-agents/config -H "$HJSON" -d '{"config":{"auto_collect":false,"collection_interval_seconds":30}}'

# Reset configuration to defaults
curl -sS -X POST $BASE/api/ml/cicd-agents/config/reset -H "$HJSON" -d '{}'

# Trigger manual evidence collection from dataset (creates evidence items)
curl -sS -X POST $BASE/api/ml/cicd-agents/collect -H "$HJSON" -d '{"mode":"dataset"}'

# Verify new evidence was added
curl -sS $BASE/api/ml/cicd-agents/evidence?limit=20
            

CI/CD Agents — HDFS Log Anomaly Detection

# Check training status of HDFS model
curl -sS $BASE/api/ml/cicd-agents/hdfs/status

# Train HDFS model (unsupervised) on 20k log lines
curl -sS -X POST $BASE/api/ml/cicd-agents/hdfs/train -H "$HJSON" -d '{"limit":20000}'

# Verify status updates to "trained"
curl -sS $BASE/api/ml/cicd-agents/hdfs/status

# Run inference on dataset (limit 5000 lines) and store results
curl -sS -X POST $BASE/api/ml/cicd-agents/hdfs/infer-dataset -H "$HJSON" -d '{"limit":5000,"store":true}'

# Retrieve the stored inference results (anomalies)
curl -sS $BASE/api/ml/cicd-agents/hdfs/latest

# Download generated graphs and training summary
curl -sS -o hdfs_dataset_matrix.png $BASE/api/ml/cicd-agents/hdfs/reports/hdfs_dataset_matrix.png
curl -sS -o hdfs_score_distribution.png $BASE/api/ml/cicd-agents/hdfs/reports/hdfs_score_distribution.png
curl -sS -o hdfs_training_summary.json $BASE/api/ml/cicd-agents/hdfs/reports/hdfs_training_summary.json
            

CI/CD Agents — Hadoop Log Failure Classification

# Check training status of Hadoop model
curl -sS $BASE/api/ml/cicd-agents/hadoop/status

# Train Hadoop supervised model on 12k log entries
curl -sS -X POST $BASE/api/ml/cicd-agents/hadoop/train -H "$HJSON" -d '{"limit":12000}'

# Verify status is "trained"
curl -sS $BASE/api/ml/cicd-agents/hadoop/status

# Run classification on dataset and store failures
curl -sS -X POST $BASE/api/ml/cicd-agents/hadoop/infer-dataset -H "$HJSON" -d '{"limit":3000,"store":true}'

# Retrieve latest detected failures
curl -sS $BASE/api/ml/cicd-agents/hadoop/latest

# Download classification metrics and confusion matrix
curl -sS -o hadoop_confusion_matrix.png $BASE/api/ml/cicd-agents/hadoop/reports/hadoop_confusion_matrix.png
curl -sS -o hadoop_metrics.png $BASE/api/ml/cicd-agents/hadoop/reports/hadoop_metrics.png
curl -sS -o hadoop_training_summary.json $BASE/api/ml/cicd-agents/hadoop/reports/hadoop_training_summary.json
            

Anomaly Detection Engine

# Check status of both anomaly models
curl -sS $BASE/api/ml/anomaly-detection/status

# Train both Commit Pattern and Dependency Anomaly models
curl -sS -X POST $BASE/api/ml/anomaly-detection/train -H "$HJSON" -d '{"models":["commit_pattern","dependency_anomaly"],"limit":2000}'

# Verify both are trained
curl -sS $BASE/api/ml/anomaly-detection/status

# Run inference for Commit Patterns and store results
curl -sS -X POST $BASE/api/ml/anomaly-detection/infer -H "$HJSON" -d '{"model":"commit_pattern","limit":500,"store":true}'
curl -sS $BASE/api/ml/anomaly-detection/latest?model=commit_pattern

# Run inference for Dependency Anomalies and store results
curl -sS -X POST $BASE/api/ml/anomaly-detection/infer -H "$HJSON" -d '{"model":"dependency_anomaly","limit":500,"store":true}'
curl -sS $BASE/api/ml/anomaly-detection/latest?model=dependency_anomaly

# Infer a single sample (manual JSON input)
curl -sS -X POST $BASE/api/ml/anomaly-detection/infer-sample -H "$HJSON" -d '{"model":"commit_pattern","sample":{"commit_message_length":240,"files_changed":12,"lines_added":900,"lines_removed":20,"is_off_hours":1,"is_weekend":0,"timestamp_hour":2,"day_of_week":6,"time_since_last_commit":3600,"author_commit_count":1}}'

# Download generated anomaly reports
curl -sS -o commit_confusion_matrix.png $BASE/api/ml/anomaly-detection/reports/commit_confusion_matrix.png
curl -sS -o commit_metrics.png $BASE/api/ml/anomaly-detection/reports/commit_metrics.png
curl -sS -o dependency_heatmap.png $BASE/api/ml/anomaly-detection/reports/dependency_correlation_heatmap.png
curl -sS -o training_summary.json $BASE/api/ml/anomaly-detection/reports/anomaly_training_summary.json
            

Integrity Verification (Hash / SBOM / Merkle / Ed25519)

# Compute SHA-256 hash of a string
curl -sS -X POST $BASE/api/ml/integrity-verification/hash -H "$HJSON" -d '{"content":"hello"}'

# Verify content matches an expected hash
curl -sS -X POST $BASE/api/ml/integrity-verification/hash/verify -H "$HJSON" -d '{"content":"hello","expected_hash":"deadbeef"}'

# Compute hash of an SBOM JSON structure
curl -sS -X POST $BASE/api/ml/integrity-verification/sbom/hash -H "$HJSON" -d '{"sbom_json":"{\"name\":\"demo\",\"dependencies\":[{\"name\":\"a\",\"version\":\"1.0.0\"}]}"}'

# Compute Merkle Root for a list of leaf hashes
curl -sS -X POST $BASE/api/ml/integrity-verification/merkle/root -H "$HJSON" -d '{"hashes":["a3f1","b2c9","c0d1"]}'

# Ed25519: Generate Keypair, Sign, and Verify
KEYS=$(curl -sS -X POST $BASE/api/ml/integrity-verification/keys/ed25519 -H "$HJSON")
PRIV=$(printf '%s' "$KEYS" | python3 -c 'import sys,json;print(json.load(sys.stdin)["private_key_pem"])')
PUB=$(printf '%s' "$KEYS" | python3 -c 'import sys,json;print(json.load(sys.stdin)["public_key_pem"])')
SIG=$(curl -sS -X POST $BASE/api/ml/integrity-verification/sign/ed25519 -H "$HJSON" -d "{\"payload\":\"hello\",\"private_key_pem\":$(python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))' <<<"$PRIV")}" | python3 -c 'import sys,json;print(json.load(sys.stdin)["signature_b64"])')
curl -sS -X POST $BASE/api/ml/integrity-verification/verify/ed25519 -H "$HJSON" -d "{\"payload\":\"hello\",\"public_key_pem\":$(python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))' <<<"$PUB"),\"signature_b64\":\"$SIG\"}"
            

Integrity Verification — Microsoft Malware Classification

# Check status of malware model
curl -sS $BASE/api/ml/integrity-verification/microsoft-malware/status

# Train malware classifier (supervised SGD)
curl -sS -X POST $BASE/api/ml/integrity-verification/microsoft-malware/train -H "$HJSON" -d '{"limit":30000}'
curl -sS $BASE/api/ml/integrity-verification/microsoft-malware/status

# Run inference on dataset and store results
curl -sS -X POST $BASE/api/ml/integrity-verification/microsoft-malware/infer-dataset -H "$HJSON" -d '{"limit":500,"store":true}'
curl -sS $BASE/api/ml/integrity-verification/microsoft-malware/latest

# Download malware classification reports
curl -sS -o microsoft_malware_confusion_matrix.png $BASE/api/ml/integrity-verification/microsoft-malware/reports/microsoft_malware_confusion_matrix.png
curl -sS -o microsoft_malware_metrics.png $BASE/api/ml/integrity-verification/microsoft-malware/reports/microsoft_malware_metrics.png
curl -sS -o microsoft_malware_accuracy_curve.png $BASE/api/ml/integrity-verification/microsoft-malware/reports/microsoft_malware_accuracy_curve.png
curl -sS -o microsoft_malware_training_summary.json $BASE/api/ml/integrity-verification/microsoft-malware/reports/microsoft_malware_training_summary.json
            

Reporting Portal (Evidence/Custody/Report)

# Fetch timeline evidence (from CI/CD + Anomalies)
curl -sS $BASE/api/ml/reporting-portal/forensic-data

# Fetch chain of custody logs
curl -sS $BASE/api/ml/reporting-portal/custody

# Generate a full forensic report JSON
curl -sS -X POST $BASE/api/ml/reporting-portal/report/generate -H "$HJSON" -d '{"format":"json"}'
            

Reporting Portal — ML Models (Attack Type / Breach Type)

# Check status of reporting models
curl -sS $BASE/api/ml/reporting-portal/status

# Train Attack Type model
curl -sS -X POST $BASE/api/ml/reporting-portal/train -H "$HJSON" -d '{"models":["attack_type"],"limit_attack":6000}'

# Train Breach Type model
curl -sS -X POST $BASE/api/ml/reporting-portal/train -H "$HJSON" -d '{"models":["breach_type"]}'

# Infer Attack Type on dataset and store
curl -sS -X POST $BASE/api/ml/reporting-portal/infer-dataset -H "$HJSON" -d '{"model":"attack_type","limit":200,"store":true}'
curl -sS $BASE/api/ml/reporting-portal/latest?model=attack_type

# Infer Breach Type on dataset and store
curl -sS -X POST $BASE/api/ml/reporting-portal/infer-dataset -H "$HJSON" -d '{"model":"breach_type","limit":200,"store":true}'
curl -sS $BASE/api/ml/reporting-portal/latest?model=breach_type

# Infer single sample: Attack Type
curl -sS -X POST $BASE/api/ml/reporting-portal/infer-sample -H "$HJSON" -d '{"model":"attack_type","sample":{"Attack Name":"Malware","Description":"Ransomware infection spreading laterally in network"}}'

# Infer single sample: Breach Type
curl -sS -X POST $BASE/api/ml/reporting-portal/infer-sample -H "$HJSON" -d '{"model":"breach_type","sample":{"Organization":"Example Corp","Summary":"Unauthorized access and data exfiltration detected","Records_Lost":120000}}'

# Download reporting model graphs
curl -sS -o attack_type_metrics.png $BASE/api/ml/reporting-portal/reports/attack_type/attack_type_metrics.png
curl -sS -o attack_type_confusion_matrix.png $BASE/api/ml/reporting-portal/reports/attack_type/attack_type_confusion_matrix.png
curl -sS -o attack_type_learning_curve.png $BASE/api/ml/reporting-portal/reports/attack_type/attack_type_learning_curve.png
curl -sS -o breach_type_metrics.png $BASE/api/ml/reporting-portal/reports/breach_type/breach_type_metrics.png
curl -sS -o breach_type_confusion_matrix.png $BASE/api/ml/reporting-portal/reports/breach_type/breach_type_confusion_matrix.png
curl -sS -o breach_type_learning_curve.png $BASE/api/ml/reporting-portal/reports/breach_type/breach_type_learning_curve.png
            

11) Troubleshooting

curl -i https://your-host/api/ml/health
                

cd /var/www/forensic-portal/backend
pm2 status
pm2 logs forensic-ml-api