Building TIDE: Threat Informed Detection Engineering
As Detection Engineers, we often find ourselves straddling two disconnected worlds. In one browser tab, we monitor our SIEM (Elastic, Splunk, etc.), managing a vast library of detection rules. In another, we scroll through threat intelligence feeds (OpenCTI, MITRE ATT&CK®), trying to keep up with the latest adversary behaviors.
But a critical question often remains unanswered: “If APT29 attacked us today, would our currently enabled rules actually catch them?”
I built TIDE (Threat Informed Detection Engine) to answer that question programmatically. TIDE is a platform that audits the health of your detection rules, ingests live threat intelligence, and maps the two together to visualize actual defensive coverage gaps.
1. The Problem: “Broken” Rules and Blind Spots
We faced two primary challenges that TIDE was designed to solve:
- Rule Rot: Detection rules can silently fail. For example, a rule might query
process.command_line, but if an ingestion update changes that field toprocess.cmdlinein your actual index, the rule stops finding threats without ever throwing an error. - Theoretical vs. Actual Coverage: We might think we have coverage for MITRE T1059 (Command-Line Interface), but do our rules cover the specific procedures used by the threat actors actually targeting our sector?
2. Rule Health: Auditing at Scale
TIDE doesn’t just list your rules; it grades them. The engine calculates a Health Score (0-100) for each rule based on two primary pillars:
- Quality Score (50 pts): TIDE parses the rule query, supporting KQL, EQL, and ESQL and uses Regex to extract every referenced field name. It then queries the specific Elasticsearch index mappings to verify those fields actually exist and checks their data types.
- Meta Score (50 pts): It checks for essential context, including investigation guides, author data, and correct MITRE ATT&CK® mapping completeness.
3. Threat Intelligence: Knowing the Enemy
To understand our defensive coverage, we first need to define what we are defending against. TIDE integrates directly with MITRE ATT&CK® and OpenCTI.
Using STIX 2.1 parsing logic, TIDE ingests “Intrusion Sets” (Threat Actors) and extracts their known “Attack Patterns” (TTPs). This data is stored locally in an analytical database, ensuring the dashboard remains responsive without needing constant external API calls.
4. The Killer Feature: Dynamic Coverage Mapping
This is where TIDE truly shines. It creates a real-time intersection between your Defensive Posture and the Offensive Landscape. In the Coverage Matrix, TIDE visualizes this intersection using a heatmap:
- Green (Covered): Techniques used by the actor for which you have active, healthy rules.
- Red (Critical Gap): Techniques the actor uses for which you have zero current visibility.
- Blue (Defense in Depth): Active rules you have that are not currently utilized by the selected actor.
5. Under the Hood: The Tech Stack
TIDE has moved away from its Streamlit origins toward a more robust, production-ready architecture:
- Backend: FastAPI drives the logic, providing high-performance asynchronous API endpoints.
- Database: DuckDB serves as our analytical engine. It is an in-process SQL OLAP database that is incredibly fast at joining thousands of rules against TTPs instantly.
- Frontend: HTMX and Tailwind CSS 4 provide a reactive, modern UI experience without the overhead of heavy JavaScript frameworks.
- Orchestration: A dedicated worker.py runs on a schedule in a separate Docker container, syncing data from Elastic, OpenCTI, and MITRE in the background.
6. The Future
TIDE moves us away from “collecting rules” to “engineering coverage.” The roadmap involves making it an active engineering tool, including:
- Automated Suggestions: Identifying “Red” gaps and suggesting open-source Sigma rules to fill them.
- Cross-Platform Conversion: Leveraging the Sigma backend to automatically convert rules between Elastic, Splunk, and Microsoft Sentinel.