First Commit
ci / go (push) Waiting to run
ci / go-db (agent) (push) Waiting to run
ci / go-db (config) (push) Waiting to run
ci / go-db (db) (push) Waiting to run
ci / go-db (evidence) (push) Waiting to run
ci / go-db (llmrec) (push) Waiting to run
ci / go-db (server) (push) Waiting to run
web / web (push) Waiting to run
docs / links (push) Canceled after 0s
detections / detections (push) Canceled after 0s
ci / go (push) Waiting to run
ci / go-db (agent) (push) Waiting to run
ci / go-db (config) (push) Waiting to run
ci / go-db (db) (push) Waiting to run
ci / go-db (evidence) (push) Waiting to run
ci / go-db (llmrec) (push) Waiting to run
ci / go-db (server) (push) Waiting to run
web / web (push) Waiting to run
docs / links (push) Canceled after 0s
detections / detections (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,456 @@
|
||||
# ARTEX detection rule tests
|
||||
|
||||
English · [한국어](README.ko.md)
|
||||
|
||||
> 한국어: 이 디렉터리는 [`../`](../)의 탐지 규칙이 실제로 발화하는지를 재현 가능하게 증명하는
|
||||
> 회귀 테스트입니다. 바이너리 캡처를 저장소에 넣지 않고, 패킷 캡처를 매번 결정론적으로 생성한 뒤
|
||||
> [Suricata](https://suricata.io)로 직접 돌려 경보 수를 확인합니다. 모든 테스트는 자신이 소유하거나
|
||||
> 서면 허가를 받은 시스템을 지키는 **방어·탐지 목적에만** 쓰십시오. 한국어 전체 문서는
|
||||
> **[README.ko.md](README.ko.md)** 를 보십시오.
|
||||
|
||||
Reproducible regression tests that prove the rules under [`../`](../) actually fire — and, just as
|
||||
important, stay silent on benign traffic. A detection rule you cannot run is a claim; these tests turn the
|
||||
claims in the rule files and the defense guide into something a reviewer can re-run from source.
|
||||
|
||||
No binary packet capture is committed. The capture is **synthesized deterministically on every run** and
|
||||
removed afterwards, so the test ships as readable source, not as an opaque fixture, and never bloats the
|
||||
repository.
|
||||
|
||||
## Run every suite at once — [`run-all.sh`](run-all.sh)
|
||||
|
||||
[`run-all.sh`](run-all.sh) runs all eight suites below in one command, in the same order as CI, so you do not
|
||||
have to invoke the eight `run.sh` scripts by hand. Each suite runs to completion even if an earlier one fails,
|
||||
the script prints a one-line PASS/FAIL summary per suite at the end, and it exits non-zero if any suite failed.
|
||||
|
||||
Before the suites, it runs a harness self-check ([`check-harness-sync.sh`](check-harness-sync.sh)) that fails
|
||||
the run if this suite list, the per-suite steps in [CI](../../.github/workflows/detections.yml), and the suite
|
||||
directories on disk ever name different suites or a different order. That is the one gap the eight suites
|
||||
cannot see on their own: a suite wired into only one of the three (a new CI step with no `run-all.sh` entry, or
|
||||
a directory never added to either) would otherwise pass every per-suite test while a green local `run-all.sh`
|
||||
quietly stopped meaning a green CI. The check is a gate, not a ninth suite: it stays out of the summary
|
||||
below, so the eight detection suites stay eight.
|
||||
|
||||
```sh
|
||||
detections/tests/run-all.sh
|
||||
```
|
||||
|
||||
Expected output (abridged):
|
||||
|
||||
```
|
||||
===== detection suites summary =====
|
||||
PASS sigma
|
||||
PASS sigma_match
|
||||
PASS sigma_lint
|
||||
PASS sigma_backends
|
||||
PASS suricata
|
||||
PASS attack
|
||||
PASS indicators
|
||||
PASS misp
|
||||
RESULT: PASS
|
||||
```
|
||||
|
||||
Because it exits non-zero on any failure, it drops straight into a pre-commit hook. A ready-to-use example
|
||||
lives in [`.pre-commit-config.yaml`](../../.pre-commit-config.yaml) at the repository root: install it with
|
||||
`pip install pre-commit && pre-commit install`, and the runner then fires on commits that touch the detection
|
||||
rules or the upstream source files they pin — the same scope as CI. The image and version overrides the
|
||||
individual suites honour (`PYTHON_IMAGE`, `SIGMA_CLI_VERSION`, `SIGMAHQ_VALIDATORS_VERSION`, `SURICATA_IMAGE`)
|
||||
are inherited by the runner, so exporting any of them applies to every suite at once.
|
||||
|
||||
## Suricata — [`suricata/`](suricata/)
|
||||
|
||||
[`suricata/run.sh`](suricata/run.sh) exercises the network rules in
|
||||
[`../suricata/artex.rules`](../suricata/artex.rules) end to end and asserts five properties:
|
||||
|
||||
- **Valid** — the whole rules file loads under `suricata -T --init-errors-fatal`, so a rule that fails to
|
||||
parse or initialise is caught even when no capture below exercises it. Plain `suricata -r` skips such a
|
||||
rule and still exits 0, so this load check is the Suricata analogue of the Sigma suite's `sigma check`
|
||||
validity assertion.
|
||||
- **Presence (enrich)** — sid `1000001` fires exactly once per enrichment probe.
|
||||
- **Velocity** — sid `1000002` fires once the `detection_filter` rate of 30 requests in 300 s per source is
|
||||
crossed.
|
||||
- **Presence (WebFetch)** — sid `1000003` fires exactly once per norma WebFetch request, and the enrich sids
|
||||
stay silent on that same capture — so the two network signatures are mutually specific, not just each
|
||||
present.
|
||||
- **Specificity** — an identical capture whose only change is a benign browser User-Agent produces **zero**
|
||||
ARTEX alerts.
|
||||
|
||||
[`suricata/gen_pcap.py`](suricata/gen_pcap.py) builds the capture with [scapy](https://scapy.net): N
|
||||
independent plaintext HTTP request/response flows from one fixed source, each carrying a chosen
|
||||
User-Agent, at a fixed base timestamp spaced one second apart. It only writes a file — it never sends a
|
||||
packet or touches a network.
|
||||
|
||||
### Run it
|
||||
|
||||
Needs only Docker; scapy and Suricata both run in containers.
|
||||
|
||||
```sh
|
||||
detections/tests/suricata/run.sh
|
||||
```
|
||||
|
||||
Expected output (abridged):
|
||||
|
||||
```
|
||||
PASS ruleset loads with zero parse/init errors (suricata -T)
|
||||
PASS sid 1000001 presence: one alert per probe (got 35, want eq 35)
|
||||
PASS sid 1000002 velocity: fires past 30-in-300s (got 5, want ge 1)
|
||||
PASS sid 1000003 presence: one alert per WebFetch request (got 8, want eq 8)
|
||||
PASS enrich sids stay silent on norma traffic (specificity) (got 0, want eq 0)
|
||||
PASS benign browser UA produces no ARTEX alerts (got 0, want eq 0)
|
||||
RESULT: PASS
|
||||
```
|
||||
|
||||
The script exits non-zero if any assertion fails, so it drops straight into CI or a pre-commit hook.
|
||||
Override the images with `SURICATA_IMAGE` / `PYTHON_IMAGE` if you mirror them internally.
|
||||
|
||||
### Why the velocity count is a floor, not an exact match
|
||||
|
||||
`run.sh` asserts the presence counts (`1000001 == 35`, `1000003 == 8`) and the benign count (`== 0`) exactly,
|
||||
because those are engine-version-independent: one alert per matching request, and no match on a different
|
||||
User-Agent. The
|
||||
velocity rule's count depends on how a given Suricata release resolves the `detection_filter` threshold at
|
||||
the boundary, so the test asserts `>= 1` and records the reference value separately. On **Suricata 8.0.7**
|
||||
the reference run produces **5** alerts on sid `1000002` (flows 31–35, after the 30-in-300 s threshold is
|
||||
crossed).
|
||||
|
||||
## Sigma — [`sigma/`](sigma/)
|
||||
|
||||
[`sigma/run.sh`](sigma/run.sh) validates the Sigma rules under [`../sigma/`](../sigma/) structurally and by
|
||||
compilation with [sigma-cli](https://github.com/SigmaHQ/sigma-cli) (pySigma), and asserts five properties:
|
||||
|
||||
- **Valid** — `sigma check` reports 0 errors, 0 condition errors, and 0 issues over the whole tree.
|
||||
- **Compiles** — `sigma convert -t splunk` turns the whole tree into a backend query language without error.
|
||||
- **Indicators survive** — each atomic indicator string (`artex-enrich/1.0`, `artex-selfupdate`, the guard
|
||||
marker, and the recording-proxy CA filename `mitmproxy-ca-cert.pem`) is still present in the compiled query,
|
||||
so a rule cannot silently lose the string it is built on.
|
||||
- **Correlations compile** — the behaviour rules in [`../sigma/correlation/`](../sigma/correlation/) emit their
|
||||
`event_count` / `value_count` aggregations rather than being dropped.
|
||||
- **Correlations are load-bearing** — converting one correlation rule *alone* fails, because it references its
|
||||
atomic base rule by `id`; the reference is enforced, not decorative. This is the Sigma analogue of the
|
||||
Suricata specificity assertion above.
|
||||
|
||||
This is the structural + compilation validation documented in [`../README.md`](../README.md), made executable
|
||||
and assertive. The companion [`sigma_match/`](sigma_match/) suite below adds the *matching* half for both the
|
||||
atomic and the correlation rules — a representative malicious event (or timeline) fires each rule and a benign
|
||||
one does not — so the Sigma rules now get both a reproducible validation test and a reproducible matching test,
|
||||
the way the Suricata rule does. (The
|
||||
earlier concern that a weak hand-written matcher would undercut the rules is addressed by delegating all parsing
|
||||
to pySigma; see the trust model in the next section.)
|
||||
|
||||
### Run it
|
||||
|
||||
Needs only Docker; sigma-cli and the splunk backend run in a container and nothing is written to the repo.
|
||||
|
||||
```sh
|
||||
detections/tests/sigma/run.sh
|
||||
```
|
||||
|
||||
Expected output (abridged):
|
||||
|
||||
```
|
||||
PASS sigma check: 0 errors, 0 condition errors, 0 issues
|
||||
PASS whole tree converts to splunk (exit 0)
|
||||
PASS indicator present: artex-enrich/1.0
|
||||
PASS correlation rule fails to convert alone — it requires its atomic base rule
|
||||
RESULT: PASS
|
||||
```
|
||||
|
||||
The script exits non-zero if any assertion fails, so it drops straight into CI or a pre-commit hook. sigma-cli
|
||||
is pinned to a reference version (`3.1.0`); override it with `SIGMA_CLI_VERSION`, or the image with
|
||||
`PYTHON_IMAGE`, if you mirror them internally.
|
||||
|
||||
## Sigma live event-matching — [`sigma_match/`](sigma_match/)
|
||||
|
||||
[`sigma_match/run.sh`](sigma_match/run.sh) proves the Sigma rules under [`../sigma/`](../sigma/) — both the
|
||||
atomic rules and the correlation rules under [`../sigma/correlation/`](../sigma/correlation/) — actually *fire*
|
||||
on a matching event (or timeline) and stay quiet on a benign one — the "a detection you cannot run is only a
|
||||
claim" guarantee the Suricata suite gives the network rule, extended here to the host/log-layer rules. It
|
||||
asserts six properties, three for the atomic rules and three for the correlations:
|
||||
|
||||
- **Rule/sample pairing** — every atomic rule has an [`events/<name>.json`](sigma_match/events/) sample file and
|
||||
every sample file maps back to a rule, so a rule added without samples fails here rather than going untested.
|
||||
- **True positives** — each rule matches every one of its malicious sample events.
|
||||
- **True negatives** — each rule matches none of its benign sample events. For example, a standalone
|
||||
`mitmproxy-ca-cert.pem` under `.mitmproxy/` does **not** trip the recording-proxy rule, because its `|all`
|
||||
modifier also requires the `_ca/` directory ARTEX writes — the matching test is what proves that discrimination.
|
||||
- **Correlation rule/timeline pairing** — every correlation rule has an
|
||||
[`events/correlation/<name>.json`](sigma_match/events/correlation/) timeline file and every timeline maps back
|
||||
to a rule. Each timeline event carries a `ts` field in relative seconds.
|
||||
- **Correlation true positives** — each rule fires on a positive timeline where the threshold is met inside the
|
||||
window within one group. For example, requests from one source (`c-ip`) fanning out to 20 distinct hosts
|
||||
within 10 minutes trip the enrichment fan-out rule.
|
||||
- **Correlation true negatives** — each rule stays quiet when the threshold is not met, when it is met but the
|
||||
events are spread beyond the window, when they are split across groups, or when a temporal rule is missing a
|
||||
leg. In particular a high-volume, low-breadth burst does **not** trip the fan-out rule: breadth, not volume, is
|
||||
the signal, and the matching test is what proves that discrimination.
|
||||
|
||||
The trust model is that pySigma — not hand-written code — parses each rule: an atomic rule into a condition tree
|
||||
(`|contains` → a wildcard value, `|all` → an AND, `1 of selection_*` → an OR), and a correlation rule into its
|
||||
aggregation spec (type, group-by, timespan, threshold, and the resolved references to the atomic base rules).
|
||||
[`check.py`](sigma_match/check.py) only walks that tree and spec, deciding which events feed a referenced rule
|
||||
with the very same atomic matcher, so the authoritative Sigma logic stays in pySigma; it raises rather than
|
||||
passing on any construct it does not explicitly support (fail-closed). Scope and limits are stated in the script
|
||||
header: the correlation window is the standard sliding-window interpretation (a `timespan`-second window
|
||||
anchored at each matching event) and a real SIEM's windowing may differ; matching is **case-insensitive** (the
|
||||
splunk-backend default the `sigma/` suite targets, which the destructive rule's own false-positive note
|
||||
assumes); and keyword matching is a full-text substring search. It is a regression test for the rules'
|
||||
field/value/condition/aggregation logic, not a substitute for validating in your own SIEM, whose field
|
||||
normalization may differ.
|
||||
|
||||
### Run it
|
||||
|
||||
Needs only Docker; pySigma runs in a container and nothing is written to the repo.
|
||||
|
||||
```sh
|
||||
detections/tests/sigma_match/run.sh
|
||||
```
|
||||
|
||||
Expected output (abridged):
|
||||
|
||||
```
|
||||
PASS rule/sample pairing: 5 atomic rules, 5 event files, no orphans
|
||||
PASS artex_enrich_user_agent: 1/1 positive events matched
|
||||
PASS artex_recording_proxy_ca: 2/2 benign events correctly not matched
|
||||
PASS rule/timeline pairing: 4 correlation rules, 4 timeline files, no orphans
|
||||
PASS artex_enrich_fanout: fired — 20 distinct hosts from one source within the 10-minute window
|
||||
PASS artex_enrich_fanout: quiet — high volume, low breadth: 25 requests from one source but only 4 distinct hosts
|
||||
RESULT: PASS
|
||||
```
|
||||
|
||||
The script exits non-zero if any assertion fails, so it drops straight into CI or a pre-commit hook. pySigma is
|
||||
pinned to a reference version (`2.0.0`); override it with `PYSIGMA_VERSION`, or the image with `PYTHON_IMAGE`,
|
||||
if you mirror them internally.
|
||||
|
||||
## Sigma backend portability — [`sigma_backends/`](sigma_backends/)
|
||||
|
||||
[`sigma_backends/run.sh`](sigma_backends/run.sh) proves the rules convert beyond the single Splunk example the
|
||||
Sigma test exercises, and keeps the per-backend support matrix in [`../README.md`](../README.md) honest. Sigma
|
||||
correlation conversion is backend-dependent, so the README tells a defender which `-t` targets take the whole
|
||||
tree and which take only the atomic rules — a claim that is only trustworthy if it is re-run. It asserts two
|
||||
properties, both positive so the test fails only on a real regression:
|
||||
|
||||
- **Correlations are portable** — the whole tree (atomic + correlation) converts on Splunk, the Elasticsearch
|
||||
`eql` target, and Grafana `loki`, with the enrich indicator surviving into each query. This shows the
|
||||
correlation rules are not Splunk-only.
|
||||
- **Atomic-only fallback works** — the five atomic rules still convert on `lucene` and the Microsoft `kusto`
|
||||
backend, which do not support Sigma correlation conversion at the pinned versions, so a defender on those
|
||||
backends can deploy the atomic rules and express the correlation window natively.
|
||||
|
||||
It deliberately does not assert the negative "backend X cannot do correlations": that would turn a backend
|
||||
*improving* into a red build. The honest limitation lives in the README, reproduced by this test's commands.
|
||||
[`sigma_backends/check.sh`](sigma_backends/check.sh) is the in-container half; it installs the pinned sigma-cli
|
||||
plus four backends and reads the rule tree mounted read-only.
|
||||
|
||||
### Run it
|
||||
|
||||
Needs only Docker; sigma-cli and the backends run in a container and nothing is written to the repo.
|
||||
|
||||
```sh
|
||||
detections/tests/sigma_backends/run.sh
|
||||
```
|
||||
|
||||
Expected output (abridged):
|
||||
|
||||
```
|
||||
PASS whole tree (atomic + correlation) converts on 'eql', enrich indicator survives
|
||||
PASS five atomic rules convert on 'kusto', enrich indicator survives
|
||||
RESULT: PASS
|
||||
```
|
||||
|
||||
The script exits non-zero if any assertion fails. sigma-cli is pinned (`3.1.0`, override with
|
||||
`SIGMA_CLI_VERSION`); the backend plugins install at their latest compatible version, so this suite is the one
|
||||
most sensitive to an upstream backend release — a plugin that drops support turns the build red, which is the
|
||||
signal to update the pin and the README matrix together.
|
||||
|
||||
## SigmaHQ convention lint — [`sigma_lint/`](sigma_lint/)
|
||||
|
||||
[`sigma_lint/run.sh`](sigma_lint/run.sh) makes the "passes `sigma check` cleanly" promise in the README and
|
||||
`CONTRIBUTING.md` cover SigmaHQ's conventions, not just pySigma's core checks. Plain `sigma check` does not
|
||||
load the `pySigma-validators-sigmahq` plugin, so title casing, field-name taxonomy, logsource taxonomy, and
|
||||
reference-link conventions go unchecked. This suite installs that plugin and runs the full set against the
|
||||
documented baseline in [`sigma_lint/validators.yml`](sigma_lint/validators.yml). It asserts two properties:
|
||||
|
||||
- **The documented baseline is clean** — `sigma check` with `validators.yml` reports 0 errors and 0 issues.
|
||||
- **The full set is live, and only the documented exclusions remain** — running every SigmaHQ validator with
|
||||
no exclusions still reports issues, and each one is among the four checks `validators.yml` deliberately
|
||||
disables (nothing else). This is the anti-vacuity guard: if the plugin failed to load, the full run would
|
||||
report nothing and the first property would pass for the wrong reason, so the known exclusions are required
|
||||
to appear.
|
||||
|
||||
The four exclusions encode SigmaHQ's monorepo filing scheme (logsource-prefixed and `correlation_` filenames)
|
||||
and taxonomy (a generic `application` logsource and a product-less `process_creation`), plus the
|
||||
branch-vs-permalink reference convention — none of which fit a small standalone rule set that references its
|
||||
own living docs. Each exclusion carries its rationale inline in `validators.yml`. Because every *other*
|
||||
SigmaHQ check is enforced, a rule that picks up a new convention issue — a mis-cased title, an off-taxonomy
|
||||
field name — turns the build red. `pySigma-validators-sigmahq` is pinned (`0.21.0`, override with
|
||||
`SIGMAHQ_VALIDATORS_VERSION`); bumping it may surface new conventions, which is the signal to update the rules
|
||||
or the documented baseline.
|
||||
|
||||
### Run it
|
||||
|
||||
Needs only Docker; sigma-cli and the validator plugin run in a container and nothing is written to the repo.
|
||||
|
||||
```sh
|
||||
detections/tests/sigma_lint/run.sh
|
||||
```
|
||||
|
||||
Expected output (abridged):
|
||||
|
||||
```
|
||||
PASS sigma check with the documented baseline: 0 errors, 0 issues
|
||||
PASS every reported issue is one of the four documented exclusions
|
||||
RESULT: PASS
|
||||
```
|
||||
|
||||
## ATT&CK layer — [`attack/`](attack/)
|
||||
|
||||
[`attack/run.sh`](attack/run.sh) checks that the [ATT&CK coverage layer](../attack/) in
|
||||
[`../attack/artex_navigator_layer.json`](../attack/artex_navigator_layer.json) stays consistent with the
|
||||
rules it claims to cover. A coverage layer that drifts from its rule set is worse than none, so this turns
|
||||
"these rules cover these ATT&CK techniques" into something a reviewer can re-run from source. It asserts:
|
||||
|
||||
- **Valid layer** — the file parses as JSON and carries the required ATT&CK Navigator v4.x fields, with a
|
||||
well-formed technique ID and a valid ATT&CK tactic on every entry.
|
||||
- **Bidirectional match** — the scored techniques are *exactly* the `attack.*` technique tags on the Sigma
|
||||
rules: no rule technique missing from the layer, no layer technique absent from the rules. The tactics
|
||||
match the same way.
|
||||
- **Grounded** — every scored technique's comment names a rule file that exists, so the layer cannot cite a
|
||||
rule that was renamed or removed.
|
||||
|
||||
This is a consistency check, not a firing test: it needs no detection backend, only the Python standard
|
||||
library, so unlike the Sigma and Suricata tests it carries no version-dependent counts. [`attack/check.py`](attack/check.py)
|
||||
is the in-container half; it reads the detections tree mounted read-only and writes nothing.
|
||||
|
||||
### Run it
|
||||
|
||||
Needs only Docker; the check runs in a Python container and nothing is written to the repo.
|
||||
|
||||
```sh
|
||||
detections/tests/attack/run.sh
|
||||
```
|
||||
|
||||
Expected output (abridged):
|
||||
|
||||
```
|
||||
PASS scored techniques match the rule set exactly (8: T1059, T1105, T1485, T1489, T1557, T1561.002, T1592, T1595)
|
||||
PASS scored tactics match the rule set exactly (collection, command-and-control, credential-access, execution, impact, reconnaissance)
|
||||
RESULT: PASS
|
||||
```
|
||||
|
||||
The script exits non-zero if any assertion fails, so it drops straight into CI or a pre-commit hook.
|
||||
Override the image with `PYTHON_IMAGE` if you mirror it internally.
|
||||
|
||||
## Indicator source-of-truth — [`indicators/`](indicators/)
|
||||
|
||||
[`indicators/run.sh`](indicators/run.sh) proves the one thing the three tests above do not: that each rule's
|
||||
pinned indicator is still the string ARTEX's own source actually emits. The Sigma test proves an indicator
|
||||
survives rule→query *compilation*; the ATT&CK test proves the layer matches the rules' tags; the Suricata
|
||||
test proves the network rule *fires* on a synthesized capture. None of them look back at the source file the
|
||||
indicator claims to come from. The rot they all miss is an upstream re-sync that bumps the prober User-Agent
|
||||
to `artex-enrich/2.0` or rewrites the guard marker: every rule still compiles, the layer still matches, the
|
||||
pcap test still fires — and the deployed rule silently stops matching real ARTEX traffic. It asserts, for
|
||||
each indicator, bidirectionally:
|
||||
|
||||
- **Source still emits it** — the value is present in the upstream source file(s) that produce it
|
||||
(`artex-enrich/1.0` in `enrich/enrich.go`, `artex-selfupdate` in `selfupdate/`, the guard marker in
|
||||
`guard/guard.go`). A missing value means an upstream change the rule has not caught up with.
|
||||
- **Rule still pins it** — the value is present in the rule built on it, so a rule edit cannot quietly move
|
||||
the indicator away from its source. The Suricata rule is checked by its `startswith` prefix, matching how
|
||||
it actually matches the wire.
|
||||
- **Deny-list correspondence** — the destructive-command tokens (`rm -rf`, `mkfs`, `DROP DATABASE`,
|
||||
`FLUSHALL`) appear both in ARTEX's guard deny-list (`db/db.go`) and in the hunting rule that mirrors it.
|
||||
These are generic hunting leads, not unique fingerprints, so the test asserts only the correspondence the
|
||||
rule actually claims.
|
||||
- **Published list stays grounded** — the machine-readable indicator list
|
||||
[`detections/indicators/artex_indicators.csv`](../indicators/artex_indicators.csv), the artifact a
|
||||
defender imports, is re-read row by row: every value must still be present in the source file(s) it cites
|
||||
and pinned in the rule(s) it cites, and every fingerprint the test grounds must appear in the list. So the
|
||||
published CSV cannot silently drift from the source it claims to come from, in either direction.
|
||||
- **Both gates fire on each pinned source** — every upstream source the test reads is covered by the two
|
||||
gates that run it: the CI workflow's `push` and `pull_request` paths filter
|
||||
([`.github/workflows/detections.yml`](../../.github/workflows/detections.yml)) and the local pre-commit
|
||||
hook's `files` regex ([`.pre-commit-config.yaml`](../../.pre-commit-config.yaml)). The required set is
|
||||
derived from the indicators themselves, so pinning a new source (as the `cmd/artex/main.go` ports once were)
|
||||
without wiring it into *both* gates fails here — otherwise a change touching only that source skips the test
|
||||
on whichever gate omits it: on CI it passes the merge gate green, on the hook it is never caught locally
|
||||
even though the hook promises "the same source scope as CI".
|
||||
|
||||
This turns [`../README.md`](../README.md)'s promise — "every indicator here is grounded in a string verified
|
||||
in this repository's source, not inferred" — and CONTRIBUTING's first contribution contract into a guard a
|
||||
reviewer can re-run. Like the ATT&CK test it needs no detection backend, only the Python standard library;
|
||||
[`indicators/check.py`](indicators/check.py) reads the rule tree, the published indicator list, the pinned
|
||||
source packages, and the two gates that fire it (the CI workflow and the pre-commit config) mounted read-only
|
||||
and writes nothing.
|
||||
|
||||
### Run it
|
||||
|
||||
Needs only Docker; the check runs in a Python container and nothing is written to the repo.
|
||||
|
||||
```sh
|
||||
detections/tests/indicators/run.sh
|
||||
```
|
||||
|
||||
Expected output (abridged):
|
||||
|
||||
```
|
||||
PASS enrichment prober User-Agent: 'artex-enrich/1.0' emitted by enrich/enrich.go
|
||||
PASS detections/sigma/artex_enrich_user_agent.yml pins 'artex-enrich/1.0'
|
||||
PASS 'FLUSHALL' present in both db/db.go and detections/sigma/destructive_command_hunting.yml
|
||||
PASS enrich-user-agent: 'artex-enrich/1.0' grounded in enrich/enrich.go
|
||||
PASS tested fingerprint 'artex-enrich/1.0' is published in the list
|
||||
PASS .github/workflows/detections.yml push paths covers cmd/artex/main.go
|
||||
PASS .pre-commit-config.yaml files covers cmd/artex/main.go
|
||||
RESULT: PASS
|
||||
```
|
||||
|
||||
The script exits non-zero if any assertion fails, so it drops straight into CI or a pre-commit hook.
|
||||
Override the image with `PYTHON_IMAGE` if you mirror it internally.
|
||||
|
||||
## MISP export consistency — [`misp/`](misp/)
|
||||
|
||||
[`misp/run.sh`](misp/run.sh) covers the second published form of the indicators — the ready-to-import MISP
|
||||
event [`detections/indicators/artex_indicators.misp.json`](../indicators/artex_indicators.misp.json). The
|
||||
indicator test above keeps the CSV grounded in the source; this test keeps the MISP event, the artifact a
|
||||
defender actually loads into a threat-intelligence platform, from drifting away from that CSV. It asserts:
|
||||
|
||||
- **It is really MISP** — the event loads under [pymisp](https://github.com/MISP/PyMISP), whose object model
|
||||
rejects any attribute whose `type` is not a genuine MISP type. A plausible-looking but invalid type fails
|
||||
here, so "valid MISP" is proven by the library a MISP server uses, not asserted.
|
||||
- **Row-for-row sync with the CSV** — every CSV row maps to exactly one MISP attribute with the intended type
|
||||
and category (`http.user-agent` → `user-agent`, the guard marker `string` → `pattern-in-file`, `port` →
|
||||
`port`, `ip-dst|port` → `ip-dst|port` with the composite `ip|port` value, and the exploration-schema `other` → `other`), and no MISP attribute is left
|
||||
without a CSV row. The event is hand-maintained alongside the CSV, so adding, removing, or retyping a CSV
|
||||
row without updating `artex_indicators.misp.json` to match in the same commit fails.
|
||||
- **`to_ids` mirrors the `rule` column** — a rule-backed indicator is `to_ids: true`; a host-forensic row
|
||||
with no rule is `to_ids: false` with `disable_correlation: true`. Flipping a flag away from what the CSV
|
||||
implies fails, so the MISP event cannot quietly over- or under-claim which fingerprints are actionable.
|
||||
- **The guard marker survives byte-for-byte** and `detections/**` is in the CI paths filter, so a change to
|
||||
the CSV or the event triggers this suite.
|
||||
|
||||
Unlike the pure-standard-library tests above, this suite installs a pinned `pymisp` inside its container
|
||||
(nothing is installed on the host); [`misp/check.py`](misp/check.py) reads the CSV, the MISP event, and the
|
||||
CI workflow mounted read-only and writes nothing.
|
||||
|
||||
### Run it
|
||||
|
||||
Needs only Docker; pymisp is installed in the container and nothing is written to the repo.
|
||||
|
||||
```sh
|
||||
detections/tests/misp/run.sh
|
||||
```
|
||||
|
||||
The script exits non-zero if any assertion fails. Override the image with `PYTHON_IMAGE` and the pinned
|
||||
library with `PYMISP_VERSION` if you mirror them internally.
|
||||
|
||||
## Contributing
|
||||
|
||||
A new detection rule is stronger with a test that shows it firing. Tests should synthesize their own input
|
||||
deterministically, assert engine-version-independent properties exactly (and softer ones as floors with a
|
||||
recorded reference), and avoid any content that reads as attack guidance. See
|
||||
[`../../CONTRIBUTING.en.md`](../../CONTRIBUTING.en.md) and the rule indexes in [`../README.md`](../README.md).
|
||||
|
||||
All eight suites run in CI (see [`../../.github/workflows/detections.yml`](../../.github/workflows/detections.yml))
|
||||
on every push or pull request that touches `detections/` — and the indicator test also runs when the upstream
|
||||
source files it pins (`enrich/`, `selfupdate/`, `guard/`, `db/`, `cmd/artex/main.go`) change — so a rule
|
||||
change that drops an indicator, drifts from the ATT&CK layer, stops converting on a documented backend,
|
||||
breaks a SigmaHQ convention, falls out of sync with the source, lets the MISP event drift from the CSV, or
|
||||
pins a new source the workflow does not yet watch turns the build red before it can merge.
|
||||
Reference in New Issue
Block a user