Skip to content

rule-insights.md

Latest commit

 

History

History
99 lines (65 loc) · 3.43 KB

rule-insights.md

File metadata and controls

99 lines (65 loc) · 3.43 KB

Insights and visualizations into rules and releases

Indexing rules for visualizing in Kibana

There are several ways to import or index rules into elasticsearch.

Indexing rules into Elasticsearch

The simplest way to index rules from the repo into elasticsearch is to run python -m detection-rules es index-rules

This will index an enriched version of all rules included and sent to the index rules-repo-<package-version>-<package_hash>

  • package-version is the version defined in detection_rules/etc/packages.yaml
  • package hash is the sha256 hash of the consolidated rules:
    • sorted by name
    • flattened
    • sorted by key
    • base64 encoded

Detailed usage

Usage: detection_rules es index-rules [OPTIONS]

  Index rules based on KQL search results to an elasticsearch instance.

Options:
  -q, --query TEXT          Optional KQL query to limit to specific rules
  -f, --from-file FILENAME  Load a previously saved uploadable bulk file
  -s, --save_files          Optionally save the bulk request to a file
  -h, --help                Show this message and exit.

The query can be any valid kql to reduce the scope of included rules, such as

-q "tags:Windows and severity>50"

Generating an index of the rules

Instead of automatically uploading the rules, you can save the files to do so locally and then import/upload

To do so, run python -m detection-rules generate-rules-index

This will generate 2 files under enriched-rule-indexes/<hash-of-package>:

  • enriched-rules-index-importable.ndjson
    • this is a standard ndjson file of flattened enriched rules
  • enriched-rules-index-uploadable.ndjson
    • this is an ndjson file in the format expected by the bulk api
    • this can be loaded via dev tools or sent as data using curl or any other method that hits the elasticsearch bulk api

The rules are enriched with several pieces of information and so are not identical representations of the rules generated with view-rule, though the hashes of the rules are generated before any enrichments are added.

Detailed usage

Usage: detection_rules generate-rules-index [OPTIONS]

  Generate enriched indexes of rules, based on a KQL search, for
  indexing/importing into elasticsearch/kibana.

Options:
  -q, --query TEXT  Optional KQL query to limit to specific rules
  --overwrite       Overwrite files in an existing folder
  -h, --help        Show this message and exit.

The query can be any valid kql to reduce the scope of included rules, such as

-q "tags:Windows and severity>50"

Importing rules via Kibana

If you have access to machine learning, you can leverage the data-visualizer to import the rules via the importable file.

After the rules have been indexed

Once indexed, the rules will need to be added to a kibana pattern, which will then make them searchable via discover or accessible in visualizations. Recommended index pattern is rules-* or rules-repo-*

For internal developers

Along with a series of other artifacts, these files are also generated at package creation, when running:

  • make release
  • python -m detection-rules build-release