Guides

Quick Start

Suggest Edits

Spectral comes with an industry's leading detector coverage with over 2500 different detectors built-in, including machine learning based detectors. We test and maintain our ever-growing set of detectors with our own proprietary data set of over 180GB of data every day.

Spectral's detector toolkit is available to you, so you could build your own detectors for any purpose you find suitable.

📘

We plan, you scan

Every week, we plan our next batch of detector additions. We constantly analyze, research and release new detectors. In fact, it's one of our internal KPIs.

Your first detector

Let's build a detector rule to identify merchant IDs. Let's say they look like this:

MKAT82jv72D2g92a
MKAT02maibnwMw37s
MKAT10dmwugmwpz9
...

Spectral is CLI driven so we can use it to generate a nice starting point for us:

$HOME/.spectral/spectral init multi hello-rules --pattern "MKAT[0-9a-zA-Z]+"

📘

Spectral Configuration

After $HOME/.spectral/spectral inits, you should have a .spectral folder ready for storing Spectral related material, such as your custom rules in rules/

A regex for these IDs is pretty simple to author:

MKAT[0-9a-zA-Z]+

Which means: "Start with MKAT, followed by one or more of alphanumerics".

Let's look at the detector rule that Spectral generated for us:

rules:
- id: RUL001
  name: Your sample rule
  pattern_group:
    patterns:
    - pattern: "MKAT[0-9a-zA-Z]+"
      pattern_type: multi

Spectral automatically picks up the new rule, so to use it we can just run:

$HOME/.spectral/spectral run --just-ids RUL001 --nosend

Evolving Your First Rule

Let's say our MKAT tokens also get a new format: 

KAT__310527195
KAT__310527110
KAT__310527100

Obviously, we could model this into the same regex, or create a new rule, but we don't have to.

Let's use Spectral's logical capabilities to create a cleaner and more efficient rule:

rules:
- id: RUL001
  name: Your sample rule
  pattern_group:
    patterns:
    - pattern: "MKAT[0-9a-zA-Z]+"
      pattern_type: multi
    - pattern: "KAT__[0-9]+"
      pattern_type: multi

Playing With Hierarchical Expressions

Spectral runs a hierarchical matching engine. Which means a pattern contain a pattern expression, in which each pattern can contain another pattern expression, of which each pattern can contain a bunch of other pattern expressions and so on.

Let's add a condition, where if we found a KAT expression, the file in which the expression is found must contain the word "production" and in general these merchant IDs are only a risk if appearing in Python source code.

rules:
- id: RUL001
  name: Your sample rule
  applies_to: # filter for the correct file pattern
  - ".*\\.py$"
  pattern_group:
    patterns:
    - pattern: "MKAT[0-9a-zA-Z]+"
      pattern_type: regex
    - pattern: "KAT__[0-9]+"
      pattern_type: regex
      pattern_group:   # the same pattern group construct repeats
        patterns:
            - pattern: ".*production.*"
              pattern_type: regex

As you can see, by nesting the same structure we get the same powerful constructs again, and again. With these capabilities, Spectral can model virtually any kind of expression you'd find interesting to have.

Playing With Testers

Each pattern can have an assortment of testers. Think of these as intelligent pieces of small detectors that are purpose-built for many types of data, that you can mix and match.

Instead of using a hierarchical expression, let's use a tester. In this case let's ensure that the last part of MKAT validates as a machine generated secret:

rules:
- id: RUL001
  name: Your sample rule
  applies_to: # filter for the correct file pattern
  - ".*\\.py$"
  pattern_group:
    patterns:
    - pattern: "MKAT[0-9a-zA-Z]+"
      pattern_type: regex
    - pattern: "KAT__(.+)"
      pattern_type: regex
      test_token:
      - on: 1
        is: true

We can lower the detection guarantees and accept any character in KAT__(.+), and the test_token machine learning based tester will do the hard work for us of finding the right match.

Concepts

The spectral rule engine has a few basic constructs that in turn, have powerful knobs and switches. In combination, it makes for a robust query engine and static analyzer for looking for security patterns.

Pattern

The basic building block in Spectral. A pattern is an expression you're looking for in text, there are several kinds of patterns that are supported and the list is always growing:

Each pattern can optionally contain a pattern group, used as a verification or additional conditions on the pattern.

Pattern Group

A pattern group aggregates a bunch of patterns by some kind of logic that you specify. For rules A, B, and C expressions like the following can be made:

  • A and B and C
  • A or (B and C)
  • not A or (B and C)

Filtering and Tagging

Each rule can activate for certain file path patterns or decativate for others. In addition, we can "tag" rules as belonging to certain categories so that we can filter these in or out in our Spectral dashboard or our Spectral configuration.

Here it is on our previous rule:

rules:
- id: RUL001
  name: Your sample rule
  applies_to: # filter for the correct file pattern
  - ".*\\.py$"
  tags:
  - python

To learn how to select categories of rules in Configuration

Updated almost 2 years ago