Spectral comes with an industry's leading detector coverage with over 2500 different detectors built-in, including machine learning based detectors. We test and maintain our ever-growing set of detectors with our own proprietary data set of over 180GB of data every day.
Spectral's detector toolkit is available to you, so you could build your own detectors for any purpose you find suitable.
We plan, you scan
Every week, we plan our next batch of detector additions. We constantly analyze, research and release new detectors. In fact, it's one of our internal KPIs.
Let's build a detector rule to identify merchant IDs. Let's say they look like this:
MKAT82jv72D2g92a MKAT02maibnwMw37s MKAT10dmwugmwpz9 ...
Spectral is CLI driven so we can use it to generate a nice starting point for us:
$HOME/.spectral/spectral init multi hello-rules --pattern "MKAT[0-9a-zA-Z]+"
$HOME/.spectral/spectral inits, you should have a
.spectralfolder ready for storing Spectral related material, such as your custom rules in
A regex for these IDs is pretty simple to author:
Which means: "Start with MKAT, followed by one or more of alphanumerics".
Let's look at the detector rule that Spectral generated for us:
rules: - id: RUL001 name: Your sample rule pattern_group: patterns: - pattern: "MKAT[0-9a-zA-Z]+" pattern_type: multi
Spectral automatically picks up the new rule, so to use it we can just run:
$HOME/.spectral/spectral run --just-ids RUL001 --nosend
Let's say our MKAT tokens also get a new format:
KAT__310527195 KAT__310527110 KAT__310527100
Obviously, we could model this into the same regex, or create a new rule, but we don't have to.
Let's use Spectral's logical capabilities to create a cleaner and more efficient rule:
rules: - id: RUL001 name: Your sample rule pattern_group: patterns: - pattern: "MKAT[0-9a-zA-Z]+" pattern_type: multi - pattern: "KAT__[0-9]+" pattern_type: multi
Spectral runs a hierarchical matching engine. Which means a pattern contain a pattern expression, in which each pattern can contain another pattern expression, of which each pattern can contain a bunch of other pattern expressions and so on.
Let's add a condition, where if we found a KAT expression, the file in which the expression is found must contain the word "production" and in general these merchant IDs are only a risk if appearing in Python source code.
rules: - id: RUL001 name: Your sample rule applies_to: # filter for the correct file pattern - ".*\\.py$" pattern_group: patterns: - pattern: "MKAT[0-9a-zA-Z]+" pattern_type: regex - pattern: "KAT__[0-9]+" pattern_type: regex pattern_group: # the same pattern group construct repeats patterns: - pattern: ".*production.*" pattern_type: regex
As you can see, by nesting the same structure we get the same powerful constructs again, and again. With these capabilities, Spectral can model virtually any kind of expression you'd find interesting to have.
Each pattern can have an assortment of testers. Think of these as intelligent pieces of small detectors that are purpose-built for many types of data, that you can mix and match.
Instead of using a hierarchical expression, let's use a tester. In this case let's ensure that the last part of MKAT validates as a machine generated secret:
rules: - id: RUL001 name: Your sample rule applies_to: # filter for the correct file pattern - ".*\\.py$" pattern_group: patterns: - pattern: "MKAT[0-9a-zA-Z]+" pattern_type: regex - pattern: "KAT__(.+)" pattern_type: regex test_token: - on: 1 is: true
We can lower the detection guarantees and accept any character in
KAT__(.+), and the
test_token machine learning based tester will do the hard work for us of finding the right match.
The spectral rule engine has a few basic constructs that in turn, have powerful knobs and switches. In combination, it makes for a robust query engine and static analyzer for looking for security patterns.
The basic building block in Spectral. A pattern is an expression you're looking for in text, there are several kinds of patterns that are supported and the list is always growing:
Each pattern can optionally contain a pattern group, used as a verification or additional conditions on the pattern.
A pattern group aggregates a bunch of patterns by some kind of logic that you specify. For rules A, B, and C expressions like the following can be made:
- A and B and C
- A or (B and C)
- not A or (B and C)
Each rule can activate for certain file path patterns or decativate for others. In addition, we can "tag" rules as belonging to certain categories so that we can filter these in or out in our Spectral dashboard or our Spectral configuration.
Here it is on our previous rule:
rules: - id: RUL001 name: Your sample rule applies_to: # filter for the correct file pattern - ".*\\.py$" tags: - python
To learn how to select categories of rules in Configuration
Updated 11 months ago