Action: enrich
Allows using CSV lookup to enrich data
This step requires special attention, as it needs to have its files attached to the pipe that will use the table lookup mechanisms. This is convenient as updated tables can be bundled with your pipes and managed centrally.
Please see the discussion and the example provided
at the end of the Table Lookup section.
Example: Add known network ports to events
File: lookup.csv
port,service
22,ssh
80,http
443,https
Input:
{"port":22}
{"port":80}
{"port":100}
{"port":443}
Pipe Language Snippet:
enrich:
lookup-file: lookup.csv
match:
- type: num
event-field: port
lookup-field: port
add:
event-field: service
lookup-field: service
Output:
{"port":22,"service":"ssh"}
{"port":80,"service":"http"}
{"port":100}
{"port":443,"service":"https"}
Field Summary
| Field Name | Type | Description | Default |
|---|---|---|---|
| condition | expression | Only run this action if the condition the specified condition is met | - |
| lookup-file | path | A file containing lookups | - |
| dynamic | bool | Do not panic if lookup-file does not immediately exist at pipe creation time | false |
| add | AddEnum | Detail on what to add to the event, based on the match. |
If there is no default value, then the output field will not be added to the event. |-| |match|Match|Detail on what to match on, associating event fields and lookup fields|-|
Fields
condition
Type: expression
Only run this action if the condition the specified condition is met
lookup-file
Type: path
A file containing lookups
dynamic
Type: bool
Default: false
Do not panic if lookup-file does not immediately exist at pipe creation time
add
Type: AddEnum
Detail on what to add to the event, based on the match. If there is no default value, then the output field will not be added to the event.
| Field Name | Type | Description | Default |
|---|---|---|---|
| add | Add | A field value to add to the event | - |
| add-multiple | AddMultiple | Field values to add to the event | - |
add
Type: Add
A field value to add to the event
| Field Name | Type | Description | Default |
|---|---|---|---|
| event-field | field | Field name to be added to the event | - |
| lookup-field | field | Field (CSV header) to lookup data to be place in event-field | - |
| default-value | field | YAML formatted default value if the event is empty | - |
event-field
Type: field
Field name to be added to the event
lookup-field
Type: field
Field (CSV header) to lookup data to be place in event-field
default-value
Type: field
YAML formatted default value if the event is empty
Example: type=cidr
File: lookup7.csv
office,network
Customer,192.168.26.0/28
Headquarters,192.168.85.0/24
Input:
{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}
Pipe Language Snippet:
enrich:
lookup-file: lookup7.csv
match:
- type: cidr
event-field: address
lookup-field: network
add:
event-field: office
lookup-field: office
default-value: unknown
Output:
{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100","office":"unknown"}
{"address":"192.168.85.100","office":"Headquarters"}
add-multiple
Type: AddMultiple
Field values to add to the event
| Field Name | Type | Description | Default |
|---|---|---|---|
| event-fields | array of (field,value) pairs | Add multiple fields to a single event based on a single match, providing a default | - |
event-fields
Type: array of (field,value) pairs
Add multiple fields to a single event based on a single match, providing a default
The limitation with this shortcut is that the lookup field name should be the same as the event field
Example: type=cidr
File: lookup8.csv
office,network
Customer,192.168.26.0/28
Headquarters,192.168.85.0/24
Input:
{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}
Pipe Language Snippet:
enrich:
lookup-file: lookup8.csv
match:
- type: cidr
event-field: address
lookup-field: network
add:
event-fields:
- office: unknown
Output:
{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100","office":"unknown"}
{"address":"192.168.85.100","office":"Headquarters"}
match
Type: Match
Detail on what to match on, associating event fields and lookup fields
| Field Name | Type | Description | Default |
|---|---|---|---|
| type | string | Type of match, one of the following: str, num, cidr, ip, num-range, num-list, str-list | - |
| event-field | field | Event field matched | - |
| lookup-field | field | Lookup field matched | - |
type
Type: string
Possible Values: str, num, cidr, ip, num-range, num-list, str-list
Type of match, one of the following: str, num, cidr, ip, num-range, num-list, str-list
Example: matching strings
File: lookup1.csv
port,service
22,ssh
80,http
443,https
Input:
{"service":"ssh"}
{"service":"http"}
{"service":"unknown"}
{"service":"https"}
Pipe Language Snippet:
enrich:
lookup-file: lookup1.csv
match:
- type: str
event-field: service
lookup-field: service
add:
event-field: port
lookup-field: port
Output:
{"service":"ssh","port":"22"}
{"service":"http","port":"80"}
{"service":"unknown"}
{"service":"https","port":"443"}
Example: matching CIDR
File: lookup2.csv
office,network
Customer,192.168.26.0/28
Headquarters,192.168.85.0/24
Input:
{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}
Pipe Language Snippet:
enrich:
lookup-file: lookup2.csv
match:
- type: cidr
event-field: address
lookup-field: network
add:
event-field: office
lookup-field: office
Output:
{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100","office":"Headquarters"}
Example: IP address (if event match.event-field does not match ip address format, the event will be filtered out)
File: lookup3.csv
fqdn,address
domain.io,206.189.28.194
Input:
{"address":"192.168.26.10"}
{"address":"206.189.28.194"}
Pipe Language Snippet:
enrich:
lookup-file: lookup3.csv
match:
- type: ip
event-field: address
lookup-field: address
add:
event-field: fqdn
lookup-field: fqdn
Output:
{"address":"192.168.26.10"}
{"address":"206.189.28.194","fqdn":"domain.io"}
Example: Type is a range of numbers
File: lookup4.csv
range,grouping
0-3,small numbers
3-10,larger numbers
Input:
{"number":1}
{"number":10}
{"number":100}
Pipe Language Snippet:
enrich:
lookup-file: lookup4.csv
match:
- type: num-range
event-field: number
lookup-field: range
add:
event-field: grouping
lookup-field: grouping
Output:
{"number":1,"grouping":"small numbers"}
{"number":10,"grouping":"larger numbers"}
{"number":100}
Example: Type is a list of strings
File: lookup5.csv
lists,grouping
"zero,two,four",even
"one,three,five",odd
Input:
{"number":"one"}
{"number":"two"}
{"number":"three"}
Pipe Language Snippet:
enrich:
lookup-file: lookup5.csv
match:
- type: str-list
event-field: number
lookup-field: lists
add:
event-field: grouping
lookup-field: grouping
Output:
{"number":"one","grouping":"odd"}
{"number":"two","grouping":"even"}
{"number":"three","grouping":"odd"}
Example: CDIR with multiple matches
File: lookup6.csv
source,destination,label
192.168.26.0/24,192.168.26.0/24,sameSide
192.168.85.0/24,192.168.85.0/24,sameSide
192.168.26.0/24,192.168.85.0/24,sameSide
192.168.26.0/24,0.0.0.0/0,outbound
192.168.85.0/24,0.0.0.0/0,outbound
0.0.0.0/0,192.168.26.0/24,inbound
0.0.0.0/0,192.168.85.0/24,inbound
0.0.0.0/0,0.0.0.0/0,unknown
Input:
{"src":"192.168.26.10","dst":"192.168.26.11"}
{"src":"192.168.26.10","dst":"192.168.85.10"}
{"src":"192.168.85.10","dst":"192.168.85.11"}
{"src":"192.168.26.10","dst":"192.168.86.10"}
{"src":"192.168.86.10","dst":"192.168.26.10"}
{"src":"192.168.86.10","dst":"192.168.86.11"}
Pipe Language Snippet:
enrich:
lookup-file: lookup6.csv
match:
- type: cidr
event-field: src
lookup-field: source
- type: cidr
event-field: dst
lookup-field: destination
add:
event-field: traffic-direction
lookup-field: label
Output:
{"src":"192.168.26.10","dst":"192.168.26.11","traffic-direction":"sameSide"}
{"src":"192.168.26.10","dst":"192.168.85.10","traffic-direction":"sameSide"}
{"src":"192.168.85.10","dst":"192.168.85.11","traffic-direction":"sameSide"}
{"src":"192.168.26.10","dst":"192.168.86.10","traffic-direction":"outbound"}
{"src":"192.168.86.10","dst":"192.168.26.10","traffic-direction":"inbound"}
{"src":"192.168.86.10","dst":"192.168.86.11","traffic-direction":"unknown"}
event-field
Type: field
Event field matched
lookup-field
Type: field
Lookup field matched