Skip to main content

Action: expand

Converts simple separated data into JSON

This is called expand because it pulls JSON data (values, key-value pairs or JSON text) from a field.

Field Summary​

Field NameTypeDescriptionDefault
conditionexpressionOnly run this action if the condition the specified condition is met-
delimstringThe delimiter (separator) to be used,
warningboolIf set to true warnigns will no longer be ignoredfalse
input-fieldfieldField containing data_raw
document-modeboolWith 'expand array' collapse input-field between doc-start and doc-end into one text eventfalse
removeboolWhether to remove field after its value is usedfalse
csvCsvConverts CSV rows into JSON records-
key-valueKeyValueConverts data in key=value format-
eventsEventsConverts a single event into multiple events-
xmlXmlEnumExpand a field containing an XML document to a JSON representation-
multilinearray of (field,regex) pairsCollect lines into fields-
jsonboolTreat incoming data as already in JSON formatfalse

Fields​

condition​

Type: expression

Only run this action if the condition the specified condition is met

delim​

Type: string

Default: ,

The delimiter (separator) to be used

Example

Input:

{"row":"1 2"}

Pipe Language Snippet:

expand:
  delim: " "
  input-field: row
  csv:
    fields:
    - one: num
    - two: str

Output:

{"row":"1 2","one":1,"two":"2"}

warning​

Type: bool

Default: false

If set to true warnigns will no longer be ignored

input-field​

Type: field

Default: _raw

Field containing data

Example

Input:

{"good":"fast,safe","bad":"slow,dangerous"}

Pipe Language Snippet:

expand:
  input-field: good
  remove: true
  csv:
    fields:
    - speed: str
    - trust: str

Output:

{"bad":"slow,dangerous","speed":"fast","trust":"safe"}

document-mode​

Type: bool

Default: false

With 'expand array' collapse input-field between doc-start and doc-end into one text event

remove​

Type: bool
Alias: remove-field
Default: false

Whether to remove field after its value is used

Example

Input:

{"row":"1 2"}

Pipe Language Snippet:

expand:
  delim: " "
  remove: true
  input-field: row
  csv:
    fields:
    - one: num
    - two: str

Output:

{"one":1,"two":"2"}

csv​

Type: Csv

Converts CSV rows into JSON records

Field NameTypeDescriptionDefault
relaxed-schemaboolAllow picking just the first given number fields, and giving those namesfalse
headerboolWhether to treat first line as a CSV headerfalse
gen-headersboolWhether to generate automatic CSV headersfalse
autoconvertboolControl auto-conversion when reading in headerstrue
fieldsarray of key-value pairsSpecify fields and their types (str, num, bool, num)-
field-filepathFile containing "name:type" pairs. Specify fields and their types (str, num, bool, num)-
header-fieldfieldField containing header (CSV column names)-
header-field-typesboolField containing header has types specified (with name:type format)false
header-field-on-changeboolWith 'header-field`, only write out headers if columns changefalse
null-valuestringA substitue string value to be used in the event that a field is null-

  relaxed-schema​

Type: bool

Default: false

Allow picking just the first given number fields, and giving those names

Example

Input:

{"row":"1,2,3,4,5"}

Pipe Language Snippet:

expand:
  input-field: row
  remove: true
  csv:
    relaxed-schema: true
    fields:
      - first: num
      - second: str

Output:

{"first":1,"second":"2"}

Type: bool

Default: false

Whether to treat first line as a CSV header

Example

Input:

text,number
one,1
two,2

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  csv:
    header: true

Output:

{"text":"one","number":1}
{"text":"two","number":2}

  gen-headers​

Type: bool

Default: false

Whether to generate automatic CSV headers

Example

Input:

one,1

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  csv:
    gen-headers: true

Output:

{"_0":"one","_1":1}

  autoconvert​

Type: bool

Default: true

Control auto-conversion when reading in headers

  fields​

Type: array of key-value pairs

Specify fields and their types (str, num, bool, num)

Example

Input:

{"data":"one,1"}

Pipe Language Snippet:

expand:
  input-field: data
  csv:
    fields:
      - text: str
      - number: num

Output:

{"data":"one,1","text":"one","number":1}

  field-file​

Type: path

File containing "name:type" pairs. Specify fields and their types (str, num, bool, num)

Example

File: some-file

text:str
number:num

Input:

{"data":"one,1"}

Pipe Language Snippet:

expand:
  input-field: data
  csv:
    field-file: some-file

Output:

{"data":"one,1","text":"one","number":1}

  header-field​

Type: field

Field containing header (CSV column names)

Example

Input:

{"header":"text,number","row":"one,1"}

Pipe Language Snippet:

expand:
  input-field: row
  remove: true
  csv:
    header-field: header

Output:

{"text":"one","number":1}

  header-field-types​

Type: bool

Default: false

Field containing header has types specified (with name:type format)

Example

Input:

{"header":"text:str,number:str","row":"one,1"}

Pipe Language Snippet:

expand:
  input-field: row
  remove: true
  csv:
    header-field: header
    header-field-types: true

Output:

{"text":"one","number":"1"}

  header-field-on-change​

Type: bool

Default: false

With 'header-field`, only write out headers if columns change

  null-value​

Type: string

A substitue string value to be used in the event that a field is null

key-value​

Type: KeyValue

Converts data in key=value format

Example: Something basic

Input:

one=1 two=2

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: '='

Output:

{"one":"1","two":"2"}

Example: Quoted strings

Input:

male="john doe" female="jane doe"

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: '='

Output:

{"male":"john doe","female":"jane doe"}
Field NameTypeDescriptionDefault
autoconvertboolAutomatically convert to numbers if possiblefalse
key-value-delimstringSet the delimiter between the key and the value=
multiplestringIn the case of repeated, choose whether to choose first, last, or to place all in an array-

  autoconvert​

Type: bool

Default: false

Automatically convert to numbers if possible

Example

Input:

one=1 two=2.0

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    autoconvert: true

Output:

{"one":1,"two":2}

  key-value-delim​

Type: string

Default: =

Set the delimiter between the key and the value

Example

Input:

one:1 two:2

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: ':'

Output:

{"one":"1","two":"2"}

  multiple​

Type: string

Possible Values: first, last, array

In the case of repeated, choose whether to choose first, last, or to place all in an array

Example: first

Input:

name=john name=jane

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: first

Output:

{"name":"john"}

Example: last

Input:

name=john name=jane

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: last

Output:

{"name":"jane"}

Example: array

Input:

name=john name=jane

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: array

Output:

{"name":["john","jane"]}

events​

Type: Events

Converts a single event into multiple events

Field NameTypeDescriptionDefault
output-split-fieldfieldA field where resulting split data is to be placed-
skip-listarray of JSON PointersAn array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"-
exclude-non-empty-arraysboolfalse

  output-split-field​

Type: field

A field where resulting split data is to be placed

Example

Input:

{"names":"john jane","family":"doe"}

Pipe Language Snippet:

expand:
  input-field: names
  remove: true
  delim: ' '
  events:
    output-split-field: name

Output:

{"family":"doe","name":"john"}
{"family":"doe","name":"jane"}

  skip-list​

Type: array of JSON Pointers

An array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"

Example

Input:

{ "array": [{"name": "jane","family":"doe" }, { "name":"john","family":"doe" }] }

Pipe Language Snippet:

expand:
  events:
    skip-list: []

Output:

{"array":{"name":"jane","family":"doe"}}
{"array":{"name":"john","family":"doe"}}

  exclude-non-empty-arrays​

Type: bool

Default: false

xml​

Type: XmlEnum

Expand a field containing an XML document to a JSON representation

Field NameTypeDescriptionDefault
yesboolSimply expand the field with no further optionsfalse
fieldsXmlExpand the field with additional options-

  yes​

Type: bool

Default: false

Simply expand the field with no further options

  fields​

Type: Xml

Expand the field with additional options

Field NameTypeDescriptionDefault
arraysarray of stringsList of fields in an xml payload to be expanded into separate events-

  arrays​

Type: array of strings

List of fields in an xml payload to be expanded into separate events

multiline​

Type: array of (field,regex) pairs

Collect lines into fields

Currently the patterns allowed are '\n' (line end) and '\n\n' (blank line)

Example: collecting pairs of lines into fields x and y

Input:

one 1
one 2
two 1
two 2

Pipe Language Snippet:

expand:
  input-field: _raw
  multiline:
    - x: '(\n)'
    - y: '(\n)'

Output:

{"x":"one 1","y":"one 2"}
{"x":"two 1","y":"two 2"}

Example: map first line to x and rest to y up to empty line

Input:

one 1
one 2
one 3
one 4

two 1
two 2

Pipe Language Snippet:

expand:
  input-field: _raw
  delim: "\n"
  multiline:
    - x: "(\n)"
    - y: "(\n\n)"

Output:

{"x":"one 1","y":"one 2\none 3\none 4"}
{"x":"two 1","y":"two 2"}

json​

Type: bool

Default: false

Treat incoming data as already in JSON format

Example

Input:

{"row":"{\"one\":1,\"two\":2}"}

Pipe Language Snippet:

expand:
  input-field: row
  remove: true
  json: true

Output:

{"one":1,"two":2}