Scripting in Pipes

script, like add is primarily used to add new fields to events (enrichment). The differentiator from add is that script can calculate and conditionally add fields.

These expressions are in Lua format, with conditions such as a > 2 and b < 1 and expressions such as 2 * a - 1. All event fields are available in these expressions.

If the data is not "flat", nested fields are accessed with a . and arrays are indexed with [].

If our event is {"a":{"b":1},"c":[10,20,30]}, then the value of 1 can be accessed with a.b. Array values are accessed starting at one, so c[1] == 10.

note

For this to work field names must be valid Lua variables. Fields must start with a letter, and contain only letters, digits, and underscores. As this restriction also applies to extract, we recommend using this format for all field names.

Regular Lua keywords such as function or end can be used as field names — with the exception of true, false, nil, or, and and.

let versus set

Provide a list of variable-expression pairs for let and constants for set. Context Expansion is always available.

# Input: {"a":10}

- script:
    let:
    - c: 2*a - 1
    set:
    - site: '{{name}}'

# Output: {"a":10,"c":19,"site":"NAME"}

script set performs a similar function to add output-fields. However, in the case of add you can add structured data like arrays and objects.

The primary reason for using set is due to the complexity of setting constants using let. As an example, notice the double quotes in name: '"hello"'.

Both script and add do not override existing fields by default. This can be forced using override: true

There may be a condition field — the condition must be true before any fields are written.

The "pseudo-field" _E is special and refers to the entire event. _E will make a copy of the event and put it into the res field:

- script:
    let:
    - res: _E

Extra Functions

• round(x) returns the nearest integer to a floating point number, like round(tmillis/1000), useful for converting bytes to kB and milliseconds since epoch to seconds since epoch
• count() is the count since Pipe start
• random(n) is a random integer between 1 and n
• pick_random(...) returns one of multiple arguments randomly
• sum(acc,val,cond) and avg(acc,val,cond) are the running sum and average of value val
• sec_s() will return seconds since epoch, sec_ms() is milliseconds since epoch
• cidr(addr, spec) will match an IPv4 network address against a CIDR specification like '10.0.0.0/24'
• ip2asn uses the Team Cymru services to match IP addresses to a domain owner
• cond(condition, value1, value2) will return value1 if a condition is true, and if not true returns value2
• condn(...) is similar to cond but for multiple conditions and values
• array(...) transforms its arguments into an array
• len(v) is the length of array or number of characters in text
• map(...) makes an object
• hashes:
  • md5(txt)
  • sha1(txt)
  • sha256(txt)
  • sha512(txt)
• uuid() returns a unique identifier each time
• encrypt(txt,key) encrypts text using AES-128-CBC with a key and encodes as Base64
• decrypt(txt,key) decrypt result of encrypt using the same key
• emit(v) writes out data directly

danger

Any field with one of the above names will overwrite the function.

Examples

The first argument of sum is the 'accumulator'. This can be any name (that isn't a field) and keeps the count. This is necessary because there may be more than one field being summed.

note

filter is the only other place where Lua expressions occur.

- script:
    let:
    - counter: count()
    - sumi: sum("s",counter)
- filter:
    condition: counter % 5 == 0

# Output:
  # {"counter":5,"sumi":15}
  # {"counter":10,"sumi":55}
  # {"counter":15,"sumi":120}
  # {"counter":20,"sumi":210}
  # ...

sum has an extra argument — if present, it will collect while it is true:

# Input:
  # {"val":10,"ok":true}
  # {"val":10,"ok":true}
  # {"val":5,"ok":false}
  # {"val":50,"ok":false}    
  # {"val":10,"ok":true}
  # {"val":10,"ok":true}

  - script:
      let:
      - s: sum("s",val,ok)

# Output:
  # {"val":10,"ok":true,"s":10}
  # {"val":10,"ok":true,"s":20}
  # {"val":5,"ok":false,"s":5}
  # {"val":50,"ok":false,"s":50}
  # {"val":10,"ok":true,"s":10}
  # {"val":10,"ok":true,"s":20}

It’s convenient to use map and array to build "deep" objects:

# Input:
  # {"a":1,"b":2}

  - script:
      let:
      - res: |
          map (
            "one",a+b,
            "two",a-b,
            "three",a*b,
            "four",array(a,b)
          )
# Output:
  # {"a":1,"b":2,"res":{"four":[1,2],"three":2,"two":-1,"one":3}}

Lua objects do not preserve order.

condn is useful for more complicated conditions and simple lookups:

# Input:
  # {"a":1}
  # {"a":2}
  # {"a":3}

  - script:
      let:
      - size: |
          condn(
            a < 2, "small",
            a == 2,"medium",
            a > 2, "large"
          )
# Output:
  # {"a":1,"size":"small"}
  # {"a":2,"size":"medium"}
  # {"a":3,"size":"large"}            

Useful Standard Lua Functions

Processing Text

• string.upper(txt) converts text to upper-case
• string.lower(txt) converts text to lower-case
• string.sub(txt,istart,iend) is a substring of text
• string.find(txt,patt) finds matching patterns
• table.concat(txt,sep) joins an array as text

name: upper_lua

input:
  text: '{"msg":"hello dolly","arr":[1,2,3],"num":3.1423}'

actions:
- script:
    overwrite: true
    let:
    - msg: string.upper(msg)
    - first: string.sub(msg,1,4)
    - n: string.len(msg)
    - arr: table.concat(arr," ")
    - num: string.format("%.02f",num)

output:
  write: console

# Output:
  # {"msg":"HELLO DOLLY","arr":"1.0 2.0 3.0","num":"3.14","first":"HELL","n":11}

Advanced Lua users may wish to use — msg: 'msg:upper()'. However, we need to use quotes because YAML is confused by values that contain colons.

It is possible to join (concatenate) strings together using the .. operator.

# Input:
  # {"first":"jonas","second":"whale"}

actions:
- script:
    let:
    - full_name: first .. " " .. second

# Output:
  # {"first":"jonas","second":"whale","full_name":"jonas whale"}    

Mathematical Functions

• math.ceil(x) is the nearest greater integer e.g., 1.4 -> 2
• math.floor(x) is the nearest smaller integer e.g., 1.4 -> 1
• math.min(...) is the smallest of multiple values
• math.max(...) is the largest of multiple values
• math.log(x,b) is the log of x with base b
• math.exp(x) exponential
• math.sin, math.cos etc., are trigonometric functions (in radians)

Visit the Lua manual for the full list of available functions.

note

Some functionality has been removed for safe sandboxing. This is why the global functions require, dofile, and load are not present.

Extending script using Lua

If the file init.lua is in the Pipe directory, it will be loaded into the Lua state. As a result, any global functions or variables become available:

-- init.lua
    function every(n)
       return count() % n == 0
    end

Following this, the counter example can be improved as seen below:

- script:
    let:
    - counter: count()
    - sumi: sum("s",counter)

- filter:
    condition: every(5)

init.lua will be loaded automatically, but it must be copied alongside the Pipe. A complete Pipe may appear as:

name: sumi

files:
- init.lua

input:
    exec:
        command: echo foo
        interval: 100ms

actions:
- remove: [_raw]

- script:
    let:
    - counter: count()
    - sumi: sum("s",counter)

- filter:
    condition: every(5)

output:
    write: console

Let's take this a step further and unpack the result of a Prometheus query. This involves taking a structured JSON document and creating events for each retrieved data point — otherwise known as expanding:

{
    "status":"success",
    "data":{
        "resultType":"matrix",
        "result":[
            {
                "metric":{
                    "__name__":"probe_success",
                    "instance":"178.62.49.144:20000",
                    "job":"APP-F4ZASD-JD-STE-RU-3033"},
                    "values":[[1571997720,"0"],[1571997780,"0"],...]
                }
            },
            ...
        ]
    }
}

init.lua contains a function that passes the entire document and uses the emit function to write out events directly:

function unpack_prom(result)
    for _,res in ipairs(result) do
        local t = res.metric
        for _,pair in ipairs(res.values) do
            t.time = pair[1]
            t.value = pair[2]
            emit(t)
        end        
    end
    return 'ok'    
end

Exercised by this Pipe:

- script:
    let:
    - a: unpack_prom(data.result)
- remove: [status,data]
# Output:
  # {"instance":"178.62.49.144:20000","__name__":"probe_success","time":1571997720,"value":"0","job":"APP-F4ZASD-JD-STE-RU-3033"}
  # ...

In this case, we are interested in the side effect of the function.