Context
Overriding a context
variable causes the redeployment of the affected Pipes, with Context Expansion happening on the Server.
There are default context
variables always available to a Pipe (v3.5.0 upwards):
- Agent name:
{{name}}
- Agent ID:
{{agent}}
- Pipe name:
{{pipe}}
It is common for a Pipe to enrich its data with the name
of the Agent it is running on:
add:
output-fields:
- name: '{{name}}'
Priority
A higher-priority context
overrides a lower-priority context
, with Agent context
being the highest priority and local context
being the lowest:
- Agent
- Tenant
- Pipe
- Tag
- global
- local
Ping Example
Consider a simple Pipe which pings a site and subsequently uses extract
for the ping
times:
name: ping
input:
exec:
command: 'ping -i 4 google.com'
actions:
- extract:
input-field: _raw
remove: true
pattern: '=(\S+) ms$'
output-fields:
- msec
- convert:
- msec: num
- add:
output-fields:
- site: google.com
output:
file:
path: /tmp/{{name}}
The following steps will:
add
the Pipe to the Server and make it available for deploymentcreate an Agent named
Office
usingadd
update
theOffice
Agent by adding theping
Pipe
$> hotrod pipes add --file ping.yml
$> hotrod agents add Office
agent Office has id 7fdfb5e9-bb54-49fc-bdc6-9e9490706cf4
$> hotrod agents update Office --add-pipe ping
[INFO] user=admin adding pipe=ping to target=Office
The Pipe definition has two arbitrary values:
The
site
itself:google.com
The
ping
interval:-i 4
Now add a context
section:
name: ping
context:
site: google.com
interval: 4
input:
exec:
command: 'ping -i {{interval}} {{site}}'
actions:
- extract:
input-field: _raw
remove: true
pattern: '=(\S+) ms$'
output-fields:
- msec
- convert:
- msec: num
- add:
output-fields:
- site: '{{site}}'
output:
file:
path: /tmp/{{name}}
Any context
values in {{}}
will undergo Context Expansion (i.e., the Agent {{name}}
).
Context Expansion occurs on the Server, the rendered (internal representation) version of the Pipe gets sent to the Agent. Context can originate from the Server (global), Agent, or Pipe.
Now update
the Pipe:
$> hotrod pipes update --file ping.yml
Agents affected (Pipe removed/added):
- Office
The resulting output
:
$> tail -f /tmp/Office
{"msec":48.3,"site":"google.com"}
{"msec":49,"site":"google.com"}
{"msec":48.7,"site":"google.com"}
...
Overriding
While this a useful way to avoid repetition, the real power of context
is the way that it can be overridden:
$> hotrod context global site=yahoo.com
[INFO] target Office (0850a5b7-3861-4787-acea-f43de43a7915) deployed
The resulting output
:
$> tail -f /tmp/Office
{"msec":374,"site":"yahoo.com"}
{"msec":306,"site":"yahoo.com"}
{"msec":351,"site":"yahoo.com"}
...
You can now add another Durban
Agent and deploy the ping
Pipe to it:
$> hotrod agents add Durban
agent Durban has id 8c71be6f-b756-428c-8d35-cbe7be0dcfbe
$> hotrod agents update Durban --add-pipe ping
[INFO] user=admin adding pipe=ping to target=Durban
Known as the local context
, it will ping yahoo.com
since the global context
overrides the original Pipe context
.
Confirm the Agent is indeed pinging yahoo.com
:
$> tail -f /tmp/Durban
{"msec":387,"site":"yahoo.com"}
{"msec":357,"site":"yahoo.com"}
{"msec":363,"site":"yahoo.com"}
Global Context
Alternately you may provide context
in the form of a simple YAML file:
$> cat vars.yml
one: 1
two: 2
$> hotrod context global -f vars.yml
[INFO] target Durban (95cb8a32-52f4-44a4-bd8a-2d7c7ea1357b) deployed
[INFO] target Office (0850a5b7-3861-4787-acea-f43de43a7915) deployed
Show the available global
context
:
$> hotrod context global
[INFO] target Durban (95cb8a32-52f4-44a4-bd8a-2d7c7ea1357b) deployed
[INFO] target Office (0850a5b7-3861-4787-acea-f43de43a7915) deployed
:::warn
hotrod pipes run -c vars.yml uptime_enrich.yml
results in an error!
:::
The resulting error:
2023-04-06T11:46:01.303Z ERROR hotrod > bad pipe: unknown field `one`, expected one of `name`, `metadata`, `trace`, `debug`, `owner`, `env`, `tests`, `hidden`, `tags`, `foreach`, `templates`, `files`, `context`, `context-map`, `input`, `actions`, `output`
For instance, there is something special about Durban and we want it to ping
Johannesburg.
$> hotrod agents context Durban site=joburg.co.za
A per-Agent context
overrides global context
as it is more specialized.
We can now create yet another Agent, Cape Town
, and attach the Tag seaside
to it, as well as to Durban
:
$> hotrod agents add 'Cape Town' --tags seaside
agent Cape Town has id 537c7521-9151-4384-b8c3-0103c417958c
$> hotrod agents update 'Cape Town' --add-pipe ping
[INFO] user=admin adding pipe=ping to target=Cape Town
$> hotrod agents update Durban --add-tag seaside
[INFO] user=admin added tag seaside to target=Durban
$> hotrod agents list
name | id | tags | pipes | last seen
-----------+--------------------------------------+-----------+--------+-----------
Office | a5bb18fc-6d5a-43fc-a5fa-7835f14d6713 | | ping |
Durban | f98ea805-4210-4ba2-a898-80148e3aaf5a | seaside | ping |
Cape Town | 537c7521-9151-4384-b8c3-0103c417958c | seaside | ping |
Tag Context
You can attach context
to a tag
:
$> hotrod context tag seaside addr=jhb.co.za
[INFO] target Durban (95cb8a32-52f4-44a4-bd8a-2d7c7ea1357b) deployed
[INFO] target Cape Town (ff18d520-f723-4fc0-8f08-e88a1454b1ca) deployed
That way, Cape Town
gets jhb.co.za
from the tag
context
while Durban
remains joburg.co.za
because Agent context
always wins:
$> hotrod agents show 'Cape Town'
id: ff18d520-f723-4fc0-8f08-e88a1454b1ca
tenant: default
tags: seaside
pipes: ping
# Context:
# site: jhb.co.za
$> hotrod agents show Durban
id: 95cb8a32-52f4-44a4-bd8a-2d7c7ea1357b
tenant: default
tags: seaside
pipes: ping
# Context:
# site: joburg.co.za
tag
context
is a useful way for a whole group of Agents to acquire different settings from the global default.
All instances of a Pipe running on different Agents share a Context — settable with Pipes context
. In this example, we can point ping
at a new site
by setting the site
in the Pipe context
— except in a case where the Agent context
overrides.
So, a higher-priority context
overrides a lower-priority context
, with Agent context
being the highest priority and local context
being the lowest.
Note that administrators with restricted privileges for a group of Agents, cannot set a Pipe, Tag or global context
as this may affect the operation of Pipes running on other Agents.
Multiple Hosts in a Pipe
For instance, if we wanted a Pipe to ping
multiple hosts, one solution would be to make copies of ping.yml
as ping1.yml
, ping2.yml
. This would, however, become a tedious exercise as all the files need to be changed together. The foreach
feature was designed to handle this easily and transparently:
name: pinga
context:
sites:
- google.com
- yahoo.com
foreach:
site: sites
input:
exec:
command: ping -i 4 {{site}}
actions:
- extract:
input-field: _raw
remove: true
pattern: '=(\S+) ms$'
output-fields:
- msec
- convert:
- msec: num
- add:
output-fields:
- site: '{{site}}'
output:
file:
path: /tmp/{{name}}
Again, the default hosts go into the context
as addresses, but as an array. foreach
says that each address should come from this context
array.
The result is that two Pipe services are created on the Agent, one with ping -i 2 google.com
and another with ping -i 2 yahoo.com
. The Pipe services will be called pinga-00
and pinga-01
. Effectively, this creates two copies of the Pipe definition for the Agent.
The array may be nested, using the same dot notation for context
expansions:
context:
my_context:
sites:
- google.com
- yahoo.com
foreach:
site: my_context.sites
Multiple Interfaces in a Pipe
We now want to ping
the same hosts, but through either one of the available interfaces
on the host:
name: pingaif
context:
sites:
- google.com
- yahoo.com
interfaces:
- eth0
- eth1
foreach:
site: sites
interface: interfaces
input:
exec:
command: ping -i 2 {{site}} -I {{interface}}
This generates four Pipe services for the possible combinations (pingaif-01
,
pingaif-02
...):
ping -i 2 google.com -I eth0
ping -i 2 google.com -I eth1
ping -i 2 yahoo.com -I eth0
ping -i 2 yahoo.com -I eth1
Another form of foreach
allows you to go over nested arrays in the following
context
:
name: hosttrio
context:
hosts:
- spoof: 192.168.66.38
agents:
- 1.1.1.1
- 2.2.2.2
port: 22
- spoof: 192.168.66.39
agents:
- 3.3.3.3
- 4.4.4.4
port: 22
foreach:
host: hosts[agents]
input:
exec:
command: echo {{host.spoof}} {{host.agents}}:{{host.port}}
raw: true
output:
write: console
# Output:
# 192.168.66.38 1.1.1.1:22
# 192.168.66.38 2.2.2.2:22
# 192.168.66.39 3.3.3.3:22
# 192.168.66.39 4.4.4.4:22
Aliases
An alias refers to an input
, output
, or a set of actions
.
Consider the following enrich.alias
:
name: enrich
actions:
- add:
output-fields:
- name: '{{name}}'
- pipe: '{{pipe}}'
- agent: '{{agent}}'
- tenant: '{{tenant}}'
- time:
output-field: '@timestamp'
It looks very much like a Pipe definition, but it does not need to have all three of the components of a Pipe (input
, actions
and output
).
This is because it is substituted into a Pipe definition, note the alias
action
:
name: uptime_enrich
input:
exec:
command: uptime
interval: 2s
actions:
- extract:
input-field: _raw
pattern: 'load average: (\S+), (\S+), (\S+)'
output-fields: [m1, m5, m15]
- alias: enrich
output:
print: STDOUT
All the site-specific enrichments can be wrapped up in an alias
definition and reused in any Pipe. Update aliases
with (currently, broken, workaround by including everything in the Pipe definition):
$> hotrod aliases update enrich.alias
Aliases can also provide predefined, reusable, definitions for input
and output
— which is useful for testing.
Then do:
$> hotrod pipes add --file uptime_enrich.yml
$> hotrod agents update Office --add-pipe ping
[INFO] user=admin adding pipe=uptime_enrich to target=Office
This Pipe, is rendered on the Server and deployed to the Agent (Server/Agent
context). Inspect the output
on the Office
Agent host:
$> journalctl -fu hotrod-agent
Apr 05 21:46:43 rocky9.ephemeric.lan hotrod-agent[296602]: {"_raw":" 21:46:43 up 5 days, 2:16, 1 user, load average: 1.00, 1.00, 1.00","m1":"1.00","m5":"1.00","m15":"1.00","name":"Office","tenant":"default","pipe":"uptime_enrich","agent":"4255b0e1-3ea5-4dcf-b6f5-9ead3e91be5d","@timestamp":"2023-04-05T19:46:43.234Z"}
The Pipe can run from the CLI, explicitly specifying the `context` file (see `hotrod pipes run --help`):
:::note
Context Expansion variables like `{{name}}`, `{{tenant}}`, `{{agent}}`, and `{{pipe}}` etc., are _not_ available in the CLI Pipe runtime as they are rendered on the Server.
:::
```sh
$> hotrod pipes run --context-files enrich.alias --file uptime_enrich.yml
{"_raw":" 23:06:33 up 5 days, 3:36, 1 user, load average: 1.13, 1.06, 1.02","name":"{{name}}","pipe":"{{pipe}}","agent":"{{agent}}","tenant":"{{tenant}}","@timestamp":"2023-04-05T21:06:33.312Z"}
...
The CLI Pipe runtime does not merge, but replaces when an alias
file is used for --context-files
. Note the above output: the extract
action has been replaced with the add
and time
actions
(missing m1
, m5
, and m15
fields).
When running Pipes from the CLI, there is no Server/Agent context
, therefore context
must be defined in the Pipe definition or in a context
file.