Executing Commands
/bin/bash is used on Unix-like systems and cmd.exe is used on Windows for
executing Pipe commands that will:
generate data for a Pipe (as input-exec)
process data inside a Pipe (as action-exec)
send data to the desired destination (as output-exec)
The Pipe directory is contained in $PWD (Present Working Directory). As a rule, do not expect any files written to this directory to survive a Pipe update, instead use absolute file paths. This directory is also in the $PATH environment variable.
Input Exec
The command is executed by /bin/bash, with all line feeds removed by default. It is possible to arrange a complicated command like the one below without needing backslashes:
input:
exec:
command: |
complex-command
--first-flag 1
--second-flag 2
You can disable this removal using no-strip-linefeeds: true.
By default, each line of output is considered a fresh event and quoted:
input:
exec:
command: echo ay; echo bee
# Output: {"_raw":"ay"}, {"_raw":"bee"}
Use raw: true to disable quoting for output that is already in JSON or that requires further mangling with action-raw.
input:
exec:
no-strip-linefeeds: true # (or use semicolons)
command: |
echo '{"msg":"hello"}'
echo '{"msg":"dolly"}'
raw: true
# Output: {"msg":"hello"}, {"msg":"dolly"}
By default, the command is run once, which is appropriate for commands such as ping that continuously create output with a specified interval.
Scheduling Commands
To run at regular intervals — the easiest way is to specify a value for interval.
Commands can be scheduled to run at regular intervals by specifying an interval value.
Scheduled commands give you the option to process the entire output from each invocation as a single event using ignore-line-breaks: true:
input:
exec:
command: echo ay; echo bee
ignore-line-breaks: true
interval: 2s
# Output: {"_raw":"ay\nbee"}
count: 1 allows you to specify a command to execute precisely once. This may be needed as ignore-line-breaks currently only works with scheduled inputs.
An event that corresponds to multiple lines of output (multiline event), requires that all data be consolidated as a single piece of text, e.g.:
$> openssl s_client -connect google.com:443 </dev/null 2>/dev/null | openssl x509 -fingerprint -dates -nocert
SHA1 Fingerprint=95:3A:FF:D9:19:64:D9:09:40:8D:EE:DA:40:48:0E:FF:5E:DA:52:8C
notBefore=Sep 3 06:36:33 2020 GMT
notAfter=Nov 26 06:36:33 2020 GMT
Here we can use expand.key-value, note delim: '\n', which is a newline.
name: openssl_client
input:
exec:
command: openssl s_client -connect google.com:443 </dev/null 2>/dev/null | openssl x509 -fingerprint -dates -nocert
ignore-linebreaks: true
interval: 1s
count: 1
actions:
# Expand key values using line-feed as a delimiter.
- expand:
input-field: _raw
remove: true
delim: '\n'
key-value:
key-value-delim: '='
output:
write: console
# Output:
# {"SHA1 Fingerprint":"95:3A:FF:D9:19:64:D9:09:40:8D:EE:DA:40:48:0E:FF:5E:DA:52:8C","notBefore":"Sep 3 06:36:33 2020 GMT","notAfter":"Nov 26 06:36:33 2020 GMT"}
Handling Errors
The previous examples have illustrated positive situations where the command has executed successfully. The result field allows for standard output (stdout), standard error (stderr), and status to be captured as custom fields:
input:
exec:
command: echo hello && foo
result:
stdout-field: out
stderr-field: err
status-field: status
# Output: {"err":"sh: 1: foo: not found\n","out":"hello\n","status":127}
stdout-field: _raw is the default in this case.
In this example, netcat (see man netcat) will try to send hi there to TCP port 3030. If unsuccessful, due to network connectivity, pause: 5s before attempting a retry. Not pausing will result in too many Connection refused errors:
input:
exec:
command: echo 'hi there' | netcat -N -v 127.0.0.1 3030
retry:
count: 3
pause: 5s
$> hotrod pipes run -f netcat.yml
netcat: connect to 127.0.0.1 port 3030 (tcp) failed: Connection refused
[ERROR] exec: failed Exited(1) input-exec step 0
LINE:
netcat: connect to 127.0.0.1 port 3030 (tcp) failed: Connection refused
[ERROR] exec: failed Exited(1) input-exec step 0
LINE:
{"_raw":"","time":"2023-03-30T08:50:11.029Z"}
Trying indefinitely is not recommended for a scheduled command. Using count: 3 instead of forever: true will result in only 3 retry attempts.
Action Exec
It’s useful to invoke a command as a series of actions. You can do so using action.exec:
- run a command for its side effects
- execute commands conditionally
- merge the output of another command into an event
By default, data passes through unmodified when using action.exec:
# Input: {"msg":"hello"}
actions:
- exec:
command: echo ${msg} >temp.txt
Here, as with any other action, we can use field expansions:
$> hotrod pipes run -f exec2.yml
{"msg":"hello"}
$> cat temp.txt
{"msg":"hello"}
Note that the event itself is passed in as the input of the command:
actions:
- exec:
command: cat >temp.txt
With previous input, temp.txt contains {"msg":"hello"} as expected.
Specifying input-field enables us to choose what we want to pass to a command input:
actions:
- exec:
input-field: msg
command: cat >temp.txt
Afterwards, temp.txt contains hello.
Do not write to any Pipe directories. Pipe directories are managed, meaning that there is no guarantee that any such files will still be available after the Pipe is re-started.
action-exec does not prompt for a missing input-field by default. This is useful because it allows for conditional execution of commands.
# Input: {"msg":"hello"}
actions:
- exec:
input-field: msg
command: netcat -N -v 127.0.0.1 3030
# Output: {"greeting":"bye"}
A TCP server listening on the port in question will receive hello when msg is an existing field. Note that it isn't a requirement for input-field to be a string — we can simply pass the entire event.
The output of a command can merge into a current event using result, which works as in input-exec:
# Input: {"msg":"hello"}
actions:
- exec:
command: whoami
result:
stdout-field: who
# Output: {"msg":"hello","who":"steve"}
This is a simple example that showcases the power of using this approach to combine various views of a system as one event.
Below is a Pipe used to monitor load average and CPU usage extracted from the uptime and mpstat commands respectively:
name: mpstat
input:
exec:
command: mpstat | tail -n 1
interval: 5s
actions:
- exec:
command: uptime
result:
stdout-field: uptime
# Extract and convert mpstat fields.
- expand:
input-field: _raw
remove: true
delim: ' '
csv:
fields:
- time: str
- CPU: str
- usr: num
- nice: num
- sys: num
- iowait: num
- irq: num
- soft: num
- steal: num
- guest: num
- gnice: num
- idle: num
# Likewise for uptime.
- extract:
input-field: uptime
remove: true
pattern: 'load average: ([^,]+),'
output-fields: [avg1]
- convert:
- avg1: num
output:
write: console
Output Exec
We recommend using the most specific output possible when executing a command. Another option is to use output-exec.
Field Expansion can be used to pass all or part of an event as a command input, as with action-exec.
A restriction applies — once the command is started, all events thereafter are written to its input by default:
output:
exec:
command: cat
While in this case it makes more sense to use write: console, this example illustrates the long-lived purpose of command and that it serves as the end of a pipeline.
As the data is passed through directly in this instance, Field Expansion cannot be used. Should you wish to use Field Expansion, command is implicitly run for each event.
output:
exec:
command: echo ${msg} >> /var/log/mypipe.log
Use streaming: false to explicitly force command to run once per event. However, streaming cannot be forced if any Field Expansion is present in the data.
As is the case with action-exec, you can provide input-field. In previous versions, input-field implied streaming: false but in this version, it is possible to override and force streaming by using streaming: true.
Using retry allows for the modification of the default behaviour (3 times with a 300ms pause).