Input: files
Read from multiple files, in order of creation
Example
Pipe Language Snippet:
files:
path: /var/log/dpkg.*
Field Summary
| Field Name | Type | Description | Default |
|---|---|---|---|
| path | path | A path with wildcards identifying files to be read | - |
| data-dir | path | The directory where checkpoints are written | the platform-specific data directory for the user executing the pipe |
| encoding | encoding | The text encoding for the monitored files | UTF-8 |
| include | array of paths | Any other paths to be included | - |
| exclude | array of paths | Paths to be excluded | - |
| ignore-older-than | integer | Ignore files older than this (epoch time in seconds) | - |
| glob-minimum-cooldown | integer | Pause after scanning for new files (in milliseconds) | 300ms |
| stop-reading-after | bool | Do not wait for new files | false |
| ignore-line-breaks | bool | Consume each file as one event | false |
| remove-after | integer | Remove files that are done reading after this period (seconds) | - |
| fingerprinting | Fingerprinter | File fingerprinting strategy | DevInode |
| oldest-first | bool | Default is youngest first | false |
| file-path-field | field | Fill this field with the field we are currently reading from | - |
| file-basename | bool | The file-path-field path will be the basename (e.g. '/path/frodo.txt' becomes just 'frodo') | false |
| start-at-beginning | bool | Ignore stored checkpoints and re-read all specified files | false |
| raw | bool | Disable 'quoting' of lines with _raw | false |
| max-read-size | integer | Will not attempt to read files larger than this | 100MB |
| max-line-size | integer | Will not attempt to read lines longer than this | 1MB |
Fields
path
Type: path
A path with wildcards identifying files to be read
data-dir
Type: path
Default: the platform-specific data directory for the user executing the pipe
The directory where checkpoints are written
encoding
Type: encoding
Default: UTF-8
The text encoding for the monitored files
include
Type: array of paths
Any other paths to be included
exclude
Type: array of paths
Paths to be excluded
ignore-older-than
Type: integer
Ignore files older than this (epoch time in seconds)
glob-minimum-cooldown
Type: integer
Alias: timeout
Default: 300ms
Pause after scanning for new files (in milliseconds)
stop-reading-after
Type: bool
Default: false
Do not wait for new files
ignore-line-breaks
Type: bool
Default: false
Consume each file as one event
remove-after
Type: integer
Remove files that are done reading after this period (seconds)
fingerprinting
Type: Fingerprinter
Default: DevInode
File fingerprinting strategy
| Field Name | Type | Description | Default |
|---|---|---|---|
| checksum | FingerprinterChecksum | Fingerprint strategy is set to a checksum of the start of the file determined by the bytes that are skipped (ignored-header-bytes) and then the fingerprint size in bytes | - |
| first-line-checksum | FingerprinterFirstLineChecksum | Fingerprint strategy that uses the first line of the file to detect if a file has changed | - |
| dev-inode | FingerprinterDevInode | Fingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated | - |
checksum
Type: FingerprinterChecksum
Fingerprint strategy is set to a checksum of the start of the file determined by the bytes that are skipped (ignored-header-bytes) and then the fingerprint size in bytes
| Field Name | Type | Description | Default |
|---|---|---|---|
| fingerprint-bytes | integer | The number of bytes to use for the fingerprint, starts after ignore-header-bytes if specified | - |
| ignored-header-bytes | integer | The number of bytes from the start of the file to ignore before starting the fingerprint | - |
fingerprint-bytes
Type: integer
Alias: fingerprint-bytes
The number of bytes to use for the fingerprint, starts after ignore-header-bytes if specified
ignored-header-bytes
Type: integer
Alias: ignored-header-bytes
The number of bytes from the start of the file to ignore before starting the fingerprint
first-line-checksum
Type: FingerprinterFirstLineChecksum
Fingerprint strategy that uses the first line of the file to detect if a file has changed
| Field Name | Type | Description | Default |
|---|---|---|---|
| max-line-length | integer | The maximum number of bytes to read from the first line of the file to calculate the fingerprint | - |
max-line-length
Type: integer
Alias: max-line-length
The maximum number of bytes to read from the first line of the file to calculate the fingerprint
dev-inode
Type: FingerprinterDevInode
Fingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated
oldest-first
Type: bool
Default: false
Default is youngest first
file-path-field
Type: field
Fill this field with the field we are currently reading from
file-basename
Type: bool
Default: false
The file-path-field path will be the basename (e.g. '/path/frodo.txt' becomes just 'frodo')
start-at-beginning
Type: bool
Default: false
Ignore stored checkpoints and re-read all specified files
raw
Type: bool
Default: false
Disable 'quoting' of lines with _raw
max-read-size
Type: integer
Default: 100MB
Will not attempt to read files larger than this
max-line-size
Type: integer
Default: 1MB
Will not attempt to read lines longer than this