Output: s3

Write events to a S3 bucket file

Field Summary

Field Name	Type	Description	Default
retry	Retry	For operations that could potentially fail	-
batch	integer or the 'document'	Maximum number of events in an output batch. If 'document' send on end of document	-
timeout	interval	Interval after which the batch is sent, to keep throughput going	100ms
header	templated text	Put a header line before the batch	-
footer	templated text	Put a header line after the last line of the batch	-
use-document-marker	bool	Enrich the pipe metadata with a document marker (for document handling in batch mode)	false
input-field	field	Use the specified field as the content for the file line	-
bucket-name	string	Bucket Name	-
disable-object-name-guid	bool	Do not append a GUID to the output object name	false
guid-prefix	string	GUID Prefix, will be prepended to the GUID, the default value is "/"	-
guid-suffix	string	GUID Suffix, will be appended to the GUID if specified	-
object-name	string	Object Name	-
object-name-field	field	Field containing the Object Name	-
region	string	Region	-
endpoint	string	S3 Endpoint	-
access-key	string	Access Key ID	-
secret-key	string	Secret Key ID	-
security-token	string	Security Token	-
session-token	string	Session Token	-
preprocessors	OutputPreProcessor	Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified	-
scuba	Scuba	If this Pipe is sending data to Scuba Lite, optionally have the Pipe configure the Scuba Lite instance	-

Fields

retry

Type: Retry

For operations that could potentially fail

Field Name	Type	Description	Default
count	integer	How many attempts to make before declaring failure	-
pause	duration	How long to pause before re-trying	-
forever	bool	Keep trying until success is declared	false

  count

Type: integer

How many attempts to make before declaring failure

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  retry:
    count: 1

Output:

{"_raw":"one two"}

  pause

Type: duration

How long to pause before re-trying

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  retry:
    count: 6
    pause: 10s

Output:

{"_raw":"one two"}

  forever

Type: bool

Default: false

Keep trying until success is declared

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  retry:
    forever: true

Output:

{"_raw":"one two"}

batch

Type: integer or the 'document'

Maximum number of events in an output batch. If 'document' send on end of document

timeout

Type: interval

Default: 100ms

Interval after which the batch is sent, to keep throughput going

header

Type: templated text

Put a header line before the batch

footer

Type: templated text

Put a header line after the last line of the batch

use-document-marker

Type: bool

Default: false

Enrich the pipe metadata with a document marker (for document handling in batch mode)

input-field

Type: field

Use the specified field as the content for the file line

bucket-name

Type: string

Bucket Name

disable-object-name-guid

Type: bool

Default: false

Do not append a GUID to the output object name

guid-prefix

Type: string

GUID Prefix, will be prepended to the GUID, the default value is "/"

guid-suffix

Type: string

GUID Suffix, will be appended to the GUID if specified

object-name

Type: string

Object Name

object-name-field

Type: field

Field containing the Object Name

region

Type: string

Region

endpoint

Type: string

S3 Endpoint

access-key

Type: string

Access Key ID

secret-key

Type: string

Secret Key ID

security-token

Type: string

Security Token

session-token

Type: string

Session Token

preprocessors

Type: OutputPreProcessor

Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified

Field Name	Type	Description	Default
gzip		Gzip the output data	-

  gzip

Gzip the output data

scuba

Type: Scuba

If this Pipe is sending data to Scuba Lite, optionally have the Pipe configure the Scuba Lite instance

Field Name	Type	Description	Default
host	string	The Scuba Lite instance (for example: https://scuba-lite.example.com)	-
insecure	bool	Allow self-signed certificates for Scuba Lite connections	false
token	string	The Scuba Lite API token	-
table-name	string	The Scuba Lite table name	-
file-pattern	string	The Scuba Lite file search pattern	-
columns	ScubaLiteColumns		-

  host

Type: string

The Scuba Lite instance (for example: https://scuba-lite.example.com)

  insecure

Type: bool

Default: false

Allow self-signed certificates for Scuba Lite connections

  token

Type: string

The Scuba Lite API token

  table-name

Type: string

The Scuba Lite table name

  file-pattern

Type: string

The Scuba Lite file search pattern

  columns

Type: ScubaLiteColumns

Field Name	Type	Description	Default
name	string	The name of the column	-
type	ScubaLiteColumnType	The type of the column	-
actor	bool	Specify if the column is an actor	false

  name

Type: string

The name of the column

  type

Type: ScubaLiteColumnType

The type of the column

Field Name	Type	Description	Default
string			-
time	string		-

  string

  time

Type: string

  actor

Type: bool

Default: false

Specify if the column is an actor