collate

Collate events by time or count.

The original goal of this filter was to merge the logs from different sources by the time of log, for example, in real-time log collection, logs can be collated by amount of 3000 logs or can be collated in 30 seconds.

The config looks like this:

filter {
  collate {
    size => 3000
    interval => "30s"
    order => "ascending"
  }
}

 

Synopsis

This plugin supports the following configuration options:

Required configuration options:

collate {
}

Available configuration options:

Setting Input typeRequiredDefault value

add_field

hash

No

{}

add_tag

array

No

[]

count

number

No

1000

interval

string

No

"1m"

order

string, one of ["ascending", "descending"]

No

"ascending"

periodic_flush

boolean

No

false

remove_field

array

No

[]

remove_tag

array

No

[]

Details

 

add_field

  • Value type is hash
  • Default value is {}

If this filter is successful, add any arbitrary fields to this event. Field names can be dynamic and include parts of the event using the %{field}.

Example:

filter {
  collate {
    add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
  }
}
# You can also add multiple fields at once:
filter {
  collate {
    add_field => {
      "foo_%{somefield}" => "Hello world, from %{host}"
      "new_field" => "new_static_value"
    }
  }
}

If the event has field "somefield" == "hello" this filter, on success, would add field foo_hello if it is present, with the value above and the %{host} piece replaced with that value from the event. The second example would also add a hardcoded field.

add_tag

  • Value type is array
  • Default value is []

If this filter is successful, add arbitrary tags to the event. Tags can be dynamic and include parts of the event using the %{field} syntax.

Example:

filter {
  collate {
    add_tag => [ "foo_%{somefield}" ]
  }
}
# You can also add multiple tags at once:
filter {
  collate {
    add_tag => [ "foo_%{somefield}", "taggedy_tag"]
  }
}

If the event has field "somefield" == "hello" this filter, on success, would add a tag foo_hello (and the second example would of course add a taggedy_tag tag).

count

  • Value type is number
  • Default value is 1000

How many logs should be collated.

exclude_tags (DEPRECATED)

  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is array
  • Default value is []

Only handle events without all/any (controlled by exclude_any config option) of these tags. Optional.

interval

  • Value type is string
  • Default value is "1m"

The interval is the time window which how long the logs should be collated. (default 1m)

order

  • Value can be any of: ascending, descending
  • Default value is "ascending"

The order collated events should appear in.

periodic_flush

  • Value type is boolean
  • Default value is false

Call the filter flush method at regular interval. Optional.

remove_field

  • Value type is array
  • Default value is []

If this filter is successful, remove arbitrary fields from this event. Example:

filter {
  collate {
    remove_field => [ "foo_%{somefield}" ]
  }
}
# You can also remove multiple fields at once:
filter {
  collate {
    remove_field => [ "foo_%{somefield}", "my_extraneous_field" ]
  }
}

If the event has field "somefield" == "hello" this filter, on success, would remove the field with name foo_hello if it is present. The second example would remove an additional, non-dynamic field.

remove_tag

  • Value type is array
  • Default value is []

If this filter is successful, remove arbitrary tags from the event. Tags can be dynamic and include parts of the event using the %{field} syntax.

Example:

filter {
  collate {
    remove_tag => [ "foo_%{somefield}" ]
  }
}
# You can also remove multiple tags at once:
filter {
  collate {
    remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"]
  }
}

If the event has field "somefield" == "hello" this filter, on success, would remove the tag foo_hello if it is present. The second example would remove a sad, unwanted tag as well.

tags (DEPRECATED)

  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is array
  • Default value is []

Only handle events with all/any (controlled by include_any config option) of these tags. Optional.

type (DEPRECATED)

  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is string
  • Default value is ""

Note that all of the specified routing options (type,tags,exclude_tags,include_fields, exclude_fields) must be met in order for the event to be handled by the filter. The type to act on. If a type is given, then this filter will only act on messages with the same type. See any input plugin’s "type" attribute for more. Optional.