Two pipelines for the same file

[hding]

My log file contains log lines of two types: "NORMAL" and "ERROR". Each line starts with its type.

The logs are stored in the same directory.

I have built two pipelines: "pipeline_1" and "pipeline_2".

How do I configure filebeat.prospectors and output.elasticsearch? Will the following work?

filebeat.prospectors:
- paths: ["/var/log/*.log"]
  include_lines: ["^NORMAL"]
  fields:
    type: "normal"
- paths: ["/var/log/*.log"]
  include_lines: ["^ERROR"]
  fields:
    type: "error"

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  pipelines:
    - pipeline: pipeline_1
      when.equals:
        fields.type: "normal"
    - pipeline: pipeline_2
      when.equals:
        fields.type: "error"

[steffens (Steffen Siering)]

Do not have any two prospectors report the same file. The files state/offset is currently stored global and these two prospectors will interleave.

You can use the when.equals condition with match or regex, to check a log message starts with any of these, and then ship do to the respective pipeline. Are the logs that different, that you really need a different pipeline? e.g. in on can configure multiple patterns in the grok processor.

[hding]

Thanks a lot, Steffen.

By saying use when.equals with match and regex, do you mean like this:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
    pipelines:
      - pipeline: pipeline_1
        when.equals:
          regex: ["^NORMAL"]
      - pipeline: pipeline_2
        when.equals:
          regex: ["^ERROR"]

The lines in one file can be indeed very different. One line type can have 80 fields while another can have only 7 fields. And there are over 20 different types... The first few letters in a line indicates its type.

Right now I am using ingest node instead of Logstash. Will using multi-patterns in grok processor slow down the ingestion speed? Does grok try to match the patterns one by one until one is matched? Should I actually consider using Logstash with Filebeat?

[hding]

I tried the following, but it doesn't work...

  # Configure indices
  indices:
    - index: "my_index_1"
      when.equals:
        match: ["^NORMAL"]
    - index: "my_index_2"
      when.equals:
        match: ["^ERROR"]

  # Configure pipelines
  # pipeline: "my_pipeline_r"
  pipelines:
    - pipeline: "my_pipeline_1"
      when.equals:
        match: ["^NORMAL"]
    - pipeline: "my_pipeline_2"
      when.equals:
        match: ["^ERROR"]

[steffens (Steffen Siering)]

Right now I am using ingest node instead of Logstash. Will using multi-patterns in grok processor slow down the ingestion speed?

No idea how grok is executed in Elasticsearch Ingest Node. But I'd assume it's one by one. Aynway, you having just 2 patterns shouldn't be much of a concern.

Which filebeat version (match was introduced somewhat later, use regex )? Any logs?

The syntax for the when clause is when.<condition>.<full field name>: <value to compare with>. You have used when.<condition>.<condition>: <value to compare with>. Both, equals and match are conditions.

Here is an example on pipelines. And documentation on conditions.

Try:

pipelines:
- pipeline: "my_pipeline_1"
  when.regexp.message: '^NORMAL'
- pipeline: "my_pipeline_2"
  when.regexp.message: '^ERROR''

Do not use double quotes for regular expressions in YAML.

[hding]

Thank you so much for your help, Steffen! It works like a magic!

I'm using Filebeat 5.4.1 and ElasticSearch 5.4.1.