Is it possible to have different logstash parse patterns for diffrent logs files linked to the same index?

Elastic Stack Logstash
1

I would like to ask if it is possible to have different logstash parse patterns for different log files to the same index,? What i want to achieve is to have different parse patterns mapping to the same attributes and still put that data in the same index

2

This is certanily possible. How are your inputs configured? The best way is to add a tag within the input plugin to flag it as a particular type of log. Then you can use an If statement within your filter plugin to handle events of different tags in different ways.

3

Thank you for your feedback.
My filebeat configuration is:

name: filebeat

filebeat.prospectors:

  • type: log
    multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2}|[a-zA-Z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
    multiline.negate: true
    multiline.match: after

    paths:

    • /var/*.log

    exclude_files: ['.yml']
    document_type: application_log

    fields_under_root: true

    fields:
    fields.datasource_name: java-application-logs
    environment: {{ envi }}

  • type: log
    multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2}|[a-zA-Z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
    multiline.negate: true
    multiline.match: after

    paths:

    • /var/opt/tomcat/*.log

    exclude_files: ['.yml']
    document_type: tomcat_log

    fields_under_root: true

    fields:
    fields.datasource_name: tomcat-logs
    environment: {{ envi }}

output:
logstash:
hosts: ["{{ logstash_host }}:{{port}}" ]

logging:
level: debug

But i don't understand what i must do on the logstash configuration ?

5

Okay you need to add the tags within Filebeat in this instance.

https://www.elastic.co/guide/en/beats/filebeat/master/add-tags.html

Try something like this:

type: log
multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2}|[a-zA-Z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
multiline.negate: true
multiline.match: after

paths: /var/*.log

exclude_files: ['.yml']
document_type: application_log

fields_under_root: true
environment: {{ envi }}

processors:
 - add_tags:
    tags: [java-application-logs]

Perhaps remove the field data_source name. Then you can use, within Logstash:

filter {
  if "java-application-logs" in [tags] {
    ... (processing java logs)
  }
  ....
}

Hope this helps.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.