Is it possible to have different logstash parse patterns for diffrent logs files linked to the same index?

I would like to ask if it is possible to have different logstash parse patterns for different log files to the same index,? What i want to achieve is to have different parse patterns mapping to the same attributes and still put that data in the same index

This is certanily possible. How are your inputs configured? The best way is to add a tag within the input plugin to flag it as a particular type of log. Then you can use an If statement within your filter plugin to handle events of different tags in different ways.

Thank you for your feedback.
My filebeat configuration is:

name: filebeat

filebeat.prospectors:

  • type: log
    multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2}|[a-zA-Z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
    multiline.negate: true
    multiline.match: after

    paths:

    • /var/*.log

    exclude_files: ['.yml']
    document_type: application_log

    fields_under_root: true

    fields:
    fields.datasource_name: java-application-logs
    environment: {{ envi }}

  • type: log
    multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2}|[a-zA-Z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
    multiline.negate: true
    multiline.match: after

    paths:

    • /var/opt/tomcat/*.log

    exclude_files: ['.yml']
    document_type: tomcat_log

    fields_under_root: true

    fields:
    fields.datasource_name: tomcat-logs
    environment: {{ envi }}

output:
logstash:
hosts: ["{{ logstash_host }}:{{port}}" ]

logging:
level: debug

But i don't understand what i must do on the logstash configuration ?

Okay you need to add the tags within Filebeat in this instance.

https://www.elastic.co/guide/en/beats/filebeat/master/add-tags.html

Try something like this:

type: log
multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2}|[a-zA-Z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
multiline.negate: true
multiline.match: after

paths: /var/*.log

exclude_files: ['.yml']
document_type: application_log

fields_under_root: true
environment: {{ envi }}

processors:
 - add_tags:
    tags: [java-application-logs]

Perhaps remove the field data_source name. Then you can use, within Logstash:

filter {
  if "java-application-logs" in [tags] {
    ... (processing java logs)
  }
  ....
}

Hope this helps.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.