Can't separated elasticsearch index using logstash

Elastic Stack Logstash
1

Hello Guys,

I'm trying to create separated indexes using logstash 6.7, basically I have a server running filebeat which sends two logs to a logstash which sends to a elasticsearch, below you may find my config files.

filebeat.yml
#=========================== Filebeat inputs =============================

filebeat.prospectors:

  • type: log
    paths:

    • /logs/access.*log
    • /logs/access.*log
    • /logs/access.*log
    • /logs/access.*log
    • /logs/access.*log
    • /logs/access.*log
      fields:
      log_type: rtdm-access
      jvm: "SASServer 7"
      team: "Middleware"
      system: "SAS - RTDM"
      env: "Production"
      role: "Engine"
      fields_under_root: true
      scan_frequency: 5s

  • type: log
    paths:

    • /Logs/tomcatgc.log.?.current
    • /Logs/tomcatgc.log.?.current
    • /Logs/tomcatgc.log.?.current
    • /Logs/tomcatgc.log.?.current
    • /Logs/tomcatgc.log.?.current
    • /Logs/tomcatgc.log.?.current
      fields:
      log_type: rtdm-gc
      jvm: "SASServer 7"
      team: "Middleware"
      system: "SAS - RTDM"
      env: "Production"
      role: "Engine"
      fields_under_root: true
      scan_frequency: 15s

#============================= Filebeat modules ===============================

filebeat.config.modules:

Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.name: "mdw_filebeat"
setup.template.pattern: "mdw_filebeat_*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["logstash-server1:9500" , "logstash-server2:9500"]
loadbalance: true

#================================ Processors =====================================

Configure processors to enhance or manipulate events generated by the beat.

processors:

  • add_host_metadata: ~
  • add_cloud_metadata: ~

logstash config file

output {
if [log_type] == "rtdm-access" {
elasticsearch {
hosts => ["elasticsearch-server1:9200",elasticsearch-server2:9200]
index => "access-rtdm"
}
}
else if [log_type] == "rtdm-gc" {
elasticsearch {
hosts => ["elasticsearch-server1:9200",elasticsearch-server2:9200]
index => "gc-rtdm"
}
}
#only for debugging
stdout { codec => rubydebug }

}

Note.: If I remove the IF/Else and configure to create a standard index this it creates, but unfortunately this is not what I need, can anyone please explain what I'm doing wrong?

Filebeat / Logstash and ElasticSearch version is 6.7

thanks in advance,

2

I would recommend you look at the events being output to stdout. I would expect the [log_type] field to actually be stored as [fields][log_type]. You may therefore need to change your conditionals to reflect this.

3

Christian_Dahlqvist,

Thanks a lot mate, by modifying my logstash output from [log_type] to [fields][log_type] - I was able to see the indexes the way I wanted, thanks a lot for your help !

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.