Can't separated elasticsearch index using logstash

[LockyZ (Manolo Mancebo)]

Hello Guys,

I'm trying to create separated indexes using logstash 6.7, basically I have a server running filebeat which sends two logs to a logstash which sends to a elasticsearch, below you may find my config files.

filebeat.yml
#=========================== Filebeat inputs =============================

filebeat.prospectors:

• type: log
  paths:

  • /logs/access.*log
  • /logs/access.*log
  • /logs/access.*log
  • /logs/access.*log
  • /logs/access.*log
  • /logs/access.*log
    fields:
    log_type: rtdm-access
    jvm: "SASServer 7"
    team: "Middleware"
    system: "SAS - RTDM"
    env: "Production"
    role: "Engine"
    fields_under_root: true
    scan_frequency: 5s

• type: log
  paths:

  • /Logs/tomcatgc.log.?.current
  • /Logs/tomcatgc.log.?.current
  • /Logs/tomcatgc.log.?.current
  • /Logs/tomcatgc.log.?.current
  • /Logs/tomcatgc.log.?.current
  • /Logs/tomcatgc.log.?.current
    fields:
    log_type: rtdm-gc
    jvm: "SASServer 7"
    team: "Middleware"
    system: "SAS - RTDM"
    env: "Production"
    role: "Engine"
    fields_under_root: true
    scan_frequency: 15s

#============================= Filebeat modules ===============================

filebeat.config.modules:

Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.name: "mdw_filebeat"
setup.template.pattern: "mdw_filebeat_*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["logstash-server1:9500" , "logstash-server2:9500"]
loadbalance: true

#================================ Processors =====================================

Configure processors to enhance or manipulate events generated by the beat.

processors:

• add_host_metadata: ~
• add_cloud_metadata: ~

logstash config file

output {
if [log_type] == "rtdm-access" {
elasticsearch {
hosts => ["elasticsearch-server1:9200",elasticsearch-server2:9200]
index => "access-rtdm"
}
}
else if [log_type] == "rtdm-gc" {
elasticsearch {
hosts => ["elasticsearch-server1:9200",elasticsearch-server2:9200]
index => "gc-rtdm"
}
}
#only for debugging
stdout { codec => rubydebug }

}

Note.: If I remove the IF/Else and configure to create a standard index this it creates, but unfortunately this is not what I need, can anyone please explain what I'm doing wrong?

Filebeat / Logstash and ElasticSearch version is 6.7

thanks in advance,

[Christian_Dahlqvist]

I would recommend you look at the events being output to stdout. I would expect the [log_type] field to actually be stored as [fields][log_type]. You may therefore need to change your conditionals to reflect this.

[LockyZ (Manolo Mancebo)]

Christian_Dahlqvist,

Thanks a lot mate, by modifying my logstash output from [log_type] to [fields][log_type] - I was able to see the indexes the way I wanted, thanks a lot for your help !