Unable to create dynamic index with conditional check in logstash

[anra137 (Anitha Raja)]

Hi,

I want to dynamically create index based on the events. I have configured configured several log files in filebeat and the events are successfully stored in ElasticSearch. But when I try to differentiate the logs in different index, it is only passing one of the condition in logstash. For example:

filter {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} *\[%{GREEDYDATA:logger}\] \(%{WORD:thread}\)  %{GREEDYDATA:message}"}
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} *\[%{JAVACLASS:logger}\] \(%{GREEDYDATA:thread}\) \{%{GREEDYDATA:mdc}\} %{GREEDYDATA:message}"}
        }
    if "test" in [tags]{
       mutate { replace  => { "[@metadata][type]" => "test" } }
    }
}
output {
if "_grokparsefailure" not in [tags] {
    elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][type]}-%{+YYYY.MM.dd}"
    }

  }
}

Filebeat config:
type: log
enabled: true
paths:
- C:\work\programs\agents\log\easyfile.log
fields:
service.id: 77877674
tags: [delivery]
fields_under_root: true
type: log
enabled: true
paths:
- C:\temp\easyfile.log
fields:
service.id: 12341234
tags: [test]
fields_under_root: true

For the above config, I can see one index filebeat-test-2018.09.18 created. But not the other one which should be filebeat-doc-2018.09.18 and got error as below

[ 2018-09-18T14:27:06,598][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-doc-2018.09.18", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x361230bb>], :response=>{"index"=>{"_index"=>"filebeat-doc-2018.09.18", "_type"=>"doc", "_id"=>"UPSl7GUBxx--wOCZ_1LN", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [tags] of different type, current_type [long], merged_type [text]"}}}}

The two log formats are same. Only difference is just one additional content

for example
2018-03-09 16:12:20,315 INFO [com.sample.teste.TestClass] (https-express.sample.com-433-8) Testing logger
2018-03-09 16:12:20,315 INFO [com.sample.teste.TestClass] (https-express.sample.com-433-8) {my-name-1=my-value-1, my-name-2=my-value-2} Testing logger

[magnusbaeck (Magnus Bäck)]

For some reason the tags field in the filebeat-doc-2018.09.18 index has been mapped with the long type which isn't compatible with what Logstash sends. Why is that? Have you indexed documents previously with an integer tags field?

[anra137 (Anitha Raja)]

No, tags field was text before and the previous documents were mapped correctly. This happens only when I try to send to 2 different index based on the conditions applied. When I add the index one by one. This issue is resolved. Otherwise always only the first condition executed successfully.