Publish same Log file logs to elastic with different index

[Vinutha_A_H]

Hi,

We are trying to read a log file twice. Sending one copy as it is (just mapping the key to each field in logs using grok) to kafka/elastic with index xyz and second copy by removing some fields (we are also doing key mapping to each field in logs using grok) to Kafka/elastic with index abcd. The Kafka broker, topic and everything else is same.

I see most of the logs being published to both the index but I also see few missing in either of the index.

So, would like to get a expert view on this. As, if it is the correct thing to do and is logstash expected to work smoothly in this scenario.

[Vinutha_A_H]

Hi,

Any comments on this topic?

[Jenni]

I don't see an obvious reason why this wouldn't work, but then again it's a very abstract question. Maybe the Logstash configuration, Elasticsearch field mappings, error logs or examples for the missing events would give us a clue.

[Vinutha_A_H]

Thanks Jenni for your response. Logstash config looks as below and dont see any errors in logstash logs.

The index mapped at elastic are 'abcd' and 'abd-efg'
Please let me know, if in case any issues with these config and mapping.

//
Input {
file {
path => ["/local/logs/filename.log"]
add_field => { "app" => "abcd" "DOMAIN" => "test"}
type => 'wxyz'
}
file {
path => ["/local/logs/filename.log"]
add_field => { "app_id" => "abc-efg" "DOMAIN" => "test"}
type => 'wxy'
}
}

filter {
if [type] == "wxyz" {
grok {
match => [ "message", "%{DATE:date}-%{TIME:time} NAME:%{GREEDYDATA:text} DATA:%{GREEDYDATA:text}" ]
}
mutate { remove_field => [ "message" , "DATA" ] }
mutate { add_field => [ "ts" , "%{date} %{time}" ] }
date {
match => [ "ts", "yy/MM/dd HH:mm:ss" ]
target => "@timestamp"
}
}
if [type] == "wxy" {
grok {
match => [ "message", "%{DATE:date}-%{TIME:time} NAME:%{GREEDYDATA:text} DATA:%{GREEDYDATA:text}" ]
}
mutate { add_field => [ "ts" , "%{date} %{time}" ] }
date {
match => [ "ts", "yy/MM/dd HH:mm:ss" ]
target => "@timestamp"
}
}
}

output {
kafka {
topic_id => ""
bootstrap_servers => "brokerhost:brokerport"
codec => json {}
security_protocol => "SSL"
ssl_keystore_location => ""
ssl_keystore_password => ""
ssl_truststore_location => ""
ssl_truststore_password => ""
}
}

[Vinutha_A_H]

Hi Jenni, I hope there is nothing wring in the config. Please confirm.