Hi,
I'm using Elasticsearch filter plugin in logstash to enrich records and calculate time difference between two events from different logs. It works, but randomly - it returns search results for some records, but not for all of them. The pain is, it's without any pattern what is wrong.
Logstash conf:
elasticsearch {
hosts => ["https://10.20.30.40:9200"]
index => "e2e-strsuni-*"
query_template => "/etc/logstash/conf.d/query.json"
fields => {
# "saptype" => "saptype"
"file_id" => "file_id"
"E2E_SFTP" => "E2E_SFTP"
"E2E_DP" => "E2E_DP"
"date_meta" => "date_meta"
"date_sftp" => "date_sftp"
"@timestamp" => "date_dp"
}
user => "some_funky_password"
password => "some_user"
ca_file => "/etc/logstash/conf.d/elasticsearch-ca.pem"
#ssl => true
}
query.json:
{"size":1,
"sort":[{"@timestamp":"desc"}],
"query":{"simple_query_string":{"query":"ext_id:%{[ext_id]}"}}}
So it should look like this 3 records - columns E2E_SFTP, E2E_DP, E2E_RESPONSE and date_processed should be filled as they inherit "true" and time filed from previous one.
If it doesn't work, it looks like this:
There is no pattern in time, or id or anything, simply some of records are "found" via filter plugin and then are processed as they should be, some of them are not so they're indexed without "inherited" fields, and there is no record in any log why is this behavior occurring. Any ideas what to check or search for?