Elasticsearch filter plugin not working

Elastic Stack Logstash
1

running multiple configs in logstash server. Last config, querying the previous logs in elasticsearch which have transferred by other config files in logstash server.

filter{
...
elasticsearch {
hosts => ["XXXXXX/"]
index => "act_test"
query => "tourid:%{[data][id]}"
fields => { "vendorcitycode" => "vendorcitycode_"
"location" => "location_"
"imagethumbnailurl" => "imagethumbnailurl_" }

...
}

log says no mapping found , but tourid 266152 already there in act_test in ELK

[2019-12-04T10:14:25,698][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"act_test", :query=>"tourid:266152", :event=>#LogStash::Event:0x6625ba12, :error=>"#<Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"No mapping found for [@timestamp] in order to sort on","index_uuid":"5TY-QtlrQWiwUkGj63unOw","index":"act_test"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"act_test","node":"vrtCsitlT-GGkmSrRaO9Hg","reason":{"type":"query_shard_exception","reason":"No mapping found for [@timestamp] in order to sort on","index_uuid":"5TY-QtlrQWiwUkGj63unOw","index":"act_test"}}]},"status":400}>"}

Seems last config read before commit the previous logs in ELK. all config files run in given priority order

2

The plugin uses @timestamp by default for sorting, can you check if act_test has a field called "@timestamp" ?

3

Hi Julien,
Thx for the reply. @timestamp" ? there in act_test. problem is data pushing to act_test config also there. seems last conf file trying to read before commit act_test.

4

Maybe pipeline to pipeline feature may help you if you need a specific order of execution, if all conf file are part of the same pipeline there is not really an order you can rely on
Thanks
Julien

5

Hi, all config execute in order. I have name all config files in alphabetical order according to priority.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.