Elasticsearch filter plugin not working

running multiple configs in logstash server. Last config, querying the previous logs in elasticsearch which have transferred by other config files in logstash server.

filter{
...
elasticsearch {
hosts => ["XXXXXX/"]
index => "act_test"
query => "tourid:%{[data][id]}"
fields => { "vendorcitycode" => "vendorcitycode_"
"location" => "location_"
"imagethumbnailurl" => "imagethumbnailurl_" }

...
}

log says no mapping found , but tourid 266152 already there in act_test in ELK

[2019-12-04T10:14:25,698][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"act_test", :query=>"tourid:266152", :event=>#LogStash::Event:0x6625ba12, :error=>"#<Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"No mapping found for [@timestamp] in order to sort on","index_uuid":"5TY-QtlrQWiwUkGj63unOw","index":"act_test"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"act_test","node":"vrtCsitlT-GGkmSrRaO9Hg","reason":{"type":"query_shard_exception","reason":"No mapping found for [@timestamp] in order to sort on","index_uuid":"5TY-QtlrQWiwUkGj63unOw","index":"act_test"}}]},"status":400}>"}

Seems last config read before commit the previous logs in ELK. all config files run in given priority order

The plugin uses @timestamp by default for sorting, can you check if act_test has a field called "@timestamp" ?

Hi Julien,
Thx for the reply. @timestamp" ? there in act_test. problem is data pushing to act_test config also there. seems last conf file trying to read before commit act_test.

Maybe pipeline to pipeline feature may help you if you need a specific order of execution, if all conf file are part of the same pipeline there is not really an order you can rely on
Thanks
Julien

Hi, all config execute in order. I have name all config files in alphabetical order according to priority.