Elasticsearch filter not working due to 1 sec refresh?

I am trying to use elasticsearch filter but with no success.
I want to search logs to find past events and add fields to actual event.
I think this is due to 1 sec refresh of indices so when past events are
too close to actual event they are not available.

I tried to:
1.change ES output options in logstash
flush_size => 1
idle_flush_time =>1
2.patched ES filter gem with action
client.indices.stats refresh: true
before search - but that's not working too!!

one working ugly solution - with
before search action
in ES filter

what am i doing/thinking wrong?
Is there another solution for searching past events in logstash?

Why do you think this is a filter problem?

What are you trying to do and what are you seeing?

my work is based on excellent tutorial
ok for example
i cannot use multiline filter for exim logs
because one mail message can have many log lines through few day(s) or second(s)
when mail message is:

I want copy fields from events 1,2 to 3 event when message is delivered.

So when events 1,2,3 stream to logstash in a second from logs
i assume - ES filter cannot find events 1,2 and write to 3 because
1,2 are not indexed yet

thats why i try to patch filter and refresh index
client.indices.refresh index: @index
before search action but that is not working too
so i am confused what is happening and why it is not working?


you need to fiddle with


in the output elasticsearch (see next page of the totorial you linked to). Set it to 1 and sleep for 1s when hitting 'delivered' and you'll have a 100% hit rate in your elastic filter. Of course this decreases performance but you can fiddle with flush_size and sleep time until your hit rate is convenient...