Elasticsearch filter not working due to 1 sec refresh?

Elastic Stack Logstash
1

I am trying to use elasticsearch filter but with no success.
I want to search logs to find past events and add fields to actual event.
I think this is due to 1 sec refresh of indices so when past events are
too close to actual event they are not available.

I tried to:
1.change ES output options in logstash
flush_size => 1
idle_flush_time =>1
2.patched ES filter gem with action
client.indices.stats refresh: true
before search - but that's not working too!!

one working ugly solution - with
sleep(20)
before search action
in ES filter

what am i doing/thinking wrong?
Is there another solution for searching past events in logstash?

2

Why do you think this is a filter problem?

What are you trying to do and what are you seeing?

3

my work is based on excellent tutorial
https://graemef.wordpress.com/2014/05/10/heres-the-science-bit-3-grok-friends/
ok for example
i cannot use multiline filter for exim logs
because one mail message can have many log lines through few day(s) or second(s)
when mail message is:
1.arrived
2.processed
3.delivered

I want copy fields from events 1,2 to 3 event when message is delivered.

So when events 1,2,3 stream to logstash in a second from logs
i assume - ES filter cannot find events 1,2 and write to 3 because
1,2 are not indexed yet

thats why i try to patch filter and refresh index
client.indices.refresh index: @index
before search action but that is not working too
so i am confused what is happening and why it is not working?

4

Hi,

you need to fiddle with

flush_size

in the output elasticsearch (see next page of the totorial you linked to). Set it to 1 and sleep for 1s when hitting 'delivered' and you'll have a 100% hit rate in your elastic filter. Of course this decreases performance but you can fiddle with flush_size and sleep time until your hit rate is convenient...