# Elasticsearch filter not working due to 1 sec refresh? **URL:** \ **Category:** Logstash\ **Created:** [June 3, 2015, 11:55am UTC](https://discuss.elastic.co/t/elasticsearch-filter-not-working-due-to-1-sec-refresh/1820 "2015-06-03T11:55:48Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![niksfirefly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niksfirefly/32/498_2.png) [@niksfirefly](https://discuss.elastic.co/u/niksfirefly)\ **Post date:** [June 3, 2015, 11:55am UTC](https://discuss.elastic.co/t/elasticsearch-filter-not-working-due-to-1-sec-refresh/1820/1 "2015-06-03T11:55:48Z")
I am trying to use elasticsearch filter but with no success. I want to search logs to find past events and add fields to actual event. I think this is due to 1 sec refresh of indices so when past events are too close to actual event they are not available. I tried to: 1.change ES output options in logstash flush\_size =\> 1 idle\_flush\_time =\>1 2.patched ES filter gem with action client.indices.stats refresh: true before search - but that's not working too!! one working ugly solution - with sleep(20) before search action in ES filter what am i doing/thinking wrong? Is there another solution for searching past events in logstash? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 3, 2015, 12:10pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-not-working-due-to-1-sec-refresh/1820/2 "2015-06-03T12:10:00Z")
Why do you think this is a filter problem? What are you trying to do and what are you seeing? ---
**Author:** ![niksfirefly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niksfirefly/32/498_2.png) [@niksfirefly](https://discuss.elastic.co/u/niksfirefly)\ **Post date:** [June 3, 2015, 1:53pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-not-working-due-to-1-sec-refresh/1820/3 "2015-06-03T13:53:47Z")
my work is based on excellent tutorial [https://graemef.wordpress.com/2014/05/10/heres-the-science-bit-3-grok-friends/](https://graemef.wordpress.com/2014/05/10/heres-the-science-bit-3-grok-friends/) ok for example i cannot use multiline filter for exim logs because one mail message can have many log lines through few day(s) or second(s) when mail message is: 1.arrived 2.processed 3.delivered I want copy fields from events 1,2 to 3 event when message is delivered. So when events 1,2,3 stream to logstash in a second from logs i assume - ES filter cannot find events 1,2 and write to 3 because 1,2 are not indexed yet thats why i try to patch filter and refresh index client.indices.refresh index: @index before search action but that is not working too so i am confused what is happening and why it is not working? ---
**Author:** ![edi](https://avatars.discourse-cdn.com/v4/letter/e/9de0a6/32.png) [@edi](https://discuss.elastic.co/u/edi)\ **Post date:** [September 11, 2015, 10:43am UTC](https://discuss.elastic.co/t/elasticsearch-filter-not-working-due-to-1-sec-refresh/1820/4 "2015-09-11T10:43:28Z")
Hi, you need to fiddle with flush\_size in the output elasticsearch (see next page of the totorial you linked to). Set it to 1 and sleep for 1s when hitting 'delivered' and you'll have a 100% hit rate in your elastic filter. Of course this decreases performance but you can fiddle with flush\_size and sleep time until your hit rate is convenient... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:29am UTC](https://discuss.elastic.co/t/elasticsearch-filter-not-working-due-to-1-sec-refresh/1820/5 "2017-07-06T05:29:23Z")