# Logstash: problem for querying elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [June 16, 2016, 10:47am UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977 "2016-06-16T10:47:58Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sen](https://discuss.elastic.co/u/sen)\ **Post date:** [June 16, 2016, 10:47am UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/1 "2016-06-16T10:47:58Z")
_Hello everyone,_ _Through logstash, I want to query elasticsearch in order to get fields from previous events and do some computation with fields of my current event and add new fields. Here is what I did:_ **input file:** {"device":"device1","count":5} {"device":"device2","count":11} {"device":"device1","count":8} {"device":"device3","count":100} {"device":"device3","count":95} {"device":"device3","count":155} {"device":"device2","count":15} {"device":"device1","count":55} **My expected output:** {"device":"device1","count":5,"previousCount=0","delta":0} {"device":"device2","count":11,"previousCount=0","delta":0} {"device":"device1","count":8,"previousCount=5","delta":3} {"device":"device3","count":100,"previousCount=0","delta":0} {"device":"device3","count":95,"previousCount=100","delta":-5} {"device":"device3","count":155,"previousCount=95","delta":60} {"device":"device2","count":15,"previousCount=11","delta":4} {"device":"device1","count":55,"previousCount=8","delta":47} **Logstash filter part:** filter { elasticsearch { hosts =\> ["localhost:9200/database"] query =\> 'device:"%{[device]}"' sort =\> "@timestamp:desc" fields =\> ['count','previousCount'] } ``` if [previousCount]{ ruby { code => "event[delta] = event[count] - event[previousCount]" } } ``` else{//for devices which are not in the database yet. For instance device1 at count:5 mutate { add\_field =\> { "previousCount" =\> "0" } add\_field =\> { "delta" =\> "0" } } } } **My problem:** _For every line of my input file I got the following error : Failed to query elasticsearch for previous event .._ _It seems that every line completely treated is not put in elasticsearch before logstash starts to treat the next line._ _I don't know if my conclusion is correct and, if yes, why it happens._ _So, do you know how I could solve this problem please ?!_ _Thank you for your attention and your help._ S ---
**Author:** ![sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sen](https://discuss.elastic.co/u/sen)\ **Post date:** [June 16, 2016, 11:54am UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/2 "2016-06-16T11:54:04Z")
an example for the output: { "device" =\> "device2", "count" =\> 15, "@version" =\> "1", "@timestamp" =\> "2016-06-16T11:52:31.594Z", "path" =\> "xxx", "host" =\> "localhost.localdomain", "previousCount" =\> "0", "delta" =\> "0" } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2016, 12:01pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/3 "2016-06-16T12:01:52Z")
The problem with this approach is that 1) Logstash sends requests to Elasticsearch in bulk and 2) indexed events are not immediately made available for search in Elasticsearch. This means there will be a lag (up to a second or more) between an index passing through Logstash and it being searchable. If your events are arriving close together you might need to store the data within the Logstash process, which MAY require a custom plugin. ---
**Author:** ![sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sen](https://discuss.elastic.co/u/sen)\ **Post date:** [June 16, 2016, 1:48pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/4 "2016-06-16T13:48:59Z")
Ok I understand, before starting to develop your solution I would like to know if it is possible to force logstash to process line per line and maybe to add a lap (a kind of sleep) to let logstash put the data in elasticsearch. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2016, 1:54pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/5 "2016-06-16T13:54:12Z")
I don't believe this is possible, and even if it was you would see very, very poor performance. ---
**Author:** ![sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sen](https://discuss.elastic.co/u/sen)\ **Post date:** [June 16, 2016, 1:55pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/6 "2016-06-16T13:55:01Z")
ok. And what about elasticsearc in cluster ? will it change something ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2016, 2:00pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/7 "2016-06-16T14:00:25Z")
No, I don't see how it would. ---
**Author:** ![sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sen](https://discuss.elastic.co/u/sen)\ **Post date:** [June 16, 2016, 2:14pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/8 "2016-06-16T14:14:05Z")
Such a shame ! I just wanted to complet the description, the error is: error =\> NoMethodError: undefined method `start\_with?' for nil:NilClass Do you know how I can solve this ? ---
**Author:** ![Alaska](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Alaska](https://discuss.elastic.co/u/Alaska)\ **Post date:** [June 16, 2016, 3:42pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/9 "2016-06-16T15:42:35Z")
See here: > [@Elasticsearch filter no longer working in 2.3.1](https://discuss.elastic.co/t/elasticsearch-filter-no-longer-working-in-2-3-1/47908/2): > > And now I am having another similar problem with translate plugin. I have been using translate filter using logstash 2.3.1 and it was working fine. like translate { field =\> "xxxx" dictionary\_path =\> "/yyyy/zzzz.csv" destination =\> "wwww" } The configuration works in logstash 2.3.1, but when running with logstash 2.2.0, I got this error The error reported is: LogStash::Filters::Translate: Bad Syntax in dictionary file /yyyy/zzzz.csv Combining this issue and the issue above, I cannot run… Seems the documentation on [Elasticsearch filter plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html) is wrong still. I did what was posted and changed fields =\> ["field1", "field2"] to fields =\> [["field1", "field2"]] and it worked for me as well. Was running into the same error as you with Logstash 2.3.2. ---
**Author:** ![sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sen](https://discuss.elastic.co/u/sen)\ **Post date:** [June 16, 2016, 4:23pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/10 "2016-06-16T16:23:07Z")
it solved the problem of start\_with but i still have my problem regarding elasticsearch that does not store data as fast as i want... ---
**Author:** ![Alaska](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Alaska](https://discuss.elastic.co/u/Alaska)\ **Post date:** [June 16, 2016, 5:42pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/11 "2016-06-16T17:42:56Z")
Ah sorry I have no answer to that, only the fix to start\_with sorry 😅 ---
**Author:** ![sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sen](https://discuss.elastic.co/u/sen)\ **Post date:** [June 17, 2016, 12:37pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/12 "2016-06-17T12:37:41Z")
Does anyone know if it is possible to configure logstash to only treate one event at atime (filter + store in elasticsearch) before treating the next one ? ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [June 17, 2016, 12:46pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/13 "2016-06-17T12:46:19Z")
Well, you could use the `http` output instead of the `elasticsearch` one. The "advantage" is that you can specify the `refresh=true` parameter when indexing the document and almost be sure that when the next event comes in, the document will be available when searched via the `elasticsearch` filter plugin. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 17, 2016, 1:04pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/14 "2016-06-17T13:04:18Z")
The problem with this approach is that all events need to be processed in sequence, and each event need to be written to Logstash and refreshed before the next one can be processed. This will be horrendously slow and will not scale. I am not even sure if it is possible to do this using Logstash as it by default processes all stages in parallel. If you want to see how badly this performs, write a simple script with this logic and try it out. Creating a custom plugin that stores this data in memory for a certain period is going to perform much better. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [June 17, 2016, 1:13pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/15 "2016-06-17T13:13:52Z")
+1 Christian How about using the existing `aggregate` filter for this? ---
**Author:** ![sen](https://avatars.discourse-cdn.com/v4/letter/s/cc9497/32.png) [@sen](https://discuss.elastic.co/u/sen)\ **Post date:** [June 17, 2016, 2:25pm UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/16 "2016-06-17T14:25:08Z")
Finally, I think that i will do the following solution: I have a logstash file that takes my data from my log and stores it in a elasticsearch index (ES-1). Then I use a second logstash file that takes my data from ES-1 and also queries my previous data from the same ES-1. The results will be store in a new index (ES-2). It seems to me more reliable and easier to implement even if it is a little bit heavy. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:52am UTC](https://discuss.elastic.co/t/logstash-problem-for-querying-elasticsearch/52977/17 "2017-07-06T04:52:10Z")