# Error in filter elasticsearch plugin **URL:** \ **Category:** Logstash\ **Created:** [August 6, 2019, 5:51pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074 "2019-08-06T17:51:54Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![leeyu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leeyu/32/48561_2.png) [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Post date:** [August 6, 2019, 5:51pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074/1 "2019-08-06T17:51:54Z")
Hi all, I am using version 7.2. I encountered some trouble while using the filter elasticsearch plugin. The behavior is very random, some of the data uploaded correctly but I also get some error Here is the log from logstash > [2019-08-06T10:07:02,177][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=\>"mapping\_device\_sn", :error=\>"[400] {"error":{"root\_cause":[{"type":"query\_shard\_exception","reason":"Failed to parse query [sn:%{[device\_sn]}]","index\_uuid":"vnxHfJgsRz6f4KIne-zZrA","index":"mapping\_device\_sn"}],"type":"search\_phase\_execution\_exception","reason":"all shards failed","phase":"query","grouped":true,"failed\_shards":[{"shard":0,"index":"mapping\_device\_sn","node":"kuAYIY0yTlCZM7Du464gYg","reason":{"type":"query\_shard\_exception","reason":"Failed to parse query [sn:%{[device\_sn]}]","index\_uuid":"vnxHfJgsRz6f4KIne-zZrA","index":"mapping\_device\_sn","caused\_by":{"type":"parse\_exception","reason":"Cannot parse 'sn:%{[device\_sn]}': Encountered \" \"]\" \"] \"\" at line 1, column 15.\nWas expecting:\n \"TO\" ...\n ","caused\_by":{"type":"parse\_exception","reason":"Encountered \" \"]\" \"] \"\" at line 1, column 15.\nWas expecting:\n \"TO\" ...\n "}}}}]},"status":400}"} Log from elasticsearch ``` [elasticsearch.server][DEBUG] All shards failed for phase: [query] [elasticsearch.server][DEBUG] [0], node[kuAYIY0yTlCZM7Du464gYg], [P], s[STARTED], a[id=NKeTm6rjTgCj_zwuFPyacA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[mapping_device_sn], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":1,"query":{"query_string":{"query":"sn:%{[device_sn]}","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":false,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},"sort":[{"@timestamp":{"order":"desc"}}]}}] ``` My logstash config ``` input { file { path => "/data/ELK_raw/IPS/data/*/ips_aggregate.csv" sincedb_path => "/dev/null" mode => "read" file_completed_action => "log" file_completed_log_path => "/data/ELK/read_log/ips_read_log.txt" type => "ips" } } filter { csv { autodetect_column_names => "true" autogenerate_column_names => "true" skip_header => "true" separator => "," } elasticsearch { hosts => ["localhost:9200"] index => "mapping_ips" query => "id:%{[id]}" result_size => 1 fields => { " signature_name" => "signature_name" " engine_rule" => "engine_rule" } } elasticsearch { hosts => ["localhost:9200"] index => "mapping_device_sn" query => "sn:%{[device_sn]}" result_size => 1 fields => { "first_industry" => "first_industry" "customer" => "customer" "is_trial" => "is_trial" "product_type" => "product_type" "second_industry" => "second_industry" "warranty_date" => "warranty_date" } } mutate { remove_field => ["@timestamp"] remove_field => ["@version"] remove_field => ["host"] remove_field => ["message"] remove_field => ["path"] remove_field => ["type"] } } output { elasticsearch { hosts => ["localhost:9200"] index => "cv_ips" } } ``` Have anyone encountered this before? Thanks ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 6, 2019, 5:59pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074/2 "2019-08-06T17:59:51Z")
The fact that %{[device\_sn]} is getting to elasticsearch suggests you have some events that do not have a [device\_sn] field. ---
**Author:** ![leeyu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leeyu/32/48561_2.png) [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Post date:** [August 6, 2019, 6:04pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074/3 "2019-08-06T18:04:56Z")
@Badger Hi Badger, Thanks for reply. I ran `grep -nrL 'device_sn'` on all my input file and return nothing. So I think all the file does contain the field. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 6, 2019, 6:24pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074/4 "2019-08-06T18:24:09Z")
Does the problem go away if you set 'pipeline.java\_execution: false' in logstash.yml. If so you may be hitting [this](https://github.com/elastic/logstash/issues/10938) issue. ---
**Author:** ![leeyu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leeyu/32/48561_2.png) [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Post date:** [August 6, 2019, 6:31pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074/5 "2019-08-06T18:31:14Z")
Hi Badger, Thanks for the reply. I will try it now. ---
**Author:** ![leeyu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leeyu/32/48561_2.png) [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Post date:** [August 6, 2019, 10:31pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074/6 "2019-08-06T22:31:13Z")
@Badger HI Badger, I tried setting 'pipeline.java\_execution: false' in logstash.yml but the result is the same. I am getting the same error. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 6, 2019, 10:32pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074/7 "2019-08-06T22:32:24Z")
Can you do the 'grep -rn ...' and verify that the line number is always 1? You do have --pipeline.workers 1 set, right? ---
**Author:** ![leeyu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leeyu/32/48561_2.png) [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Post date:** [August 6, 2019, 11:20pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074/8 "2019-08-06T23:20:24Z")
@Badger It turns out that it was due to an error in merging. So we are working on fixing that right now, but it should be fine now. Thanks!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 3, 2019, 11:20pm UTC](https://discuss.elastic.co/t/error-in-filter-elasticsearch-plugin/194074/9 "2019-09-03T23:20:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.