Error in filter elasticsearch plugin

Elastic Stack Logstash
1

Hi all,

I am using version 7.2.

I encountered some trouble while using the filter elasticsearch plugin.

The behavior is very random, some of the data uploaded correctly but I also get some error

Here is the log from logstash

[2019-08-06T10:07:02,177][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"mapping_device_sn", :error=>"[400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"Failed to parse query [sn:%{[device_sn]}]","index_uuid":"vnxHfJgsRz6f4KIne-zZrA","index":"mapping_device_sn"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"mapping_device_sn","node":"kuAYIY0yTlCZM7Du464gYg","reason":{"type":"query_shard_exception","reason":"Failed to parse query [sn:%{[device_sn]}]","index_uuid":"vnxHfJgsRz6f4KIne-zZrA","index":"mapping_device_sn","caused_by":{"type":"parse_exception","reason":"Cannot parse 'sn:%{[device_sn]}': Encountered \" \"]\" \"] \"\" at line 1, column 15.\nWas expecting:\n \"TO\" ...\n ","caused_by":{"type":"parse_exception","reason":"Encountered \" \"]\" \"] \"\" at line 1, column 15.\nWas expecting:\n \"TO\" ...\n "}}}}]},"status":400}"}

Log from elasticsearch

[elasticsearch.server][DEBUG] All shards failed for phase: [query]

[elasticsearch.server][DEBUG] [0], node[kuAYIY0yTlCZM7Du464gYg], [P], s[STARTED], a[id=NKeTm6rjTgCj_zwuFPyacA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[mapping_device_sn], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":1,"query":{"query_string":{"query":"sn:%{[device_sn]}","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":false,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},"sort":[{"@timestamp":{"order":"desc"}}]}}]

My logstash config

input {
    file {
        path => "/data/ELK_raw/IPS/data/*/ips_aggregate.csv"
        sincedb_path => "/dev/null"
        mode => "read"
        file_completed_action => "log"
        file_completed_log_path => "/data/ELK/read_log/ips_read_log.txt"
        type => "ips"
    }
}

filter {
    csv {
        autodetect_column_names => "true"
        autogenerate_column_names => "true"
        skip_header => "true"
        separator => ","
    }
elasticsearch {
        hosts => ["localhost:9200"]
        index => "mapping_ips"
        query => "id:%{[id]}"
        result_size => 1
        fields => {
            " signature_name" => "signature_name"
            " engine_rule" => "engine_rule"
        }
    }

   elasticsearch {
        hosts => ["localhost:9200"]
        index => "mapping_device_sn"
        query => "sn:%{[device_sn]}"
        result_size => 1
        fields => {
            "first_industry" => "first_industry"
            "customer" => "customer"
            "is_trial" => "is_trial"
            "product_type" => "product_type"
            "second_industry" => "second_industry"
            "warranty_date" => "warranty_date"
        }
    }

   mutate {
        remove_field => [ "@timestamp" ]
        remove_field => [ "@version" ]
        remove_field => [ "host" ]
        remove_field => [ "message" ]
        remove_field => [ "path" ] 
        remove_field => [ "type" ] 
    }   
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "cv_ips"
    }
}

Have anyone encountered this before?

Thanks

2

The fact that %{[device_sn]} is getting to elasticsearch suggests you have some events that do not have a [device_sn] field.

1 Like
3

@Badger

Hi Badger,

Thanks for reply.

I ran grep -nrL 'device_sn' on all my input file and return nothing.

So I think all the file does contain the field.

4

Does the problem go away if you set 'pipeline.java_execution: false' in logstash.yml. If so you may be hitting this issue.

5

Hi Badger,

Thanks for the reply. I will try it now.

6

@Badger

HI Badger,

I tried setting 'pipeline.java_execution: false' in logstash.yml but the result is the same.

I am getting the same error.

7

Can you do the 'grep -rn ...' and verify that the line number is always 1?

You do have --pipeline.workers 1 set, right?

1 Like
8

@Badger

It turns out that it was due to an error in merging.

So we are working on fixing that right now, but it should be fine now.

Thanks!!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.