Error in filter elasticsearch plugin

Hi all,

I am using version 7.2.

I encountered some trouble while using the filter elasticsearch plugin.

The behavior is very random, some of the data uploaded correctly but I also get some error

Here is the log from logstash

[2019-08-06T10:07:02,177][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"mapping_device_sn", :error=>"[400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"Failed to parse query [sn:%{[device_sn]}]","index_uuid":"vnxHfJgsRz6f4KIne-zZrA","index":"mapping_device_sn"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"mapping_device_sn","node":"kuAYIY0yTlCZM7Du464gYg","reason":{"type":"query_shard_exception","reason":"Failed to parse query [sn:%{[device_sn]}]","index_uuid":"vnxHfJgsRz6f4KIne-zZrA","index":"mapping_device_sn","caused_by":{"type":"parse_exception","reason":"Cannot parse 'sn:%{[device_sn]}': Encountered \" \"]\" \"] \"\" at line 1, column 15.\nWas expecting:\n \"TO\" ...\n ","caused_by":{"type":"parse_exception","reason":"Encountered \" \"]\" \"] \"\" at line 1, column 15.\nWas expecting:\n \"TO\" ...\n "}}}}]},"status":400}"}

Log from elasticsearch

[elasticsearch.server][DEBUG] All shards failed for phase: [query]

[elasticsearch.server][DEBUG] [0], node[kuAYIY0yTlCZM7Du464gYg], [P], s[STARTED], a[id=NKeTm6rjTgCj_zwuFPyacA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[mapping_device_sn], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":1,"query":{"query_string":{"query":"sn:%{[device_sn]}","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":false,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},"sort":[{"@timestamp":{"order":"desc"}}]}}]

My logstash config

input {
    file {
        path => "/data/ELK_raw/IPS/data/*/ips_aggregate.csv"
        sincedb_path => "/dev/null"
        mode => "read"
        file_completed_action => "log"
        file_completed_log_path => "/data/ELK/read_log/ips_read_log.txt"
        type => "ips"
    }
}

filter {
    csv {
        autodetect_column_names => "true"
        autogenerate_column_names => "true"
        skip_header => "true"
        separator => ","
    }
elasticsearch {
        hosts => ["localhost:9200"]
        index => "mapping_ips"
        query => "id:%{[id]}"
        result_size => 1
        fields => {
            " signature_name" => "signature_name"
            " engine_rule" => "engine_rule"
        }
    }

   elasticsearch {
        hosts => ["localhost:9200"]
        index => "mapping_device_sn"
        query => "sn:%{[device_sn]}"
        result_size => 1
        fields => {
            "first_industry" => "first_industry"
            "customer" => "customer"
            "is_trial" => "is_trial"
            "product_type" => "product_type"
            "second_industry" => "second_industry"
            "warranty_date" => "warranty_date"
        }
    }

   mutate {
        remove_field => [ "@timestamp" ]
        remove_field => [ "@version" ]
        remove_field => [ "host" ]
        remove_field => [ "message" ]
        remove_field => [ "path" ] 
        remove_field => [ "type" ] 
    }   
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "cv_ips"
    }
}

Have anyone encountered this before?

Thanks

The fact that %{[device_sn]} is getting to elasticsearch suggests you have some events that do not have a [device_sn] field.

1 Like

@Badger

Hi Badger,

Thanks for reply.

I ran grep -nrL 'device_sn' on all my input file and return nothing.

So I think all the file does contain the field.

Does the problem go away if you set 'pipeline.java_execution: false' in logstash.yml. If so you may be hitting this issue.

Hi Badger,

Thanks for the reply. I will try it now.

@Badger

HI Badger,

I tried setting 'pipeline.java_execution: false' in logstash.yml but the result is the same.

I am getting the same error.

Can you do the 'grep -rn ...' and verify that the line number is always 1?

You do have --pipeline.workers 1 set, right?

1 Like

@Badger

It turns out that it was due to an error in merging.

So we are working on fixing that right now, but it should be fine now.

Thanks!!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.