Http filter plugin parse error - Unexpected character ('\\\\' (code 92)): was expecting double-quote to start field name

I am using Logstash 6.5 with Elasticsearch 6.5
What I am trying to achieve is to remove from ELS document fields, that in input has null values.
Following advice from other thread, I am planning to use ruby filter to search for the fields and then http filter with _update API to remove fields. Code looks like the following:

filter {
    ruby {
        code => "
        event.set('fields_remove_script', '') 
        event.to_hash.each { |key,value|
            if value == nil
                event.set('fields_remove_script', event.get('fields_remove_script') + 'ctx._source.remove(\'' + key + '\');')
                event.remove(key)
            end
        }
        "
    }
    
    if [fields_remove_script] != '' { 
        http {
            url => "https://localhost:9200/test_idx/_update/%{[elastic_id]}"
            verb => "POST"
            user => "elastic"
            password => "changeme"
            cacert => '/logstash/certs/ca.crt'
            body_format => "json"
            body => "{\"script\" : \"%{[fields_remove_script]}\"}"
        }
    }
}

and creates the following error:

[ERROR] 2019-09-25 19:23:02.688 [Ruby-0-Thread-7: :1] http - error during HTTP request {:url=>"https://localhost:9200/test_idx/_update/id1", :code=>400, :response=>"{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"i_o_exception","reason":"Unexpected character ('\\' (code 92)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@71f72bb3; line: 1, column: 3]"}},"status":400}"}

So basically parse error because of Unexpected character ('\\' (code 92)): was expecting double-quote to start field name.
Any ideas about how to make it work?

If you try

mutate { add_field => { "body" => "{\"script\" : \"%{[fields_remove_script]}\"}" } }

you will see the problem. This results in

"body" => "{\\\"script\\\" : \\\"ctx._source.remove('bar');ctx._source.remove('foo');\\\"}",

I suggest

"body" => '{"script" : "%{[fields_remove_script]}"}'

which results in

"body" => "{"script" : "ctx._source.remove('bar');ctx._source.remove('foo');"}",

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.