I am using Logstash 6.5 with Elasticsearch 6.5
What I am trying to achieve is to remove from ELS document fields, that in input has null values.
Following advice from other thread, I am planning to use ruby
filter to search for the fields and then http
filter with _update
API to remove fields. Code looks like the following:
filter {
ruby {
code => "
event.set('fields_remove_script', '')
event.to_hash.each { |key,value|
if value == nil
event.set('fields_remove_script', event.get('fields_remove_script') + 'ctx._source.remove(\'' + key + '\');')
event.remove(key)
end
}
"
}
if [fields_remove_script] != '' {
http {
url => "https://localhost:9200/test_idx/_update/%{[elastic_id]}"
verb => "POST"
user => "elastic"
password => "changeme"
cacert => '/logstash/certs/ca.crt'
body_format => "json"
body => "{\"script\" : \"%{[fields_remove_script]}\"}"
}
}
}
and creates the following error:
[ERROR] 2019-09-25 19:23:02.688 [Ruby-0-Thread-7: :1] http - error during HTTP request {:url=>"https://localhost:9200/test_idx/_update/id1", :code=>400, :response=>"{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"i_o_exception","reason":"Unexpected character ('\\' (code 92)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@71f72bb3; line: 1, column: 3]"}},"status":400}"}
So basically parse error because of Unexpected character ('\\' (code 92)): was expecting double-quote to start field name
.
Any ideas about how to make it work?