I have this python code that works as expected. I will like to know how to modify this code to suit the "output" parameters.
import copy
with open('myfile.txt', 'w') as f:
for doc1 in df_json:
import json
doc = mydict(doc1)
docnew = copy.deepcopy(doc)
del docnew['destination']
action = '{ "update": {"_id": %s }}\n' % doc['destination']
f.write(action)
entry = '{ "script" : { "inline": "ctx._source.parent += [\'user\': user, \'messageid\': messageid, \'message\': message, \'code\': code, \'mobile\': mobile, \'mytimestamp\': mytimestamp]", "lang" : "groovy", "params" : %s}, "upsert": {"parent" : [%s ] }}\n' % (doc, docnew)
f.write(entry)
df_json is a python object that is nothing but python dataframe flattened to json.
curl -s -XPOST XXX.xx.xx.x:9200/test_index222/doc/_bulk --data-binary @myfile.txt; echo
I used bulk API to update the documents and it is working correctly.
I am not able to use the similar code in logstash. I tried the following configuration ...
output {
elasticsearch {
action => "update"
hosts => ["172.17.0.1"]
document_id => "%{destination}"
index => "logstash3-%{+YYYY.MM.dd}"
workers => 1
script => "ctx._source.parent += ['user': user, 'messageid': messageid, 'message': message, 'code': code, 'mobile': mobile, 'mytimestamp': mytimestamp]"
script_type => "inline"
script_lang => "groovy"
scripted_upsert => "true"
}
}
I get the following error...
:response=>{"update"=>{"_index"=>"logstash4-2016.07.29", "_type"=>"csv", "_id"=>"919359000000",
"status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to execute script",
"caused_by"=>{"type"=>"script_exception", "reason"=>"failed to run in line script
[ctx._source.parent += ['user': user, 'messageid': messageid, 'message': message, 'code': code, 'mobile': mobile, 'mytimestamp': mytimestamp]]
using lang [groovy]", "caused_by"=>{"type"=>"missing_property_exception", "reason"=>"No such property: user for class: fe1b423dc4966b0f0b511b732474637705bf3bb1"}}}}}, :level=>:warn}