It appears to me that the elasticsearch output module is not properly passing at least the parent parameter along with it. I'm not sure why. The document_id and parent parameters are in the data, and confirmed to be consistent between them and between runs.
Log message in ES says -
[2016-04-18 10:21:26,853][INFO ][rest.suppressed ] /_bulk Params: {}
java.lang.IllegalArgumentException: Can't specify parent if no parent field has been configured
logstash config elasticsearch section is
elasticsearch {
hosts => [ "localhost:9200" ]
index => "relationship"
parent => "%{fingerprint}"
routing => "%{fingerprint}"
#template => "iprel-basic.template"
#template_name => "relationships"
#template_overwrite => true
}
logstash config elasticsearch config for the parent document
elasticsearch {
action => "update"
doc_as_upsert => true
document_id => "%{fingerprint}"
routing => "%{fingerprint}"
hosts => [ "localhost:9200" ]
index => "relationship"
template => "iprel-basic.template"
template_name => "relationships"
template_overwrite => true
}
My mapping is
{
"template" : "relationships",
"settings" : {
"index": {
"number_of_shards" : "1",
"number_of_replicas" : "0",
"refresh_interval" : "1s"
}
},
"mappings" : {
"ip-relationship": {
"_type": "ip_relationship",
"_all" : {"enabled" : true}
},
"ip-conversation" : {
"_type": "ip-conversation",
"_all" : {"enabled" : true},
"_routing": {
"required": true
},
"_parent": {
"type": "ip-relationship"
}
}
}
}