Creating/updating array of objects in elasticsearch logstash output

(Nandan Phadke) #1

I am facing an issue using elastic search output with logstash. Here is my sample event


Here is what I want to have in my elastic search document after 2 upserts:


Here is my current elastic search output setting:

elasticsearch {
    index => "elastictest"
    action => "update"
    document_type => "summary"
    document_id => "%{guid}"
    doc_as_upsert => true
    script_lang => "groovy"
    script_type => "inline"
    retry_on_conflict => 3
    script => "
    if (ctx._source.nestedObjects) {
    ctx._source.nestedObjects += event.nestedObject
    } else {
    ctx._source.nestedObjects = [event.nestedObject]

Here is the error I am getting:

response=>{"update"=>{"_index"=>"elastictest", "_type"=>"summary",    "_id"=>"64648dd3-c1e9-45fd-a00b-5a4332c91ee9", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [event.nestedObject]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"unknown property [field1]"}}}}

Update Script in Elasticsearch Output Parsing Array as String
Create array in logstash -> elastic
(Edgar Samudio) #2

I think that you should ensure that nestedObject is an array to begin with, meaning that you should initialize it. They type error might be because you're trying to append to a non array field. I use a ruby filter for that (in the filter section):

ruby {
code => "
event['nestedObject'] = [];

Afterwards, you can append to the array of the document using something like(inside of the ES output plugin):

script => '

(Nandan Phadke) #3

Thanks Edgar, however the event from logstash was not the issue.

The issue turned out to be internally generated mapping in elastic due to other documents with same document_type with conflicting type on nestedObject. This caused elastic to throw a parsing exception. Fixing this, fixed this issue.

(system) #4