Updating elasticsearch document, without adding "doc"

[T1mmy (Tim Rauh)]

I'm trying to update my users in elasticsearch via logstash couchdb_changes. But everytime a change is made to the user, logstatsh adds a "doc"-subarray with the new changes to my document in elasticsearch, instead of updating the document itself.

i.e. when i got:

[
user_id => 1,
username => franky,
firstname =>  frank,
lastname => mauer
]

and i change the lastname, i get

 [
   user_id => 1,
   username => franky,
   firstname =>  frank,
   lastname => mauer,
   doc => [ 
        user_id => 1,
        username => franky,
        firstname =>  frank,
        lastname => whatever,
   ]

]

this is my conf:

    couchdb_changes {
        host => "couchdb"
        db => "user"
        username => "name"
        password => "***"
        sequence_path=>"/usr/share/logstash/.couchdb_seq_user"
        initial_sequence => 0
    }


      elasticsearch { 
            hosts => "elasticsearch:9200"
            upsert => "%{[doc]}"
            index => "%{type}"
            document_id => "%{[doc][username]}"
            action => "%{[@metadata][action]}"
            template => "/usr/share/logstash/templates/template_user.json"
            template_name => "user"
        }

Any idea how i can stop this behavoir and just update the document like every normal thinking human would expect.
Thanks.

[yaauie (Ry Biesemeyer)]

I'm not familiar with that plugin, but as a change-stream provider my best guess is that it needs a way to tell the downstream that a document has been deleted, and it can't do that and also support all of the fields at top-level.

Once you've selected only new- and updated-updated documents, you can likely use a filter like the following ruby filter to move the contents of the doc field up to the root.

filter {
  ruby {
    code => "
      doc = event.get('doc')
      if doc.kind_of?(Hash)
        event.remove('doc')
        doc.each do |key, value|
          event.set(key, value)
        end
      end
    "
  }
}