Logstash Update a document in elasticsearch

Elastic Stack Logstash
1

Trying to update a specific field in elasticsearch through logstash. Is it possible to update only a set of fields through logstash ?

Please find the code below,

filter {
csv {
separator => "|"
columns => ["NAME","ID","COUNTRYNAME","DATE","STATECODE", "G_NAME","G_ID","G_COUNTRYNAME","
G_DATE","G_STATECODE"]
}
ruby {
code => "
hash = event.to_hash
hash.each do |k,v|
if v == nil
event.remove(k)
end
end
"
}
}
We are using the above configuration when we use this the null value field is getting removed instead of skipping null value update.

Sample Complete document log input

John|892|Canada|31-01-2017|QC
Updating a document with null value should not remove the field instead the null value fields should be skipped and the reside with existing date 31-01-2017 value. John|892|Canada||QC

2

Hi,

yes this is possible, you have to set action => 'update' in your elasticsearch output.

You have to remember that this will severly impact you elasticsearch cluster so handle with care.

Paul.

1 Like
3

how to achieve ?can you tell me the sample configuration

4

I have updated like this but i am not getting proper results
filter {
csv {
separator => "|"
columns => ["NAME","ID","COUNTRYNAME","DATE","STATECODE", "G_NAME","G_ID","G_COUNTRYNAME","
G_DATE","G_STATECODE"]
}
mutate {
update => { "COUNTRYNAME" => "qqqqqqq" }
}

}

5

EDIT: I for to mention that you need to generate your own document_id and set that. So in order to update a existing document you need to set a id on the initial document and reuse that. (bad example: ie: email_address) or query ES for the document_id using the logstash-filter-elasticsearch plugin.

It will look something like this.

output {
    elasticsearch {
        host => 'your es host'
        action => 'update'
        document_id => 'your generated document id'
        index => 'your index name
    }
}

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action

6

output {
elasticsearch {
hosts => ["localhost:9200"]
action => "update"
index => "logstash-data-monitor"
document_id => "%{DOC_ID}%"
}
}

This is what we already have. When we try to update it the entire document is getting updated instead of updating a single field. How to achieve this ?

2 Likes
7

I am not really sure what you mean here...

With this configuration you should be able to update a single field or any fields send to ES.
For example, you have field A, B and C and you send a doc with a updated field A it leaves B and C alone and it updates A and it will update you document @Version (but thats arbitrary)

Is this not what happens?

8

yeah this is not happening. can you suggest us on this ?

9

Sorry I cannot help. I have used a similar config and it updated the fields which I needed... if that does not work for you then I am lost to...

10

Can you share the complete document which you have used for updating. A sample one which we could use to compare with our code.

11

Can you please help on how to check null condition for a field if not null then update the values ?

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.