SystemDate or time to elasticsearch fields

Elastic Stack Logstash
1

We are trying to set current system date to one of the elastic field while inserting records using logstash script.

mutate {
add_field => {"logdate" => "%{[+YYYY-MM-dd]}"}
}

output {
elasticsearch{
hosts => ["localhost:9200"]
"script" => '{
"lst_use_dt" => "%{[logdate]}"
}'
}
}

But its not inserting/updating anything to lst_use_dt field.

2

add_field => {"logdate" => "%{[+YYYY-MM-dd]}"}

This syntax is wrong. Remove the angle brackets to populate the logdate field with a YYYY-MM-dd date from on the @timestamp field.

"script" => '{
"lst_use_dt" => "%{[logdate]}"
}'

This isn't a valid script. I'm not even sure what you're trying to do here.

3

Hi Magnus,

We have an index with fields iss_id, cntry_cde, financial_inst, create_dt, update_date, exp_dt.

And we are receiving a data file from other team with values for the fields iss_id, cntry_cde, financial_inst only.

So our requirement is to insert the records if they are not already exists in ElasticSearch including other fields like create_dt, update_date, exp_dt where create_dt and update_dt holds sysdate and exp_dt as 12-12-9999 00:00:000

If record already exists then we have update the record with the values from data file and also only update update_dt to sysdate not touching create_dt and exp_dt.

We tried to use script and upsert but it didn't work. So we need help in implementing this requirement.

4

If you strictly want to update existing documents but never create documents you probably have to use an elasticsearch filter to obtain the current document (if it exists), update its contents, and send it back with an elasticsearch output. I haven't tried anything like that myself.

5

Hi Magnus,

Thanks for the information. Could you please give me some examples on how to read existing document from Elasticsearch and update the same with new values from the input file?

Thanks in advance.

6

Did you look at the example in the elasticsearch filter documentation?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.