How to update index in Elasticsearch

[zaeemmasood]

Hi All,

I have an ELK 7.6.2 stack running in the environment.

In my setup logstash is set to ingest file as an input and performs certain operations on it and creates an index in ES.

First few lines of the logstash conf file looks like below:

input {
  file {
    path => "/opt/gtal/ital/elasticsearch/app/logstash/stage/MQA_UAT_STATS-*.txt"

An index named demo-csv-2021.06.13 (based on date) gets created as a result of the ingestion.

The file MQA_UAT_STATS-*.txt gets updated every hour.

I need the index demo-csv-2021.06.13 to get updated with the new values without having to restart logstash or deleting the index.

Please help on how can I get this done? I am fairly new to ELK.

Thanks.

[Iker (Iker Saint)]

Probably your best option is set the id of each document with a fingerprint (Fingerprint filter plugin | Logstash Reference [7.13] | Elastic) or another unique id, so, each time that logstash tries to index a new document, it will update the document with the same id; your pipeline will look something like this:

elasticsearch {
        hosts => ["localhost:9200"]
        index => "demo-csv-2021.06.13"
        action => "update"
        document_id => "%{my_fingerprint}"
}

[zaeemmasood]

Thanks lker.

My call to ES from logstash looks like below:

    elasticsearch {
     hosts => [ "xx-xxxx-xxxx:3045" ]
     user => "xxxxx"
     password => "xxxxxxxx"
         index => "demo-csv-%{+YYYY.MM.dd}"

Would fingerprint work with the daily changing index name?

Thanks

[zaeemmasood]

Also I tried various options to set the fingerprint in my logstash conf file but it did not work out.

Can you please guide me on what additions would be required in the logstash config file with the following ES output:

    elasticsearch {
     hosts => [ "xx-xxx-xxx:3045" ]
     user => "xxxxx"
     password => "xxxxxxxxxxxxxxxx"
     index => "demo-csv-%{+YYYY.MM.dd}"
     action => "update"
     document_id => "%{my_fingerprint}"

I kept getting the following error:

[2021-06-14T14:53:01,610][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>404, :action=>["update", {:_id=>"%{fingerprint}", :_index=>"demo-csv-2021.06.14", :routing=>nil, :_type=>"_doc", :retry_on_conflict=>1}, #<LogStash::Event:0x43566efc>], :response=>{"update"=>{"_index"=>"demo-csv-2021.06.14", "_type"=>"_doc", "_id"=>"%{fingerprint}", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[_doc][%{fingerprint}]: document missing", "index_uuid"=>"5yJpaTRpSvC6SI9mTwpJwg", "shard"=>"0", "index"=>"demo-csv-2021.06.14"}}}}

How do I generate the fingerprint? I am not very clear on https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html

Thanks

[Iker (Iker Saint)]

If you don't have another pipeline to create the base documents this error happens because the document with the fingerprint specified doesn't exists yet, you have to add the option "doc_as_upsert" in your output to create or update:

    elasticsearch {
     hosts => [ "xx-xxx-xxx:3045" ]
     user => "xxxxx"
     password => "xxxxxxxxxxxxxxxx"
     index => "demo-csv-%{+YYYY.MM.dd}"
     doc_as_upsert => true
     action => "update"
     document_id => "%{my_fingerprint}"

About the fingerprint, is very easy to use, you specify which field contains a invariant text across all the document updates and just use the resulting hash as document_id:

filter {
	fingerprint {
		method => "SHA256"
		source => "[myfield]"
		target => "[my_fingerprint]"
  }
}

Check out this links:

plugins-outputs-elasticsearch-action
plugins-outputs-elasticsearch-doc_as_upsert

[zaeemmasood]

Thanks lker! It worked just fine. However I noticed that logstash consumed first couple of files and stopped consuming after 3 to 4.

Logstash is set up to start as:

./logstash -r -f $LOGSTASH_CONFIG/mqa.conf

[zaeemmasood]

Thanks lker. It really helped!