Logstash output elasticsearch - document_id double document produced

Elastic Stack Logstash
1

Hi, I have the following thing I'm trying to do:

I a log file in .csv that contains info about a server (localisation, etc.)
And along with this .csv, I got several folder (typically, 1/day) that contains a .txt file with its status (Up/Down)
So I want to have in ES my csv data plus the updated field status from this other input.

I've parsed and stored successfully the .csv, as for the txt, I'm using a multiline with a timeout to read the whole file and have all the info.

For both files, the Path contains an ID that I will use in the document_id.

So for the output, I specify to send it to ES with this document_id, and added a script to add the status field into the document.

But with my current configuration, I end up with 2 documents for the same doc_id.
Do I miss something?

Input filter

file{
    path => "Some/path/UID/*/*/*.csv"
    start_position => "beginning"
    tags => ["to_update", "to_search"]
    sincedb_path => "NUL"
}

file{
    path => "Some/path/UID/*/*/*/Monitoring*"
    start_position => "beginning"
    sincedb_path => "NUL"
    close_older => 15
    tags => ["to_update"]
    codec => multiline {
        pattern => "/.*./g"
        negate => true
        what => "previous"
    }

CSV filter for UID
(partial)

ruby { 
    code => "event.set('doc_id', event.get('path').split('/')[-2])"
}

TXT filter for UID
(partial)

ruby { 
    code => " event.set('tmp_id', event.get('path').split('/')[-3]) "
}

or

elasticsearch {
        query => "tags:'to_search' AND doc_id:'%{tmp_id}'"
        fields => { "doc_id" => "doc_id" }
        remove_field => [ "tmp_id" ]
}

(tried both)

Output Filter

elasticsearch { 			
            hosts => ["localhost:9200"]  
            index => "logstash-mydoc"
            action => "update"
            document_id => "%{doc_id}"
            doc_as_upsert => true
            script_lang => "painless"
            script_type => "inline"
            script => 'ctx._source.status = params.event.get("status")'
}

And there is no error on output (checked with stdout { codec => rubydebug {} })

2

Edit

Problem solved.
As _type was not the same, it wasn't merged correctly.
After fix, works like a charm

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.