# Logstash output elasticsearch - document\_id double document produced

**URL:** <https://discuss.elastic.co/t/logstash-output-elasticsearch-document-id-double-document-produced/83995>\
**Category:** Logstash\
**Created:** [April 28, 2017, 10:09am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-document-id-double-document-produced/83995 "2017-04-28T10:09:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nico-DF](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@Nico-DF](https://discuss.elastic.co/u/Nico-DF)\
**Post date:** [April 28, 2017, 10:09am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-document-id-double-document-produced/83995/1 "2017-04-28T10:09:33Z")

</div>

Hi, I have the following thing I'm trying to do:

I a log file in .csv that contains info about a server (localisation, etc.)  
And along with this .csv, I got several folder (typically, 1/day) that contains a .txt file with its status (Up/Down)  
So I want to have in ES my csv data plus the updated field status from this other input.

* * *

I've parsed and stored successfully the .csv, as for the txt, I'm using a multiline with a timeout to read the whole file and have all the info.

For both files, the Path contains an ID that I will use in the document\_id.

So for the output, I specify to send it to ES with this document\_id, and added a script to add the status field into the document.

But with my current configuration, **I end up with 2 documents for the same doc\_id.**  
Do I miss something?

* * *

**Input filter**

```
file{
    path => "Some/path/UID/*/*/*.csv"
    start_position => "beginning"
    tags => ["to_update", "to_search"]
    sincedb_path => "NUL"
}

file{
    path => "Some/path/UID/*/*/*/Monitoring*"
    start_position => "beginning"
    sincedb_path => "NUL"
    close_older => 15
    tags => ["to_update"]
    codec => multiline {
        pattern => "/.*./g"
        negate => true
        what => "previous"
    }

```

* * *

**CSV filter for UID**  
(partial)

```
ruby { 
    code => "event.set('doc_id', event.get('path').split('/')[-2])"
}

```

* * *

**TXT filter for UID**  
(partial)

```
ruby { 
    code => " event.set('tmp_id', event.get('path').split('/')[-3]) "
}

```

or

```
elasticsearch {
        query => "tags:'to_search' AND doc_id:'%{tmp_id}'"
        fields => { "doc_id" => "doc_id" }
        remove_field => ["tmp_id"]
}

```

(tried both)

* * *

**Output Filter**

```
elasticsearch { 			
            hosts => ["localhost:9200"]  
            index => "logstash-mydoc"
            action => "update"
            document_id => "%{doc_id}"
            doc_as_upsert => true
            script_lang => "painless"
            script_type => "inline"
            script => 'ctx._source.status = params.event.get("status")'
}

```

And there is no error on output (checked with stdout { codec =\> rubydebug {} })

---

<div class="post-metadata">

**Author:** ![Nico-DF](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@Nico-DF](https://discuss.elastic.co/u/Nico-DF)\
**Post date:** [April 28, 2017, 12:42pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-document-id-double-document-produced/83995/2 "2017-04-28T12:42:43Z")

</div>

**Edit**

Problem solved.  
As \_type was not the same, it wasn't merged correctly.  
After fix, works like a charm

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 26, 2017, 12:58pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-document-id-double-document-produced/83995/3 "2017-05-26T12:58:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.