# Increase logstash logging

**URL:** <https://discuss.elastic.co/t/increase-logstash-logging/138214>\
**Category:** Logstash\
**Created:** [July 2, 2018, 1:15pm UTC](https://discuss.elastic.co/t/increase-logstash-logging/138214 "2018-07-02T13:15:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anabella\_Cristaldi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anabella_cristaldi/32/23612_2.png) [@Anabella\_Cristaldi](https://discuss.elastic.co/u/Anabella_Cristaldi)\
**Post date:** [July 2, 2018, 1:15pm UTC](https://discuss.elastic.co/t/increase-logstash-logging/138214/1 "2018-07-02T13:15:16Z")

</div>

Hi all,  
I've been using logstash since February in order to index data about CDR (Call Detail Record) from our international central (basically a cvs with the data)  
So far so good: logstash behaved in a stable way, with no problems.  
But since last week I've notice that several lines in some files where not indexed. I've looked in the logstash-plain.log but I see no errors.

How Can I debug this behaviour?  
Thank you very much in advance  
Regards  
Ana

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 2, 2018, 2:09pm UTC](https://discuss.elastic.co/t/increase-logstash-logging/138214/2 "2018-07-02T14:09:27Z")

</div>

If ES rejects a document Logstash will log that rejection. What does the elasticsearch output in your Logstash configuration look like?

---

<div class="post-metadata">

**Author:** ![Anabella\_Cristaldi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anabella_cristaldi/32/23612_2.png) [@Anabella\_Cristaldi](https://discuss.elastic.co/u/Anabella_Cristaldi)\
**Post date:** [July 3, 2018, 7:03am UTC](https://discuss.elastic.co/t/increase-logstash-logging/138214/3 "2018-07-03T07:03:17Z")

</div>

Hi Magnus,  
I do not see neither rejection in logstash's log nor errors in the elasticsearch's log.  
My ES output configuration is:

```
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "temp_sbc2"
        document_id => "%{[@metadata][fingerprint]}"
    }
}

```

What I should specifically look into if it happens again?  
Thank you  
Regards  
Ana

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 3, 2018, 7:16am UTC](https://discuss.elastic.co/t/increase-logstash-logging/138214/4 "2018-07-03T07:16:19Z")

</div>

How is `[@metadata][fingerprint]` generated? Is it possible that two or more documents might've had the same id?

---

<div class="post-metadata">

**Author:** ![Anabella\_Cristaldi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anabella_cristaldi/32/23612_2.png) [@Anabella\_Cristaldi](https://discuss.elastic.co/u/Anabella_Cristaldi)\
**Post date:** [July 3, 2018, 7:26am UTC](https://discuss.elastic.co/t/increase-logstash-logging/138214/5 "2018-07-03T07:26:56Z")

</div>

Hi Magnus,

I Have already checked that, and in my analisis comparing the records in the file vs documents indexed I no not have into account the duplicated records in my file.

We do have some duplicates documents and we use this in order that duplicate elements have the same doc id

```
    fingerprint {
            source => "message"
            target => "[@metadata][fingerprint]"
            method => "MD5"
            key => "xxxxxxx"
    }

```

Thank you!  
Regards  
Ana

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 31, 2018, 7:33am UTC](https://discuss.elastic.co/t/increase-logstash-logging/138214/6 "2018-07-31T07:33:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.