Increase logstash logging

[Anabella_Cristaldi (Anna)]

Hi all,
I've been using logstash since February in order to index data about CDR (Call Detail Record) from our international central (basically a cvs with the data)
So far so good: logstash behaved in a stable way, with no problems.
But since last week I've notice that several lines in some files where not indexed. I've looked in the logstash-plain.log but I see no errors.

How Can I debug this behaviour?
Thank you very much in advance
Regards
Ana

[magnusbaeck (Magnus Bäck)]

If ES rejects a document Logstash will log that rejection. What does the elasticsearch output in your Logstash configuration look like?

[Anabella_Cristaldi (Anna)]

Hi Magnus,
I do not see neither rejection in logstash's log nor errors in the elasticsearch's log.
My ES output configuration is:

output {
    elasticsearch {
        hosts => [ "localhost:9200"]
        index => "temp_sbc2"
        document_id => "%{[@metadata][fingerprint]}"
    }
}

What I should specifically look into if it happens again?
Thank you
Regards
Ana

[magnusbaeck (Magnus Bäck)]

How is [@metadata][fingerprint] generated? Is it possible that two or more documents might've had the same id?

[Anabella_Cristaldi (Anna)]

Hi Magnus,

I Have already checked that, and in my analisis comparing the records in the file vs documents indexed I no not have into account the duplicated records in my file.

We do have some duplicates documents and we use this in order that duplicate elements have the same doc id

    fingerprint {
            source => "message"
            target => "[@metadata][fingerprint]"
            method => "MD5"
            key => "xxxxxxx"
    }

Thank you!
Regards
Ana