Dead_letter_queue

Elastic Stack Logstash
1

Hello,

Actually I'm using a cluster of Amazon ElasticSearch and I'm trying to load some documents from S3 (input) using logstash.

I've one pipeline (file.conf) with 25 inputs , one for each different folder in S3.

The ouput is AmazonElasticSearch.

When logstash run I can see in my index that logstash yes index the documents but sometimes I Can see that not all documents.

I try to enable dead_letter_queue to know what queues anda data I'm lossing , but never see anything.

I change this parameters in my logstash.yml:

dead_letter_queue.enable: true

queue.type: persisted

I'm not sure if I need change another configuration or something.. ?

Thanks for advanced.

King regards.

2

I believe not all output plugins support the dead letter queue, and it is possible that the amazon_es plugin (if that is what you are using) is one of them.

3

Yes I'm using amazon_es. How I can check if this ouput accept dead_letter_queue ?

Thanks

4

I would recommend you check with Amazon.

5

Hi christian, finally amazon_e snot support dead_letter_quee.

I've many doubts about if I use elasticsearch ouput with dead_letter_queue...

The problem tha tI0ve is that when I idnex documetns with logstash I loss some documents, but not is a problem of mapping, I think that is a problem of CPU logstash or something, because in logs of logstash I don't have any error or warning....

If I use elasticsearch liek output, I will see this docuemtns in dead_letter_queue or only insert this with errors when index documents ... ?

Another doubt that I've is that my .conf file IO've 23 differents inputs...where is better 1 file with many inputs or 1 input with 1 file.conf .... ?

Thansk foradvanced.

6

And I've another doubt.... It's possible have many logstash and run this logstash sequencly... ?

7

Hi, I'm tryin gto use elasticsearch and not use amazon_es , but I follow to see anything in dead_letter_queue ....

I use PERSISTEND queue type and I enable the dead_letter_queue : true ...

How I can enable correctly dead_letter_queue ... ? I try to index json with mapping error but I can't see in dead_letter_queue.

Thanks for advanced.

King regards.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.