Dead queue log files

Elastic Stack Elasticsearch
1

Hi, ELK Community please help me to resolve this issue related to delete log files in dead queue.

I reindex the documents logged in dead queue logs, but still the size of log files is still the same? what should i do of those log files ? Do i have to delete them manually ? How can validate if the logged documents in dead queue are now reindexed or not? if i have a huge no of mapping errors logged in dead queue, Please Help.

2

I would verify number of Documents.
If you are getting mapping pasers errors, dcouments are not getting indexed. If you using default template the data might be coming in different format. For eg. something that came in earlier as number is coming now as text or otherwise..
Also if failed indexing messages goes to DLQ in your setup, than all of those failed message wold be back to DLQ.

Hope this helps, otherwise try posting mapping errors.

3

Hi @jay224 I added input pipeline to read from dead Q with a filter to fix mapping errors it seems like working when i tested it with small data all dead Q documents are now indexed into elasticsearch.
I just want to confirm is there any way to verify if the documents logged in Dead Q are reindex ?
The size of dead queue logs remains same even after doing above steps can you help with any of above problems ?

4

Upon succesful ingestion you should delete message from DLQ, just like you probably doing for actual queue.
Check cloudwatch log , depending on how much you are looging you should be able to verify successful execution or error in cloudwatch logs.

1 Like
5

Thanks @jay224 i will look into this tomorrow, then will share my thoughts with you.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.