How make multi shipping without duplicates

Elastic Stack Logstash
1

Hello everybody,

I was try it some examples (architectural) from the documentation : https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html#deploying-scaling

But I have now a problem : The multi shipping from one or many source creates duplicates logs.
Of course, if I have 3 shpping binding one same source, i will have 3 same logs.
Has you can see below, an example of the infrastructure I trying to make:

https://www.elastic.co/guide/en/logstash/current/static/images/deploy_7.png

Now, I want to remove the duplicates logs from differente shipping (Have 1 log on the queue, not 3 same logs because I have 3 shipping on the same source).

Someone can help me ?

Thank you to read me and thank you a lot for the reponse(s) ! :grinning:

2

You have multiple LS shippers reading a single source?

3

Hello warkolm and thank you to reply me,

Yes, I have multiple LS shippers reading a single source (avoiding the Single Point Of Failure).

For exemple, I have 3 LS shippers reading a same channel on one IRC. The result : I have 3 duplicates messages.

Here somes pictures :

The configuration of LS shippers number 2 (the other has the same idea, just user/real/nick change) :

The result (duplicates logs):

Pasted image1198×86 29.3 KB

4

Well that'd be why, the IRC input is pretty dumb in that regards.

Take a look at https://www.elastic.co/guide/en/logstash/current/plugins-filters-checksum.html, it may help.