Troubles with elastic + logstash + rabbit

Elastic Stack Logstash
#1

Hi everybody,
We are using elastic stack for storing logs data.
We use rabbitmq as an input for logstash.

For some reasons our elasticsearch cluster has been down for several times. And logstash was unable to deliver massages, but them also gone from rabbitmq.
What should I do in such cases to prevent message disappearing?

#2

This shouldn't be a problem. Logstash should halt the pipeline when ES is unavailable. Do you have any logs that can give clues into what happened?

#3

Thanks for your response.
It seems that logstash does not do that. May be I need to specify some option in logstash.conf ?

The issue is probably dots-contained filed keys.

MapperParsingException[Field name [f5269b709d9646188e77c14467419d5bYii.User.flash.success] cannot contain '.']

Can it be the cause of elasticsearch failing ?

#4

Yes, that would explain things.

#5

In addition to containing dots, the first part looks like a potentially dynamic component. If that is the case, be aware that this can cause mapping explosion.

#6

Thanks! In that case I won't have to worry about dot contained keys cuz they would be the values =)

#7