Will frequent mapping exceptions slowdown my ingestion rate?

Elastic Stack Elasticsearch
1

Some times we have noticed that the consumer lag for certain logstash ingestion index is constantly growing, upon investigating we found that there are a few mapping exceptions on elasticsearch side for that particular index. But based on what I understood about bulk api, bad

based on the stats, during exceptions our index rate drops from 45k to 30k

2

That is very much possible. As due to exceptions documents may not get indexed if there are parsing exceptions/failures. Ideally, you should not have any mapping/parsing exceptions as you may end up having incomplete/incorrect data in the index.

3

I am not sure what effect mapping failures can have, but adding new dynamic fields can slow down indexing as the cluster state need to be updated and propagated for every change. The more fields you have got , the slower this typically gets.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.