Some logs are not being ingested to Elasticsearch index

Elastic Stack Elasticsearch
1

Hi,

I am experiencing a weird problem with Elasticsearch. Some logs are being ingested and some are not to a particular index. All other indexes are working perfectly fine.

I thought the problem was with index.mapping.total_fields.limit and changed it to 2000 using,

PUT my_index/_settings
{
  "index.mapping.total_fields.limit": 2000
}

I don't have index template. Do I need to include one? Please let me know what the issue could be?

Thanks

2

Too few datails.

  1. How do you feed data into ES? Logstash?
  2. Can you identify the logs that are not ingested?
3

Yes, with Logstash.

Syslog -Logstash1-Logstah2-Elasticsearch

yes I can. Format for missing and non-missing is exactly same.

I am thinking there is serious issue with this particular index but I don't see any errors in Elasticsearch log files.

4

What about logstash logs?

Maybe the format is the same but the values causes the problem.

5

No error logs in Logstash logs too.

What else I can do to find the issue?

6

I would try to manually insert the non-ingested doc into ES and see the response.

7

Okay, but why would index need to perform very slowly if the documents are not ingested? Shouldn't there be any underlying index related issue?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.