Some logs are not being ingested to Elasticsearch index


(Amruth) #1

Hi,

I am experiencing a weird problem with Elasticsearch. Some logs are being ingested and some are not to a particular index. All other indexes are working perfectly fine.

I thought the problem was with index.mapping.total_fields.limit and changed it to 2000 using,

PUT my_index/_settings
{
  "index.mapping.total_fields.limit": 2000
}

I don't have index template. Do I need to include one? Please let me know what the issue could be?

Thanks


#2

Too few datails.

  1. How do you feed data into ES? Logstash?
  2. Can you identify the logs that are not ingested?

(Amruth) #3

Yes, with Logstash.

Syslog -Logstash1-Logstah2-Elasticsearch

yes I can. Format for missing and non-missing is exactly same.

I am thinking there is serious issue with this particular index but I don't see any errors in Elasticsearch log files.


#4

What about logstash logs?

Maybe the format is the same but the values causes the problem.


(Amruth) #5

No error logs in Logstash logs too.

What else I can do to find the issue?


#6

I would try to manually insert the non-ingested doc into ES and see the response.


(Amruth) #7

Okay, but why would index need to perform very slowly if the documents are not ingested? Shouldn't there be any underlying index related issue?


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.