# Some logs are not being ingested to Elasticsearch index

**URL:** <https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184>\
**Category:** Elasticsearch\
**Created:** [May 9, 2018, 2:42pm UTC](https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184 "2018-05-09T14:42:06Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [May 9, 2018, 2:42pm UTC](https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184/1 "2018-05-09T14:42:06Z")

</div>

Hi,

I am experiencing a weird problem with Elasticsearch. Some logs are being ingested and some are not to a particular index. All other indexes are working perfectly fine.

I thought the problem was with index.mapping.total\_fields.limit and changed it to 2000 using,

```
PUT my_index/_settings
{
  "index.mapping.total_fields.limit": 2000
}

```

I don't have index template. Do I need to include one? Please let me know what the issue could be?

Thanks

---

<div class="post-metadata">

**Author:** ![ventrca](https://avatars.discourse-cdn.com/v4/letter/v/a87d85/32.png) [@ventrca](https://discuss.elastic.co/u/ventrca)\
**Post date:** [May 9, 2018, 3:15pm UTC](https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184/2 "2018-05-09T15:15:10Z")

</div>

Too few datails.

1. How do you feed data into ES? Logstash?
2. Can you identify the logs that are not ingested?

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [May 9, 2018, 4:19pm UTC](https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184/3 "2018-05-09T16:19:17Z")

</div>

> [@ventrca](#):
>
> How do you feed data into ES? Logstash?

Yes, with Logstash.

Syslog -Logstash1-Logstah2-Elasticsearch

> [@ventrca](#):
>
> Can you identify the logs that are not ingested?

yes I can. Format for missing and non-missing is exactly same.

I am thinking there is serious issue with this particular index but I don't see any errors in Elasticsearch log files.

---

<div class="post-metadata">

**Author:** ![ventrca](https://avatars.discourse-cdn.com/v4/letter/v/a87d85/32.png) [@ventrca](https://discuss.elastic.co/u/ventrca)\
**Post date:** [May 9, 2018, 4:28pm UTC](https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184/4 "2018-05-09T16:28:23Z")

</div>

What about logstash logs?

Maybe the format is the same but the values causes the problem.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [May 9, 2018, 4:30pm UTC](https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184/5 "2018-05-09T16:30:57Z")

</div>

No error logs in Logstash logs too.

What else I can do to find the issue?

---

<div class="post-metadata">

**Author:** ![ventrca](https://avatars.discourse-cdn.com/v4/letter/v/a87d85/32.png) [@ventrca](https://discuss.elastic.co/u/ventrca)\
**Post date:** [May 9, 2018, 4:42pm UTC](https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184/6 "2018-05-09T16:42:49Z")

</div>

I would try to manually insert the non-ingested doc into ES and see the response.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [May 9, 2018, 4:44pm UTC](https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184/7 "2018-05-09T16:44:34Z")

</div>

Okay, but why would index need to perform very slowly if the documents are not ingested? Shouldn't there be any underlying index related issue?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 6, 2018, 4:44pm UTC](https://discuss.elastic.co/t/some-logs-are-not-being-ingested-to-elasticsearch-index/131184/8 "2018-06-06T16:44:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.