# Disappearing log records: what can I do?

**URL:** <https://discuss.elastic.co/t/disappearing-log-records-what-can-i-do/22267>\
**Category:** Elasticsearch\
**Created:** [February 19, 2015, 2:15pm UTC](https://discuss.elastic.co/t/disappearing-log-records-what-can-i-do/22267 "2015-02-19T14:15:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![r\_grosmann](https://avatars.discourse-cdn.com/v4/letter/r/3d9bf3/32.png) [@r\_grosmann](https://discuss.elastic.co/u/r_grosmann)\
**Post date:** [February 19, 2015, 2:15pm UTC](https://discuss.elastic.co/t/disappearing-log-records-what-can-i-do/22267/1 "2015-02-19T14:15:32Z")

</div>

hello group,

I have the following stack:  
fluentd (td-agent 2.1.3), elasticsearch (1.4.4), kibana (3.1.2)  
to manage the logging of an in-company application.

At first glance, this seems to work OK, but it appears that from time to  
time, the reported records in kibana don't match the line count of the  
logfiles.

Diving into this, it appears that when very large logfile are put in the  
flluentd logdirectory, not all records show up in elasticsearch. This does  
not show up in the logging of either fluentd or elasticsearch, so at first  
glance, everyting seems fine. I started with looking at fluentd and managed  
to get extra information, which seems to indicate that all of the log lines  
are processed.

When comparing the wc -l of the logfile and the contents of ES, the  
difference becomes visible:  
ES: 645551 wc: 647506 groot.log

Looking at the thread pool statistics with the REST api, ES reports 60  
bulk.rejects.

Right now, I have a very simple configuration.  
cluster.name: cwc-dev  
index.number\_of\_replicas: 0  
index.indexing.slowlog.threshold.index.warn: 10s  
index.indexing.slowlog.threshold.index.info: 5s  
index.indexing.slowlog.threshold.index.debug: 2s  
path.data: /data/elasticsearch

I hope you can support me on how to tackle this, since I am quite new to  
ES. So I don't know which ways are available to get extra information on  
this.

thanks in advance, Ruud

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/15a4958b-9bb8-4ca8-852b-967e15111305%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/15a4958b-9bb8-4ca8-852b-967e15111305%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:31am UTC](https://discuss.elastic.co/t/disappearing-log-records-what-can-i-do/22267/2 "2017-07-06T00:31:32Z")

</div>