hello group,
I have the following stack:
fluentd (td-agent 2.1.3), elasticsearch (1.4.4), kibana (3.1.2)
to manage the logging of an in-company application.
At first glance, this seems to work OK, but it appears that from time to
time, the reported records in kibana don't match the line count of the
logfiles.
Diving into this, it appears that when very large logfile are put in the
flluentd logdirectory, not all records show up in elasticsearch. This does
not show up in the logging of either fluentd or elasticsearch, so at first
glance, everyting seems fine. I started with looking at fluentd and managed
to get extra information, which seems to indicate that all of the log lines
are processed.
When comparing the wc -l of the logfile and the contents of ES, the
difference becomes visible:
ES: 645551 wc: 647506 groot.log
Looking at the thread pool statistics with the REST api, ES reports 60
bulk.rejects.
Right now, I have a very simple configuration.
cluster.name: cwc-dev
index.number_of_replicas: 0
index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
path.data: /data/elasticsearch
I hope you can support me on how to tackle this, since I am quite new to
ES. So I don't know which ways are available to get extra information on
this.
thanks in advance, Ruud
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/15a4958b-9bb8-4ca8-852b-967e15111305%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.