Kibana not showing the Latest logs

JawadKM · October 10, 2018, 6:42pm

Hi,
I am using Elasticsearch/Kibana v 6.4.1
I configured my environment in such a way that I am using syslog-ng to push logs into elasticsearch and then display in kibana. The problem I am facing right now is that I was able to view my logs up-till Oct 3rd, 2018 and after that it just stopped working.
This is the first time I am working with Kibana and Elasticsearch. your help in this regards will be highly appreciated.
Below are the logs I have

elasticsearch]# sudo tail -f gc.log.0.current
2018-10-10T13:52:22.820-0400: 1286493.354: Total time for which application threads were stopped: 0.0062639 seconds, Stopping threads took: 0.0000711 seconds
2018-10-10T13:52:28.799-0400: 1286499.332: Total time for which application threads were stopped: 0.0007719 seconds, Stopping threads took: 0.0001206 seconds
2018-10-10T13:52:33.837-0400: 1286504.370: [GC (Allocation Failure) 2018-10-10T13:52:33.837-0400: 1286504.370: [ParNew
Desired survivor size 8716288 bytes, new threshold 6 (max 6)

age 1: 483856 bytes, 483856 total
age 2: 32 bytes, 483888 total
age 3: 64 bytes, 483952 total
age 4: 6576 bytes, 490528 total
age 6: 576 bytes, 491104 total

curl -XGET http://localhost:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana aibdk5WFTT65EiOh8ed0gg 1 0 3 0 19.8kb 19.8kb
yellow open syslog-ng nKG09rgtTiumAW5vZPbySw 5 1 65378579 0 9.1gb 9.1gb

curl -XGET http://localhost:9200/syslog-ng

curl -XGET http://localhost:9200/syslog-ng-2018-10-10

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"syslog-ng-2018-10-10","index_uuid":"na","index":"syslog-ng-2018-10-10"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"syslog-ng-2018-10-10","index_uuid":"na","index":"syslog-ng-2018-10-10"},"status":404}

Looking forward to hearing from you soon.

JawadKM · October 10, 2018, 6:46pm

curl -XGET http://localhost:9200/syslog-ng 2018-10-10
{"syslog-ng":{"aliases":{},"mappings":{"test":{"properties":{"0":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"FACILITY":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"FILE_NAME":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"HOST":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"HOST_FROM":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ISODATE":{"type":"date"},"LEGACY_MSGHDR":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"MESSAGE":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"PID":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"PRIORITY":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"PROGRAM":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"SOURCE":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"_SDATA":{"properties":{"meta":{"properties":{"sequenceId":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"amsg":{"properties":{"acct":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"addr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"cipher":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"cmd":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"comm":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"cwd":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"default-context":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"direction":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exe":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fp":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"grantors":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"grp":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"hostname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"kind":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ksize":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"laddr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"lport":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"mac":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"op":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"pfs":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"res":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"rport":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"selected-context":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"spid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"suid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"terminal":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"unit":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"auditd":{"properties":{"a0":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"a1":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"a2":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"a3":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"arch":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"auid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"comm":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"dev":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"egid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"euid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exe":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exit":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsgid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsuid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"gid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ino":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"items":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"key":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"msg":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"old-auid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"old-ses":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ppid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"proctitle":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"res":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"scontext":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ses":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sgid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"subj":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"success":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"suid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"syscall":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tclass":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tcontext":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tty":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"uid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}},"settings":{"index":{"number_of_shards":"5","blocks":{"read_only_allow_delete":"true"},"provided_name":"syslog-ng","creation_date":"1537284866694","number_of_replicas":"1","uuid":"nKG09rgtTiumAW5vZPbySw","version":{"created":"6040099","upgraded":"6040199"}}}}}curl: (6) Could not resolve host: 2018-10-10; Unknown error

Marius_Dragomir · October 11, 2018, 12:45pm

Sounds like there is a problem between syslog and Elasticsearch since the index doesn't sem to be present for October 10th (from your request). Any information in the Elasticsearch logs? Maybe the disk full watermark hit and closed the node for writing or some kind of connection problem.

JawadKM · October 11, 2018, 3:14pm

Thanks Marius,
I am having issue locating Elasticsearch logs can you guide me where can I find them?

Marius_Dragomir · October 14, 2018, 2:38pm

it depends on what method you used to install Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html and if you set any specific path for the logs.

JawadKM · October 22, 2018, 6:46pm

@Marius
I uninstalled and re installed elasticsearch, deleted the index from kibana and added it again, and started getting the latest logs. What could be the reason to have caused the elasticsearch not to process the logs and display in Kibana.

Also what I have notices is that there are a lot of dropped messages drop messages,even with the new install I am getting a lot of drop messages, would you be able to assist me with this issue.
I believe this whole issue was caused by huge the number of dropped messages. (just my observation)

$ sudo syslog-ng-ctl stats

SourceName;SourceId;SourceInstance;State;Type;Number
source;s_network;;a;processed;545
destination;d_elastic;;a;processed;545
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;dropped;37
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;processed;545
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;queued;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;written;508
center;;received;a;processed;545
center;;queued;a;processed;545
global;scratch_buffers_count;;a;queued;167503724576
global;payload_reallocs;;a;processed;4
global;sdata_updates;;a;processed;545
global;scratch_buffers_bytes;;a;queued;0
global;msg_clones;;a;processed;0

looking forward to here from you soon.

system · November 19, 2018, 6:51pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.