Kibana not showing the Latest logs


#1

Hi,
I am using Elasticsearch/Kibana v 6.4.1
I configured my environment in such a way that I am using syslog-ng to push logs into elasticsearch and then display in kibana. The problem I am facing right now is that I was able to view my logs up-till Oct 3rd, 2018 and after that it just stopped working.
This is the first time I am working with Kibana and Elasticsearch. your help in this regards will be highly appreciated.
Below are the logs I have

elasticsearch]# sudo tail -f gc.log.0.current
2018-10-10T13:52:22.820-0400: 1286493.354: Total time for which application threads were stopped: 0.0062639 seconds, Stopping threads took: 0.0000711 seconds
2018-10-10T13:52:28.799-0400: 1286499.332: Total time for which application threads were stopped: 0.0007719 seconds, Stopping threads took: 0.0001206 seconds
2018-10-10T13:52:33.837-0400: 1286504.370: [GC (Allocation Failure) 2018-10-10T13:52:33.837-0400: 1286504.370: [ParNew
Desired survivor size 8716288 bytes, new threshold 6 (max 6)

  • age 1: 483856 bytes, 483856 total
  • age 2: 32 bytes, 483888 total
  • age 3: 64 bytes, 483952 total
  • age 4: 6576 bytes, 490528 total
  • age 6: 576 bytes, 491104 total

curl -XGET http://localhost:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana aibdk5WFTT65EiOh8ed0gg 1 0 3 0 19.8kb 19.8kb
yellow open syslog-ng nKG09rgtTiumAW5vZPbySw 5 1 65378579 0 9.1gb 9.1gb

curl -XGET http://localhost:9200/syslog-ng

{"syslog-ng":{"aliases":{},"mappings":{"test":{"properties":{"0":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"FACILITY":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"FILE_NAME":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"HOST":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"HOST_FROM":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ISODATE":{"type":"date"},"LEGACY_MSGHDR":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"MESSAGE":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"PID":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"PRIORITY":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"PROGRAM":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"SOURCE":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"_SDATA":{"properties":{"meta":{"properties":{"sequenceId":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"amsg":{"properties":{"acct":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"addr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"cipher":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"cmd":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"comm":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"cwd":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"default-context":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"direction":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exe":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fp":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"grantors":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"grp":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"hostname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"kind":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ksize":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"laddr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"lport":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"mac":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"op":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"pfs":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"res":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"rport":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"selected-context":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"spid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"suid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"terminal":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"unit":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"auditd":{"properties":{"a0":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"a1":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"a2":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"a3":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"arch":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"auid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"comm":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"dev":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"egid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"euid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exe":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exit":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsgid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}},"settings":{"index":{"number_of_shards":"5","blocks":{"read_only_allow_delete":"true"},"provided_name":"syslog-ng","creation_date":"1537284866694","number_of_replicas":"1","uuid":"nKG09rgtTiumAW5vZPbySw","version":{"created":"6040099","upgraded":"6040199"}}}}}

curl -XGET http://localhost:9200/syslog-ng-2018-10-10

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"syslog-ng-2018-10-10","index_uuid":"na","index":"syslog-ng-2018-10-10"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"syslog-ng-2018-10-10","index_uuid":"na","index":"syslog-ng-2018-10-10"},"status":404}

Looking forward to hearing from you soon.


#2

curl -XGET http://localhost:9200/syslog-ng 2018-10-10
{"syslog-ng":{"aliases":{},"mappings":{"test":{"properties":{"0":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"FACILITY":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"FILE_NAME":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"HOST":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"HOST_FROM":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ISODATE":{"type":"date"},"LEGACY_MSGHDR":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"MESSAGE":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"PID":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"PRIORITY":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"PROGRAM":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"SOURCE":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"_SDATA":{"properties":{"meta":{"properties":{"sequenceId":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"amsg":{"properties":{"acct":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"addr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"cipher":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"cmd":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"comm":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"cwd":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"default-context":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"direction":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exe":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fp":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"grantors":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"grp":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"hostname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"kind":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ksize":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"laddr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"lport":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"mac":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"op":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"pfs":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"res":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"rport":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"selected-context":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"spid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"suid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"terminal":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"unit":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"auditd":{"properties":{"a0":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"a1":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"a2":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"a3":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"arch":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"auid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"comm":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"dev":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"egid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"euid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exe":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"exit":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsgid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsuid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"gid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ino":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"items":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"key":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"msg":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"old-auid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"old-ses":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"pid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ppid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"proctitle":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"res":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"scontext":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"ses":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sgid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"subj":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"success":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"suid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"syscall":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tclass":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tcontext":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tty":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"uid":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}},"settings":{"index":{"number_of_shards":"5","blocks":{"read_only_allow_delete":"true"},"provided_name":"syslog-ng","creation_date":"1537284866694","number_of_replicas":"1","uuid":"nKG09rgtTiumAW5vZPbySw","version":{"created":"6040099","upgraded":"6040199"}}}}}curl: (6) Could not resolve host: 2018-10-10; Unknown error


(Marius Dragomir) #3

Sounds like there is a problem between syslog and Elasticsearch since the index doesn't sem to be present for October 10th (from your request). Any information in the Elasticsearch logs? Maybe the disk full watermark hit and closed the node for writing or some kind of connection problem.


#4

Thanks Marius,
I am having issue locating Elasticsearch logs can you guide me where can I find them?


(Marius Dragomir) #5

it depends on what method you used to install Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html and if you set any specific path for the logs.


#6

@Marius
I uninstalled and re installed elasticsearch, deleted the index from kibana and added it again, and started getting the latest logs. What could be the reason to have caused the elasticsearch not to process the logs and display in Kibana.

Also what I have notices is that there are a lot of dropped messages drop messages,even with the new install I am getting a lot of drop messages, would you be able to assist me with this issue.
I believe this whole issue was caused by huge the number of dropped messages. (just my observation)

$ sudo syslog-ng-ctl stats

SourceName;SourceId;SourceInstance;State;Type;Number
source;s_network;;a;processed;545
destination;d_elastic;;a;processed;545
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;dropped;37
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;processed;545
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;queued;0
dst.java;d_elastic#0;java_dst,ElasticSearch_v2,syslog-ng >> /etc/elasticsearch/elasticsearch.yml,syslog-ng;a;written;508
center;;received;a;processed;545
center;;queued;a;processed;545
global;scratch_buffers_count;;a;queued;167503724576
global;payload_reallocs;;a;processed;4
global;sdata_updates;;a;processed;545
global;scratch_buffers_bytes;;a;queued;0
global;msg_clones;;a;processed;0

looking forward to here from you soon.


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.