Kibana doesn't show logs

[wrkilu]

Hi,
I think this is some problem with timestamp of logs but I don't know how to solve this. Generally when I create index without 'contains time-based events ' (this checkbox during creating), then I see logs but of course without ability to choose according to date. In the other hand when I create index with this option (like on screenshot

) Kibana says 'no results found :(' . Application responsible for logs delivering is Filebeat.

The result of
curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty' is :

}, {
"_index" : "filebeat-2017.01.05",
"_type" : "syslog",
"_id" : "AVlu0pLnQQBiZCJUK7eL",
"_score" : 1.0,
"_source" : {
"message" : "10.172.81.39 - - [05/Jan/2017:04:51:10 +0100] "GET /.lvs.php HTTP/1.1" 200 8 "-" "check_http/v1.4.16 (nagios-plugins 1.4.16)"",
"@version" : "1",
"@timestamp" : "2017-01-05T13:30:07.131Z",
"count" : 1,
"fields" : null,
"beat" : {
"hostname" : "f1.smaker.rc.htp.iadm",
"name" : "f1.smaker.rc.htp.iadm"
},
"source" : "/var/log/httpd/smaker.pl.rc.htp-custom_log",
"type" : "syslog",
"input_type" : "log",
"offset" : 21294,
"host" : "f1.smaker.rc.htp.iadm",
"tags" : [ "beats_input_codec_plain_applied" ]
}
}, {
"_index" : "filebeat-2017.01.05",
"_type" : "syslog",
"_id" : "AVlu0pLnQQBiZCJUK7eQ",
"_score" : 1.0,
"_source" : {
"message" : "10.172.81.39 - - [05/Jan/2017:04:56:10 +0100] "GET /.lvs.php HTTP/1.1" 200 8 "-" "check_http/v1.4.16 (nagios-plugins 1.4.16)"",
"@version" : "1",
"@timestamp" : "2017-01-05T13:30:07.131Z",
"type" : "syslog",
"input_type" : "log",
"fields" : null,
"beat" : {
"hostname" : "f1.smaker.rc.htp.iadm",
"name" : "f1.smaker.rc.htp.iadm"
},
"source" : "/var/log/httpd/smaker.pl.rc.htp-custom_log",
"offset" : 21924,
"count" : 1,
"host" : "f1.smaker.rc.htp.iadm",
"tags" : [ "beats_input_codec_plain_applied" ]
}
} ]

What is going on ? Please help.

[jbudz (Jon Budzenski)]

If you increase the time range of the timepicker in the top right corner are you able to see any results?

[wrkilu]

I did that but nothing. Still no results.

[jbudz (Jon Budzenski)]

Can you share your mappings for the @timestamp field? In the UI, if you select "Use event times to create index names" do things work?

[wrkilu]

@jbudz

Can you share your mappings for the @timestamp field?

Hmm.. where its defined ? In Kibana I didn't change anything related with timestamp.

My output in Logstash looks like this:

output {
elasticsearch {
hosts => ["10.209.3.134:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+yyyy-MM-dd HH:mm:ss:SSSZ}"
document_type => "%{[@metadata][type]}"
}
}

However I see in logstash.log now:

"error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [filebeat-2017-01-10 11:48:13:775+0000], must not contain the following characters [\, /, *, ?, ", <, >, |, , ,]", "index"=>"filebeat-2017-01-10 11:48:13:775+0000"}}}, :level=>:warn}

@jbudz

In the UI, if you select "Use event times to create index names" do things work?

No, also nothing.

[jbudz (Jon Budzenski)]

Mappings can be found by requesting /filebeat-*/_mapping from your elasticsearch node.

Regarding the logstash output, I'm guessing you don't want a new index every millisecond. Daily and monthly indices are common ({+YYYY.MM.dd}, {+YYYY.MM}). The invalid character is the space.

[wrkilu]

Ok, I've solved the problem. I've changed only output.conf in Logstash from above to minimalistic:

output {
elasticsearch {
hosts => ["10.209.3.135:9200"]
}
}

...and it started work . What is interesting in logs (in Kibana) I have now index named (from default of course) " logstash-2017.01.11". Earlier I created filebeat-* so it couldn't work of course. On the other hand I created earlier also index "*" but it didn't show anyting either so this is strange.

Anyway... my first settings in output.conf were somehow exaggerated and thats why it didn't work.

I'm closing the case and many thanks to you Jon for enagagement in this post
wrkilu