Logs that are visible in stdout not showing up in es/kibana

Elastic Stack Logstash
1

My output plugin is using BOTH
stdout { codec => rubydebug }
and an elasticsearch output connecting to our cluster, I do this sometimes to help me debug.

When I run a test script to send a bunch of data to logstash, with stdout enabled, I am having the incredibly strange experience where I see all of the data in my stdout screen, looking directly into logstash, and yet only about half of the data shows up in the corresponding index in kibana. Neither logstash nor elasticsearch produces any logs during this transaction.

What are possible explanations for disappearing logs on indexing??

thanks

2

Does any of this data contain timestamps? And are you using the date filter to replace @timestamp with it or even having Kibana view that field for the time? When I first started with Elastic Stack I tested inputting some data and some of those events weren't appearing. That was because some of them had old dates. Even now I get some events coming in with a bad UNIX timestamp (they come in with a date in 1970). One thing that helped me figure this out was the fact that I had daily indexes with the format name-YYYY.MM.dd. The events with a bad UNIX timestamp went into index name-1970.01.01. When I looked at all my indexes that stuck out because it shouldn't be there. In Kibana I looked at the range from December 31, 1969 to January 2, 1970 and sure enough some events appeared and from there I saw they had a bad timestamp.
Although this might not even apply if you don't use the date filter or have kibana set to view that specific field. But if you are then what I am saying is to look at your data's timestamps as well as your indicies (if they're separated by dates) and that might give you a clue as to why they aren't appearing.

1 Like
3

Holy shit you're right!!! We didn't have any wrong timestamps but we had an inaccurate mapping where we were being sent a timestamp in 12-hour time, and our configuration expects 24 hour time, which means whenever we sent it 4pm, (16:00 in 24 hour), it just sees 4:00 as 4:00am and indexes it 12 hours previous to now.

DAAAAMN.

Thanks so much @marke72!!!

4

Glad to hear that was it. It was long shot but if it happened to me I'm sure it could of happened to someone else.

5

Yeah at least the process of expanding your time window to check for [any possible type of timestamping issue] ought to be a basic tool in anybody's toolbox when looking for missing logs. At least it's gonna be in mine now!

1 Like