Filebeat missing log lines - resolved logstash version problem

[Tim_Burt]

Here is an example:

Logs are being processed through two pathways.

ruby-app --> TCP --> OldLogstash --> OldKibana

ruby-app --> /var/log/json/file.log --> filebeats --> kafka --> NewLogstash --> NewKibana

Tracing one log instance (Tonya-Ca....) from two log files, a metrics log and a request log:

In this kibana screen you can see the double entries for each ruby activity:

My focus is on the Tonya-Ca... that appears mid screen.

Here is the same time period from the NewKibana:

Note the identical timestamp 16:36:30.619 on the Tonya-Cole......
Note that in the OldKibana, there are many lines missing between James-Tow... and Tonya-Cole.
We are seeing about 8 in 10 lines lost, which approximately coincides with the screenshots.

In Kafka, there is only one entry recorded:

@timestamp":"2016-10-14T23:36:31.084Z","beat":{"hostname":"ip-172-31-51-10","name":"ip-172-31-51-10","version":"6.0.0-alpha1"},"fields":{"App":"masscache","Availzone":"us-west-2a","Beat":"masscache","Env":"production","Imageid":"ami-e5875985","Insttype":"c4.xlarge","Ipaddr":"172.31.51.10","Role":"appserver"},"input_type":"log","message":"{\"name\":\"masscache\",\"environment\":\"production\",\"hostname\":\"ip-172-31-51-10\",\"pid\":2814,\"level\":30,\"message\":\"GET /Tonya-ColXXXXX 200 HIT\",\"req_id\":\"ec9b2204-8221-4c88-ba5e-a305b9b5f00c\",\"agent\":\"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)\",\"clientip\":\"207.46.13.163\",\"verb\":\"GET\",\"origin\":\"http://www.XXXX.com\",\"request\":\"/Tonya-ColXXXXX\",\"page_type\":\"name\",\"band_segment\":8,\"query\":{},\"response\":200,\"cache_status\":\"HIT\",\"cache_key\":\"cd014d74703d89f0d8ee06c170db1d7db94b0f5a\",\"totalms\":27.26,\"program\":\"masscache\",\"metrics\":{},\"timestamp\":\"2016-10-14 23:36:30.619\",\"msg\":\"\",\"time\":\"2016-10-14T23:36:30.619Z\",\"v\":0}","offset":2871070136,"source":"/var/log/json/request.log","type":"json"}

[Tim_Burt]

There is a flaw in this analysis... I need to do more research.... tomorrow.

[ruflin]

@Tim_Burt Can you edit your post with 3 ` (ticks) before and after the code parts to make it more readable?

Before I do some investigations, I wait for your update.

[Tim_Burt]

I have been able to confirm that this occurance is true. I have traced through another example and I can confirm that lines are missing from Kafka.

Log lines that are present in the logs, do not appear in Kafka.

[tudor (Tudor Golubenco)]

The logs indicate that you are using the nightlies in this test. Is that correct?

[ruflin]

@Tim_Burt Did you find any pattern on which log lines are missing? It comes back to the question: Is it filebeat logic or is it output / Kafka logic? A little bit a problem with the setup above is that we have lots of parts inside which makes it quite hard to detect the problem (it could be FB, LS, Kafka, etc). Best would be if we would find a reproducible example or could isolate the problem to just filebeat which would mean test it with file output.

[Tim_Burt]

I think I was too focused on beats... I was finally able to trace an example through Kafka and conclude that beats is properly forwarding my logs. I did make a few changes to the beats configs, so who knows... Maybe I fixed a config too.

In the end, I found that my Logstash version (2.3) was not version matched with the Kafka input plugin and the Kafka version (0.10) I was using. I upgraded the Logstash to 2.4 and my missing log line problems have gone away.

Release 5.0 will be a blessing.. Good to know your team has this planned.

Thanks again for all your help...

[Tim_Burt]

I pulled the master branch from github and compiled locally after confirming that the fix I needed was included.

[ruflin]

Glad you got it working and thanks for keeping investigating.