Elasticsearch logs are not Logstash friendly

Elastic Stack Elasticsearch
1

If you've ever dissected a strange log file, you know the pain something like this brings:

[2015-12-22 23:59:40,262][INFO ][monitor.jvm ] [Love] [gc][old][1332914][40809] duration [6.5s], collections [1]/[7.3s], total [6.5s]/[4.3h], memory [15.7gb]->[15gb]/[15.8gb], all_pools {[young] [1.4gb]->[911.3mb]/[1.4gb]}{[survivor] [124mb]->[0b]/[191.3mb]}{[old] [14.1gb]->[14.1gb]/[14.1gb]}

What is [1332914]? What is [40809]? Why is everything inside square brackets? What's with the trailing spaces after 'monitor.jvm'? Are you trying to make things easier for a human to read? Where in your documentation does it breaks out the field keys for values like "old". What's "old", what should I call that [(parameter)]?

This is very hard to grok with Logstash. You make both Elasticsearch and Logstash, so this is just disappointing I guess.

Sorry, thanks for letting me vent.

  • Dave
2

Isn't that generated by JVM memory logging ?

3

Yeah, I know your pain there.

There is a few similar things on github; https://github.com/elastic/elasticsearch/issues/15430, https://github.com/elastic/elasticsearch/issues/12305 and https://github.com/elastic/elasticsearch/issues/8786 and there maybe more I missed.

4

Yes, and no. It's JVM GC info but ES is generating the formatting.

5

For posterity, [1332914] looks like its the number of times the gc checker thread woke up and looked at GC info. I'm not sure what good it does but yeah.

[40809] is a mirror of the collection count from the JVM.

I actually really like having non-message stuff in some kind of marker. Square brackets are painful to regex so I wouldn't have chosen them but I'll take them over nothing.

This is the logging category, spaced out to 25 characters and left aligned. Its pretty standard to do that kind of thing when logging in java. I like it. The rest of the line is a ton of stuff all crammed together so its hard to read even if you know what it all means.

I dunno. Its almost certainly missing. "old" is Elasticsearch normalizing the JVM's names for its memory pools. Its probably "CMS Old Gen" if you are using the default configuration. I honestly tend to use Java's GC logging when I want to debug this sort of thing. Its more standard so its easier to look up on google.