Is it possible to setup the same Color for aggregated log files?

[dmccuk (Dennis)]

Hi,

I'm attempting to setup some visualizations in Kibana based on some data I'm collecting. I have a working Visualization showing me the right information but in order to make the visualization work in a more continuous way, I need to do something with the colors that get generated for the sources.

Let me try and explain:

My Graph is listing the following:

Y: Axis:
I'm using MAX value to show an integer called backlogsize.

X: Axis:
Date Histogram using @timestamp

X: Axis: Split series:
Source.keyword :Terms (This is multiple log files)

X: Axis: Split chart:
beat.name.keyword - Terms (This is one of multiple hostnames)

I see a line/bar chart showing the backlog over time with colored source.keyword (separate log files). I have a separate box for each server which is perfect for how I want to display the data.

So this works well but the issue I have is:

1. Log files rotate everyday at midnight. If my time range goes back over midnight, I see two source.keyword log files. One for today and one for yesterday. I would like them both to be the same color so you can follow the information as one continuous color on the visualization. I don't want to do this manually.

2. The log files themselves rotate once they reach a certain size. I.E. logfile.0.log will become logfile.1.log, which will become logifle.2.log (but the data being written goes to the newest file. Not to logfile.0.log as in the linux logrotate world). Again, each source.keyword are treated as a new source and get a new color so you lose the continuity on the visualization. But they are just a continuation of the same data but in a new file.

Hopefully you can follow what I have and what I'm look for. Does anyone know if it's possible to make the source.keyword match part of a name and keep the same color instead of treat it as a new source?

I'm open to any help you can give me and I can provide more information. Please Just ask.

Thank you for your hep in advance.

Regards

Dennis

[Bargs (Matt Bargar)]

I would recommend adding a field to your documents that represents the type of log it is instead of relying on the full filename. So for documents that come from logfile.1.log, the value of this new field would just be logfile. You can then group on this field instead of the source.keyword field.

In a pinch you can use a scripted field to create a new field based on the source.keyword field and do a terms agg on it instead. Scripted fields can have a performance impact though, so it would be better to add this metadata to your docs at ingest time.

[dmccuk (Dennis)]

Morning Bargs,

Just back after a bank holiday weekend in the UK. Thanks for your reply.

I have resolved the issue in logstash by doing something very similar to what you have described.

I took the logname (source) in Logstash and mutated it (gsub) creating a new field to provide a shortname of the logfile striping out the path and the logfile extension. I referenced the shortname in the split series part of my graph in Kibana and this has had the effect of only showing me a continuation of one color per log file which has resolved my issue.

Thanks.