Tying an ID back to it's name which resolves in a different log

(Seth S) #1

I'm monitoring our Mesos Master for metrics on different Mesos Frameworks.

The issue is that the logs which provide metrics for frameworks only provide a numeric ID, such as 20160603-064656-2785394442-5050-26628-2075, whereas in other logs I'm collecting, that data resolves to a name such as 'MesosFramework27', which is the data I need to tie back to.

Can anyone give me an idea of where to start to look for a resolution to this? My understanding is that this is doable with an ES front end, but not as straight-forward with a Kibana front end.

Any help or guidance is much-appreciated. Running latest stable versions of E, L & K.

(Christian Dahlqvist) #2

If you know the mapping between IDs and names, you should be able to do the translation in Logstash prior to indexing it into Elasticsearch, possibly by using the translate filter.

(Seth S) #3

The problem is they're dynamically generated. Any Framework only lasts an hour or so.

(Christian Dahlqvist) #4

How do you determine the correct mapping?

(Seth S) #5

Another log contains that info - similar to [Header] 'FrameworkID' (FrameworkName) [Message]

Effectively, Users create a framework with whatever name they want, Mesos automatically creates an indexable key that it references with ~90% of the logs. There are about 50 frameworks at any given time, and plenty of logs do resolve the ID to the Name, I just need to find a way to tie that back in for any message containing the FrameworkID.

(system) #6