Tons of formats, one path structure mesos/marathon


(Liammac) #1

I'm trying to sort out the best way to go about this, I have a mesos/marathon cluster I'm trying to setup comprehensive logging on,

Requirements,
-new applications must be logged even if all fields are not extracted properly
-multiline logs must be joined
-defined applications are logged and tags all extracted correctly
-few/no duplicated logs

So the setup I have now is one prospector prospecting

/var/lib/mesos/slave/slaves//frameworks//executors/*/runs/latest/stderr

and one prospector prospecting

/var/lib/mesos/slave/slaves//frameworks//executors/*/runs/latest/stdout

Now this is great and all but it means I can only use 1 multiline codec/regex to try and deal with all the multiline stuff that could ever be run on this cluster... unfortunately that list is extensive including multiline python stack traces, Java Stack traces, multiline JSON, and more

What I'd like to be able to do but doesn't seem possible is create prospectors for known apps and have a catchall

eg

p1 - /var/lib/mesos/slave/slaves//frameworks//executors/app1*/runs/latest/stderr

p2 - /var/lib/mesos/slave/slaves//frameworks//executors/app2*/runs/latest/stderr

p3 - /var/lib/mesos/slave/slaves//frameworks//executors/app3*/runs/latest/stderr

p4 (catchall) - /var/lib/mesos/slave/slaves//frameworks//executors/!(app1|app2|app3)*/runs/latest/stderr

however I don't believe glowing supports any type of negation. Can someone clarify this or let me know if there's some sort of workaround that might help here?


(Steffen Siering) #2

in filebeat all prospectors do work independently + scan directories independendly of each other. Plus glob doesn't really support negations (See Pattern syntax).

One workaround for "catchall" case might be using exlucde_files setting. Exclude files is a list of regular expressions applied to full file path.

So catchall prospector might become:

- input_type: log
  paths:
    -  '/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stderr'
  exclude_files:
    - 'executors/(app1|app2|app3)'

regexes used for matching do search for a substring matching the pattern given.

The initial proposal for multiline support did ask for multi-pattern support (which might solve your problem in general), but this hasn't been implemented yet.


(system) #3

This topic was automatically closed after 21 days. New replies are no longer allowed.