I'm trying to sort out the best way to go about this, I have a mesos/marathon cluster I'm trying to setup comprehensive logging on,
Requirements,
-new applications must be logged even if all fields are not extracted properly
-multiline logs must be joined
-defined applications are logged and tags all extracted correctly
-few/no duplicated logs
So the setup I have now is one prospector prospecting
/var/lib/mesos/slave/slaves//frameworks//executors/*/runs/latest/stderr
and one prospector prospecting
/var/lib/mesos/slave/slaves//frameworks//executors/*/runs/latest/stdout
Now this is great and all but it means I can only use 1 multiline codec/regex to try and deal with all the multiline stuff that could ever be run on this cluster... unfortunately that list is extensive including multiline python stack traces, Java Stack traces, multiline JSON, and more
What I'd like to be able to do but doesn't seem possible is create prospectors for known apps and have a catchall
eg
p1 - /var/lib/mesos/slave/slaves//frameworks//executors/app1*/runs/latest/stderr
p2 - /var/lib/mesos/slave/slaves//frameworks//executors/app2*/runs/latest/stderr
p3 - /var/lib/mesos/slave/slaves//frameworks//executors/app3*/runs/latest/stderr
p4 (catchall) - /var/lib/mesos/slave/slaves//frameworks//executors/!(app1|app2|app3)*/runs/latest/stderr
however I don't believe glowing supports any type of negation. Can someone clarify this or let me know if there's some sort of workaround that might help here?