# Tons of formats, one path structure mesos/marathon

**URL:** <https://discuss.elastic.co/t/tons-of-formats-one-path-structure-mesos-marathon/57618>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 9, 2016, 5:14pm UTC](https://discuss.elastic.co/t/tons-of-formats-one-path-structure-mesos-marathon/57618 "2016-08-09T17:14:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![liammac](https://avatars.discourse-cdn.com/v4/letter/l/9e8a1a/32.png) [@liammac](https://discuss.elastic.co/u/liammac)\
**Post date:** [August 9, 2016, 5:14pm UTC](https://discuss.elastic.co/t/tons-of-formats-one-path-structure-mesos-marathon/57618/1 "2016-08-09T17:14:50Z")

</div>

I'm trying to sort out the best way to go about this, I have a mesos/marathon cluster I'm trying to setup comprehensive logging on,

Requirements,  
-new applications must be logged even if all fields are not extracted properly  
-multiline logs must be joined  
-defined applications are logged and tags all extracted correctly  
-few/no duplicated logs

So the setup I have now is one prospector prospecting

/var/lib/mesos/slave/slaves/_/frameworks/_/executors/\*/runs/latest/stderr

and one prospector prospecting

/var/lib/mesos/slave/slaves/_/frameworks/_/executors/\*/runs/latest/stdout

Now this is great and all but it means I can only use 1 multiline codec/regex to try and deal with all the multiline stuff that could ever be run on this cluster... unfortunately that list is extensive including multiline python stack traces, Java Stack traces, multiline JSON, and more

What I'd like to be able to do but doesn't seem possible is create prospectors for known apps and have a catchall

eg

p1 - /var/lib/mesos/slave/slaves/_/frameworks/_/executors/app1\*/runs/latest/stderr

p2 - /var/lib/mesos/slave/slaves/_/frameworks/_/executors/app2\*/runs/latest/stderr

p3 - /var/lib/mesos/slave/slaves/_/frameworks/_/executors/app3\*/runs/latest/stderr

p4 (catchall) - /var/lib/mesos/slave/slaves/_/frameworks/_/executors/!(app1|app2|app3)\*/runs/latest/stderr

however I don't believe glowing supports any type of negation. Can someone clarify this or let me know if there's some sort of workaround that might help here?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 10, 2016, 12:13pm UTC](https://discuss.elastic.co/t/tons-of-formats-one-path-structure-mesos-marathon/57618/2 "2016-08-10T12:13:01Z")

</div>

in filebeat all prospectors do work independently + scan directories independendly of each other. Plus glob doesn't really support negations (See [Pattern syntax](https://golang.org/pkg/path/filepath/#Match)).

One workaround for "catchall" case might be using [`exlucde_files` setting](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#exclude-files). Exclude files is a list of regular expressions applied to full file path.

So catchall prospector might become:

```auto
- input_type: log
  paths:
    - '/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stderr'
  exclude_files:
    - 'executors/(app1|app2|app3)'

```

regexes used for matching do search for a substring matching the pattern given.

The [initial proposal](https://github.com/elastic/beats/issues/461) for multiline support did ask for multi-pattern support (which might solve your problem in general), but this hasn't been implemented yet.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 30, 2016, 5:14pm UTC](https://discuss.elastic.co/t/tons-of-formats-one-path-structure-mesos-marathon/57618/3 "2016-08-30T17:14:56Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.