# Using a regex in the custom field of Filebeat

**URL:** <https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252>\
**Category:** Beats\
**Created:** [March 30, 2018, 3:44pm UTC](https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252 "2018-03-30T15:44:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [March 30, 2018, 3:44pm UTC](https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252/1 "2018-03-30T15:44:37Z")

</div>

[Here](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#configuration-fields) I can read that when configuring a prospect I can add a custom field to the data, which later I can use for filtering.

So for example I can write

```
- type: log
  paths:
    - /my/path/app1.csv
  fields:
    app_name: app1
- type: log
  paths:
    - /my/path/app2.csv
  fields:
    app_name: app2

```

This means that anytime I will have a new CSV file to track I have to add it to the `filebeat.yml` file adding the custom `app_name` field accordingly.

I was wondering if I could use a regex with a capture group in the prospect definition to "automatically" track any new file and assign the right `app_name` value. Something like this:

```
- type: log
  paths:
    - /my/path/(.*).csv
  fields:
    app_name: \1

```

What do you think? I didn't find any [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/regexp-support.html) regarding this possibility with the `fields` feature.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 3:58pm UTC](https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252/2 "2018-03-30T15:58:10Z")

</div>

hello @gvdm90, Currently, it's not possible to dynamically extract that information from an event and reuse it as a field with Filebeat, but we plan to add something in beats that will work like the [dissect filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html) in Logstash.

But you can solve your problem by either one of the following options:

1. Use the [ingest node](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html) feature to do the processing, you can extract the `app_name` part using a `grok processor` and do more filtering after.

2. Use [Logstash](https://www.elastic.co/products/logstash) with the beats inputs and the grok filter and send your events to Logstash instead of sending it directly to Elasticsearch.

What kind of filtering are you doing?

---

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [March 30, 2018, 4:00pm UTC](https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252/3 "2018-03-30T16:00:26Z")

</div>

Hi @pierhugues

at the moment I'm already using an Elasticsearch pipeline to parse the filebeat data, so I would be happy if I could add a behaviour to that pipeline instead of using Logstash for this purpose.  
So it is possible to retrieve the path of the filebeat data from the data itself after it has been sent?

---

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [March 30, 2018, 4:02pm UTC](https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252/4 "2018-03-30T16:02:19Z")

</div>

I'm responding to myself at the last question: yes, the path is sent by filebeat with the data!  
It is the `source` field. Did you mind that field for my purpose or were you thinking about one another solution?

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 4:03pm UTC](https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252/5 "2018-03-30T16:03:46Z")

</div>

@gvdm90 Yes, I was thinking about that field, I was just getting an example of the format 😉

```auto
  "@timestamp": "2018-03-30T16:02:33.440Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "offset": 55229,
  "message": "ho ho",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "sashimi",
    "hostname": "sashimi",
    "version": "7.0.0-alpha1"
  },
  "source": "/var/log/system.log"
}
``
```

---

<div class="post-metadata">

**Author:** ![gvdm90](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gvdm90](https://discuss.elastic.co/u/gvdm90)\
**Post date:** [March 30, 2018, 4:04pm UTC](https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252/6 "2018-03-30T16:04:22Z")

</div>

Cool then, I will try this path and let you know 🙂  
Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2018, 6:04pm UTC](https://discuss.elastic.co/t/using-a-regex-in-the-custom-field-of-filebeat/126252/7 "2018-04-27T18:04:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.