Using a regex in the custom field of Filebeat

Elastic Stack Beats
1

Here I can read that when configuring a prospect I can add a custom field to the data, which later I can use for filtering.

So for example I can write

- type: log
  paths:
    - /my/path/app1.csv
  fields:
    app_name: app1
- type: log
  paths:
    - /my/path/app2.csv
  fields:
    app_name: app2

This means that anytime I will have a new CSV file to track I have to add it to the filebeat.yml file adding the custom app_name field accordingly.

I was wondering if I could use a regex with a capture group in the prospect definition to "automatically" track any new file and assign the right app_name value. Something like this:

- type: log
  paths:
    - /my/path/(.*).csv
  fields:
    app_name: \1

What do you think? I didn't find any documentation regarding this possibility with the fields feature.

2

hello @gvdm90, Currently, it's not possible to dynamically extract that information from an event and reuse it as a field with Filebeat, but we plan to add something in beats that will work like the dissect filter in Logstash.

But you can solve your problem by either one of the following options:

  1. Use the ingest node feature to do the processing, you can extract the app_name part using a grok processor and do more filtering after.

  2. Use Logstash with the beats inputs and the grok filter and send your events to Logstash instead of sending it directly to Elasticsearch.

What kind of filtering are you doing?

3

Hi @pierhugues

at the moment I'm already using an Elasticsearch pipeline to parse the filebeat data, so I would be happy if I could add a behaviour to that pipeline instead of using Logstash for this purpose.
So it is possible to retrieve the path of the filebeat data from the data itself after it has been sent?

4

I'm responding to myself at the last question: yes, the path is sent by filebeat with the data!
It is the source field. Did you mind that field for my purpose or were you thinking about one another solution?

1 Like
5

@gvdm90 Yes, I was thinking about that field, I was just getting an example of the format :wink:

  "@timestamp": "2018-03-30T16:02:33.440Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "offset": 55229,
  "message": "ho ho",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "sashimi",
    "hostname": "sashimi",
    "version": "7.0.0-alpha1"
  },
  "source": "/var/log/system.log"
}
``
6

Cool then, I will try this path and let you know :slight_smile:
Thank you

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.