# How to tag log files in filebeat for logstash ingestion?

**URL:** <https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 17, 2016, 1:54pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713 "2016-03-17T13:54:18Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![jclose](https://avatars.discourse-cdn.com/v4/letter/j/df705f/32.png) [@jclose](https://discuss.elastic.co/u/jclose)\
**Post date:** [March 17, 2016, 1:54pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/1 "2016-03-17T13:54:18Z")

</div>

If I have several different log files in a directory, and I'm wanting to forward them to logstash for grok'ing and buffering, and then to downstream Elasticsearch.

As the files are coming out of Filebeat, how do I tag them with something so that logstash knows which filter to apply?

So if I have:

> ```
> paths:
> - /path/to/logs/access.log
> - /path/to/other/logs/errors.log
> 
> ```

What do I do with the beats input within logstash that allows me to send the right logs to the right grok filter?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 17, 2016, 2:53pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/2 "2016-03-17T14:53:09Z")

</div>

You can add custom [fields](https://www.elastic.co/guide/en/beats/filebeat/1.1/configuration-filebeat-options.html#configuration-fields) to the events that you can then use to conditional filtering in Logstash.

You can define [multiple prospectors](https://www.elastic.co/guide/en/beats/filebeat/1.1/multiple-prospectors.html) in the Filebeat configuration. So group the files that need the same processing under the same prospector so that the same custom fields are added.

---

<div class="post-metadata">

**Author:** ![jclose](https://avatars.discourse-cdn.com/v4/letter/j/df705f/32.png) [@jclose](https://discuss.elastic.co/u/jclose)\
**Post date:** [March 18, 2016, 1:04pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/3 "2016-03-18T13:04:11Z")

</div>

So to do this, I would need to define multiple prospectors?

I was really hoping that I could define the paths, and then be able to use some sort of macro/variable within the fields variable. So something like:

> paths:
> 
> - /path/to/logs/access.log
> - /path/to/other/logs/errors.log  
> fields: %{filename}

Where the field coming across the wire for each beat would be 'access.log' and 'errors.log'.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 18, 2016, 1:22pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/4 "2016-03-18T13:22:26Z")

</div>

If you want to apply conditional filtering based on the filename then use the [`source`](https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-log.html#_source) field.

---

<div class="post-metadata">

**Author:** ![jclose](https://avatars.discourse-cdn.com/v4/letter/j/df705f/32.png) [@jclose](https://discuss.elastic.co/u/jclose)\
**Post date:** [March 18, 2016, 1:40pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/5 "2016-03-18T13:40:08Z")

</div>

Would the alternative (the custom fields method you suggested, using multiple prospectors) look something like this?

```
filebeat:
    prospectors:
      -
         paths:
             - /path/to/logs/access.log
         fields: {log_type: access}
      - 
         paths:
             - /path/to/other/logs/errors.log
         fields: {log_type: errors}

```

If so, what are the options on the logstash side for setting the field type in the 'input' for beats, so that I can tag appropriately, so that my filter easily directs each message to the appropriate grok (I'm still learning about the macros available)?

I could see wanting do say something like:

```
input {
    beats {
        port => 5044
        type => %{log_type}
    }
}

```

Thanks for your help, by the way. I haven't been able to find any explicit examples on how to handle sending messages from beats to logstash, where logstash will handle the grok and parsing into JSON for downstream Elastic.

---

<div class="post-metadata">

**Author:** ![jclose](https://avatars.discourse-cdn.com/v4/letter/j/df705f/32.png) [@jclose](https://discuss.elastic.co/u/jclose)\
**Post date:** [March 18, 2016, 1:58pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/6 "2016-03-18T13:58:44Z")

</div>

Digging deeper, it seems like I should be able to say:

```
type => "%{[@metadata][log_type]}"

```

or would it be

```
type =>"%{[log_type]}"

```

Or if I were trying to use the source field, using:

```
type => "%{[@metadata][source]}"

```

Does that look right? Is that how I would access a field within logstash, so that I can appropriately type the data within the input?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 18, 2016, 2:16pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/7 "2016-03-18T14:16:23Z")

</div>

The first part is correct on its own, but you may want to change it based on how you intend to write your Logstash config.

The second part won't work. You cannot override the `type` field once it's set. This is documented [here](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-type). If you want to change the `type` then set [`document_type: access`](https://www.elastic.co/guide/en/beats/filebeat/1.1/configuration-filebeat-options.html#_document_type) in your Filebeat configuration.

Here's an example Logstash config based on the Filebeat config you gave:

Filebeat:

```auto
filebeat:
  prospectors:
    - paths:
        - /path/to/logs/access.log
      fields: {log_type: access}
    -   
      paths:
        - /path/to/other/logs/errors.log
      fields: {log_type: errors}

```

Logstash:

```auto
input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][log_type] == "access" {
    mutate {
      add_field => { "foo" => "var" }
    }
  }
}

output {
  stdout { codec => rubydebug{} }
}

```

---

<div class="post-metadata">

**Author:** ![jclose](https://avatars.discourse-cdn.com/v4/letter/j/df705f/32.png) [@jclose](https://discuss.elastic.co/u/jclose)\
**Post date:** [March 18, 2016, 3:00pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/8 "2016-03-18T15:00:21Z")

</div>

Thank you so much. That will work!

> [@andrewkroh](#):
>
> You cannot override the `type` field once it's set.

I'm guessing this is related to the `{dynamic_type}` in the template file? Is there no way to dynamically set that type based upon the file input? Would I need to create a separate mapping for each of the different log types, identify those mappings in the yml file, and then do it that way?

In the end, with what you've shown me, I don't think it matters much. As long as I can quickly filter and grok based upon the incoming message type, then I'm fine. I'm just trying to learn a little more about how filebeat works at this point.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 18, 2016, 4:03pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/9 "2016-03-18T16:03:18Z")

</div>

> [@jclose](#):
>
> I'm guessing this is related to the {dynamic\_type} in the template file?

No, it is because the logstash-beats-input doesn't allow it to be overridden if it's already set.

> [@jclose](#):
>
> Is there no way to dynamically set that type based upon the file input?

In Filebeat you can use `document_type: mytype` on a per prospector basis. And in Logstash there are many ways to do this. My personal preference would be set type at the source using `document_type`. But here's an arbitrary Logstash example.

```auto
filter {
  if ([source] =~ /access.log$/) {
    mutate {
      replace => {
        "[@metadata][type]" => "access"
        "[type]" => "access"
      }
    }
  }

```

> [@jclose](#):
>
> Would I need to create a separate mapping for each of the different log types

The provided index template will handle this assuming you are writing all these types to a `filebeat-*` index.

---

<div class="post-metadata">

**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\
**Post date:** [December 1, 2016, 4:36pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/10 "2016-12-01T16:36:32Z")

</div>

hi , i was new to elk was trying to figure out , how can i differentiate multiple files  
/var/log/kakfa/\*.log  
these are the files i have in that path  
controller.log kafka-authorizer.log kafkaServer-gc.log log-cleaner.log state-change.log controller.log controller.log kafka-request.log server.log

filebeat::  
filebeat.prospectors:

- input\_type: log
  - /var/log/kafka/\*.log

Logstash::

input {  
beats {  
port =\> 5044  
tags =\> ["kafka"]  
}

output {

```
if "kafka" in [tags] {
    elasticsearch {
        action => "index"
        hosts => "elasticsearch:port"
        index => "kafka"

    }
}
```

---

<div class="post-metadata">

**Author:** ![mohsin106](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohsin106/32/65203_2.png) [@mohsin106](https://discuss.elastic.co/u/mohsin106)\
**Post date:** [March 1, 2017, 6:10pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/11 "2017-03-01T18:10:07Z")

</div>

bob-bza,

your config worked for me but what if you have multiple paths configured in filebeats and you want to have a separate index for each path?

For example what if I have this in my filebeat.yml:

> filebeat::  
> filebeat.prospectors:  
> input\_type: log  
> /var/log/kafka/_.log  
> /var/log/message_

Then how do I configure my logstash config file so that the kafka logs will be associated to the "kafka" index and the message logs will be associated to "message" index?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 3, 2017, 9:37am UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/12 "2017-03-03T09:37:35Z")

</div>

For your case you can use multiple prospectors and define fields / tags for each prospector.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:49pm UTC](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/13 "2017-07-05T21:49:52Z")

</div>