# Recommended logic for handling different log files on same host?

**URL:** <https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040>\
**Category:** Logstash\
**Created:** [August 13, 2019, 2:11pm UTC](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040 "2019-08-13T14:11:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![hueyg](https://avatars.discourse-cdn.com/v4/letter/h/8c91f0/32.png) [@hueyg](https://discuss.elastic.co/u/hueyg)\
**Post date:** [August 13, 2019, 2:11pm UTC](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040/1 "2019-08-13T14:11:17Z")

</div>

I have a host with three different log files. Using one instance of filebeat to ship all of the logs. Started this process back on 6.2 and discovered the "type" option. So based on the log file path I would set the type to say "access, audit, etc..." and ship out to Logstash. The logstash output would then use this type field as part of the index name so they would be easily identifiable in Kibana.

Used this simple code: index =\> "%{[type]}-%{+YYYY.MM.dd}"

This worked great, until I started adding other types of beats. Take Packetbeat. It comes with this fantastic dashboard for DNS. Well the packetbeat yml files uses types to differentiate the different protocols so now I have indecies that say dns-2019.08.13 and icmp-2019.08.13. All correct since that is how I coded my logstash output, however, the dashboard is expecting packetbeat-2019.08.13 as the index name.

So bottom line, is there another option (metadata?) that I can use to create different custom index names from the same host besides using type?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2019, 3:22pm UTC](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040/2 "2019-08-13T15:22:58Z")

</div>

Use a conditional to build the index name in a metadata field

```
if <event from packetbeat> {
    mutate { add_field => { "[@metadata][indexName]" => "packetbeat-%{YYYY.MM.dd}" } }
} else {
    mutate { add_field => { "[@metadata][indexName]" => "%{[type]}" } }
}

```

then in the output use

```
index => "%{[@metadata][indexName]}

```

The test for '\<event from packetbeat\>' would be something like [beat][name] == "packetbeat" but I do not have packetbeat running so I cannot test exactly what it should be.

Do the packetbeat index names not include the version number?

---

<div class="post-metadata">

**Author:** ![hueyg](https://avatars.discourse-cdn.com/v4/letter/h/8c91f0/32.png) [@hueyg](https://discuss.elastic.co/u/hueyg)\
**Post date:** [August 13, 2019, 4:29pm UTC](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040/3 "2019-08-13T16:29:39Z")

</div>

This definitely looks promising. I am pretty ignorant on what @metadata fields are available and if they are consistent across all types of "beats". In regards to your version question, I am not sure if you are referring to the version of the beats but that is available but don't understand how you would use it.

Going to dig into this more.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2019, 4:37pm UTC](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040/4 "2019-08-13T16:37:37Z")

</div>

If I recall correctly, some beats include the version number as well as the beat name in the index name that they expect to use for the dashboards. Thus the dashboards are versioned. You'll know it if it is happening, if not then do not worry about it.

---

<div class="post-metadata">

**Author:** ![hueyg](https://avatars.discourse-cdn.com/v4/letter/h/8c91f0/32.png) [@hueyg](https://discuss.elastic.co/u/hueyg)\
**Post date:** [August 13, 2019, 4:51pm UTC](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040/5 "2019-08-13T16:51:52Z")

</div>

I am searching like crazy, but not having much success finding what default/standard metadata fields exist for beats. Do you know if there is a way to query this information from a running beat? Not to keep adding work so if you don't recall I will keep searching.

---

<div class="post-metadata">

**Author:** ![hueyg](https://avatars.discourse-cdn.com/v4/letter/h/8c91f0/32.png) [@hueyg](https://discuss.elastic.co/u/hueyg)\
**Post date:** [August 13, 2019, 4:57pm UTC](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040/6 "2019-08-13T16:57:03Z")

</div>

This seems to reference some of the fields, but still no definitive list. This looks like it would work with your logic.

[https://www.elastic.co/guide/en/logstash/7.2/plugins-inputs-beats.html#plugins-inputs-beats-versioned-indexes](https://www.elastic.co/guide/en/logstash/7.2/plugins-inputs-beats.html#plugins-inputs-beats-versioned-indexes)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 10, 2019, 4:57pm UTC](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040/7 "2019-09-10T16:57:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.