# Adding a type field dynamically for logs **URL:** \ **Category:** Logstash\ **Created:** [February 3, 2017, 4:41pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883 "2017-02-03T16:41:12Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Maria\_Delarosa](https://avatars.discourse-cdn.com/v4/letter/m/87869e/32.png) [@Maria\_Delarosa](https://discuss.elastic.co/u/Maria_Delarosa)\ **Post date:** [February 3, 2017, 4:41pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/1 "2017-02-03T16:41:12Z")
Logstash is gathering logs from multiple directories. Each directory represents logs from an specific application. I am using the `type` to indicate the type of logs. However, I have many applications and adding each one with its correponding `type` to the configuration file is becoming quite overwhelming. Is there a way to this dynamically?(similar to shown below). The type is determined by the name of the log file. Current: ``` file { path => ["/var/log/logsfrommanyapps/app1/app1.log"] type => "app1", ignore_older => 7776000 start_position => "beginning" sincedb_path => "/dev/null" } file { path => ["/var/log/logsfrommanyapps/app2/app2.log"] type => "app2", ignore_older => 7776000 start_position => "beginning" sincedb_path => "/dev/null" } file { path => ["/var/log/logsfrommanyapps/app3/app3.log"] type => "app3", ignore_older => 7776000 start_position => "beginning" sincedb_path => "/dev/null" } output { stdout { codec => rubydebug { metadata => true } } } ``` Desired way dynamically: ``` file { path => ["/var/log/logsfrommanyapps/**/*.log"] type => "%{type}" ignore_older => 7776000 start_position => "beginning" sincedb_path => "/dev/null" } output { stdout { codec => rubydebug { metadata => true } } } ``` ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [February 4, 2017, 11:12am UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/2 "2017-02-04T11:12:14Z")
You can use grok filter to do that : ``` input { file { path => ["/var/log/logsfrommanyapps/*/*.log"] ignore_older => 7776000 start_position => "beginning" sincedb_path => "/dev/null" } } filter { grok { match => { "path" => "/var/log/logsfrommanyapps/%{WORD:type}/%{DATA}" } overwrite => ["type"] } } output { stdout { codec => rubydebug { metadata => true } } } ``` ---
**Author:** ![Maria\_Delarosa](https://avatars.discourse-cdn.com/v4/letter/m/87869e/32.png) [@Maria\_Delarosa](https://discuss.elastic.co/u/Maria_Delarosa)\ **Post date:** [February 8, 2017, 3:51pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/3 "2017-02-08T15:51:45Z")
Awesome that works great! One additional question in relation to this. Logstash adds a `host` field name. Currently it shows e.g. `"host" => "servername.mydomain.com"`. How would I be able to parse that field so it only outputs `"host" => "servername"`? ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [February 8, 2017, 4:54pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/4 "2017-02-08T16:54:28Z")
It is the same idea: filter { grok { match =\> { "host" =\> "^%{WORD:host}." } overwrite =\> ["host"] } } ---
**Author:** ![Maria\_Delarosa](https://avatars.discourse-cdn.com/v4/letter/m/87869e/32.png) [@Maria\_Delarosa](https://discuss.elastic.co/u/Maria_Delarosa)\ **Post date:** [February 8, 2017, 5:02pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/5 "2017-02-08T17:02:09Z")
Ah perfect, I was close, thanks. Ran into an issue with the first grok for the `type`. I have the following log `"/var/log/logsfrommanyapps/app3/app3-access.log"`with the current grok it shows type `type => "app3"` but would like for the `type` to be the filename e.g. `type => "app3-access"` ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [February 8, 2017, 7:54pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/6 "2017-02-08T19:54:33Z")
filter { grok { match =\> { "path" =\> "^/var/log/logsfrommanyapps/%{WORD}/%{NOTSPACE:type}.log$" } overwrite =\> ["type"] } } ---
**Author:** ![Maria\_Delarosa](https://avatars.discourse-cdn.com/v4/letter/m/87869e/32.png) [@Maria\_Delarosa](https://discuss.elastic.co/u/Maria_Delarosa)\ **Post date:** [February 8, 2017, 11:24pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/7 "2017-02-08T23:24:37Z")
Sorry to bother again and this will be the last. I realized I also have logs in the base directory `/var/log/logsfrommanyapps/*.log` the `grok` parse works well for setting`type` for logs coming from the subdirectories: `/var/log/logsfrommanyapps/*/*.log` but not from the base directory `/var/log/logsfrommanyapps/*.log`. How to grok for such logs? Below is my updated configuration with your previous answer: ``` input{ file { path => ["/var/log/logsfrommanyapps/*/*.log"] ignore_older => 7776000 start_position => "beginning" sincedb_path => "/dev/null" } file { path => ["/var/log/logsfrommanyapps/*.log"] type => "%{type}" ignore_older => 7776000 start_position => "beginning" sincedb_path => "/dev/null" } filter { grok { match => { "path" => "^/var/log/logsfrommanyapps/%{WORD}/%{NOTSPACE:type}.log$"} overwrite => ["type"] } } output { stdout { codec => rubydebug { metadata => true } } } } ``` ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [February 9, 2017, 5:46pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/8 "2017-02-09T17:46:28Z")
You could do that using the following configuration. If first pattern is not matched, the second is used. ``` filter { grok { match => { "path" => [ "^/var/log/logsfrommanyapps/%{WORD}/%{NOTSPACE:type}.log$" "^/var/log/logsfrommanyapps/%{NOTSPACE:type}.log$" ]} overwrite => ["type"] } } ``` One last thing : it is useless to set `type => "%{type}"` in file input. ---
**Author:** ![Maria\_Delarosa](https://avatars.discourse-cdn.com/v4/letter/m/87869e/32.png) [@Maria\_Delarosa](https://discuss.elastic.co/u/Maria_Delarosa)\ **Post date:** [February 9, 2017, 8:27pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/9 "2017-02-09T20:27:06Z")
Great! Below is my final code. I decided for any logs under a sub-directory to determine the `type` based on that directory name. For example `/var/log/logsfrommanyapps/app1/app1-test.log` is of `type => app1` (using the grok filter that you show in your first response). Then for any logs in the base directory, I am using the name of the log file. E.g. `/var/log/logsfrommanyapps/test.log` is of `type => test` as you show above. However, there is an issue in displaying the `type` for logs under a sub-directory. For `/var/log/logsfrommanyapps/app1/app1-test.log` it shows as `type => app1/app1-test` instead of `type => app1`. **Edit** : to avoid any further problems for any logs under a subdirectory, is it possible to just give it the second level directory name to `type`. For example `/var/log/logsfrommanyapps/app1/anotherdirectory/app1-test.log` will give `type => app1`. ``` input{ file { path => ["/var/log/logsfrommanyapps/*/*.log"] ignore_older => 7776000 start_position => "beginning" sincedb_path => "/dev/null" } file { path => ["/var/log/logsfrommanyapps/*.log"] ignore_older => 7776000 start_position => "beginning" sincedb_path => "/dev/null" } filter { grok { match => { "path" => [ "^/var/log/logsfrommanyapps/%{WORD:type}/%{DATA}$" "^/var/log/logsfrommanyapps/%{NOTSPACE:type}.log$" ]} overwrite => ["type"] } } output { stdout { codec => rubydebug { metadata => true } } } } ``` ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [February 10, 2017, 7:15am UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/10 "2017-02-10T07:15:26Z")
You set "//var" instead of "/var" in your first grok pattern ---
**Author:** ![Maria\_Delarosa](https://avatars.discourse-cdn.com/v4/letter/m/87869e/32.png) [@Maria\_Delarosa](https://discuss.elastic.co/u/Maria_Delarosa)\ **Post date:** [February 10, 2017, 3:35pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/11 "2017-02-10T15:35:42Z")
My apologies. It was a typo, fixed now. However, it is not showing the correct type for logs in sub-directories. The grok works well for logs in the base directory. ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [February 10, 2017, 5:59pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/12 "2017-02-10T17:59:43Z")
It should work with examples you give. But I guess you have different path names that you offuscate for confidential reasons I suppose. To test/debug your grok patterns, I invite you to use : [https://grokdebug.herokuapp.com](https://grokdebug.herokuapp.com) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 10, 2017, 6:00pm UTC](https://discuss.elastic.co/t/adding-a-type-field-dynamically-for-logs/73883/13 "2017-03-10T18:00:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.