# Output file configuration

**URL:** <https://discuss.elastic.co/t/output-file-configuration/136269>\
**Category:** Logstash\
**Created:** [June 18, 2018, 9:17am UTC](https://discuss.elastic.co/t/output-file-configuration/136269 "2018-06-18T09:17:03Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 18, 2018, 9:17am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/1 "2018-06-18T09:17:03Z")

</div>

Hi,

I have this configuration on 30-elasticsearch-output.conf

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}  
file {  
path =\> "/tmp/log/Syslog\_Server\_Logs/%{host[name]}/%{host[name]}-%{+YYYY-MM-dd}.log"  
codec =\> line { format =\> "%{source} - %{message}" }  
}  
}

My output files are something like my\_servers\_hostname-2018-06-18.log . I would like to have the application\_name-date.log. How i can modify my file output to do this.??

My filter configuration is

filter {  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
add\_field =\> ["received\_from", "%{host}"]  
}  
syslog\_pri { }  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
}  
}

I saw that in the message the application name is related with the %{DATA:syslog\_program}.  
I changed the file path as  
path =\> "/tmp/log/Syslog\_Server\_Logs/%{host[name]}/%{DATA:syslog\_program}-%{+YYYY-MM-dd}.log"

but it didn't bring me the name of the application on the output .log file.

Do you have any idea on that?

I use elasticsearch 6.3 - kibana 6.3 - logstash 6.3.

Best Regards,  
Thanos

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 18, 2018, 10:14am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/2 "2018-06-18T10:14:02Z")

</div>

If the program name is stored in the `syslog_program` field you should include `%{syslog_program}` in the `path` option of your file output.

Also, `%{host[name]}` isn't the right syntax. It might work now but you should change it to `%{[host][name]}`. See [https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references).

---

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 18, 2018, 11:26am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/3 "2018-06-18T11:26:30Z")

</div>

Thanks a lot for your reply,

I did the change

file {  
path =\> "/tmp/log/Syslog\_Server\_Logs/%{[host][name]}/%{syslog\_program}-%{+YYYY-MM-dd}.log"  
codec =\> line { format =\> "%{source} - %{message}" }  
}

but i get now the file as  
%{syslog\_program}-2018-06-18.log

do i have to write it as %{[DATA][syslog\_program]} ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 18, 2018, 1:06pm UTC](https://discuss.elastic.co/t/output-file-configuration/136269/4 "2018-06-18T13:06:55Z")

</div>

> but i get now the file as  
> %{syslog\_program}-2018-06-18.log

Then the event didn't have a `syslog_program` field.

---

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 18, 2018, 1:39pm UTC](https://discuss.elastic.co/t/output-file-configuration/136269/5 "2018-06-18T13:39:00Z")

</div>

But as i can see from my filter settings  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }

this filter send me e message like this

Jun 18 16:30:01 atmonxh01 systemd: Started Session 7431 of user apache.

From that message i understand that the systemd: comes from %{DATA:syslog\_program} from my filter.  
Am i right??

How i can see the fields?

BR,  
Thanos

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 18, 2018, 1:46pm UTC](https://discuss.elastic.co/t/output-file-configuration/136269/6 "2018-06-18T13:46:10Z")

</div>

> From that message i understand that the systemd: comes from %{DATA:syslog\_program} from my filter.

Yes, judging by the configuration you've shown us I'd expect events to have a `syslog_program` field if the grok filter is successful.

> How i can see the fields?

You're storing the complete events in Elasticsearch. During debugging I always recommend people to use a `stdout { codec => rubydebug }` output.

---

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 18, 2018, 1:47pm UTC](https://discuss.elastic.co/t/output-file-configuration/136269/7 "2018-06-18T13:47:48Z")

</div>

A ok thanks a lot for this!  
And where i add this code?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 18, 2018, 6:29pm UTC](https://discuss.elastic.co/t/output-file-configuration/136269/8 "2018-06-18T18:29:23Z")

</div>

> And where i add this code?

As I said it's an output. They go in the output block.

---

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 19, 2018, 9:10am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/9 "2018-06-19T09:10:05Z")

</div>

> [@magnusbaeck](#):
>
> stdout { codec =\> rubydebug }

Hi again,

i modified a litle bit my filter to that

filter {  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{GREEDYDATA:syslog\_message}" }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
add\_field =\> ["received\_from", "%{host}"]  
}  
syslog\_pri { }  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
}  
}

And the result is the same.  
The application is stored all in the %{[message]}. I don't know how to get only this information and put it on the file output .log name

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 19, 2018, 9:48am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/10 "2018-06-19T09:48:16Z")

</div>

???

You had a grok filter that looked like it would extract e.g. the `syslog_program` field and now you replaced it with a grok filter that doesn't do anything useful. Reinstate your old grok filter and report what do resulting events look like.

---

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 19, 2018, 10:24am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/11 "2018-06-19T10:24:42Z")

</div>

> [@magnusbaeck](#):
>
> `syslog_program`

i had this one

filter {  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
add\_field =\> ["received\_from", "%{host}"]  
}  
syslog\_pri { }  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
}  
}

and now i changed it to this one

filter {  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{GREEDYDATA:syslog\_message}" }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
add\_field =\> ["received\_from", "%{host}"]  
}  
syslog\_pri { }  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
}  
}

but the result on the kibana is the same as before.  
i get the messages as i got with the first filter without problem

That is strange.

that is mean that all the information for example  
"Jun 19 12:03:58 atdevxh104 nrpe[11297]: Client request was invalid, bailing out..."  
is stored to the "message"

The problem is that from the "message" i want to take the application name "nrpe" and put it on output file name .log

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 19, 2018, 10:45am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/12 "2018-06-19T10:45:26Z")

</div>

So what does the `stdout { codec => rubydebug }` output produce, i.e. what exactly do the processed events look like?

---

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 19, 2018, 10:49am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/13 "2018-06-19T10:49:08Z")

</div>

For example that i get from /var/log/messages ELK server

jun 19 13:46:05 atdevxhv03 logstash: [2018-06-19T13:46:05,115][INFO][logstash.outputs.file] Opening file {:path=\>"/tmp/log/Syslog\_Server\_Logs/atdevxh104.emea.nsn-net.net/atdevxh104.emea.nsn-net.net-2018-06-19.log"}  
Jun 19 13:46:05 atdevxhv03 logstash: {  
Jun 19 13:46:05 atdevxhv03 logstash: "message" =\> "Jun 19 13:46:01 atdevxh104 nrpe[21148]: Error: Request packet type/version was invalid!",  
Jun 19 13:46:05 atdevxhv03 logstash: "@timestamp" =\> 2018-06-19T10:46:05.070Z,  
Jun 19 13:46:05 atdevxhv03 logstash: "input" =\> {  
Jun 19 13:46:05 atdevxhv03 logstash: "type" =\> "log"  
Jun 19 13:46:05 atdevxhv03 logstash: },  
Jun 19 13:46:05 atdevxhv03 logstash: "beat" =\> {  
Jun 19 13:46:05 atdevxhv03 logstash: "name" =\> "[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)",  
Jun 19 13:46:05 atdevxhv03 logstash: "hostname" =\> "[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)",  
Jun 19 13:46:05 atdevxhv03 logstash: "version" =\> "6.3.0"  
Jun 19 13:46:05 atdevxhv03 logstash: },  
Jun 19 13:46:05 atdevxhv03 logstash: "offset" =\> 612897,  
Jun 19 13:46:05 atdevxhv03 logstash: "tags" =\> [  
Jun 19 13:46:05 atdevxhv03 logstash: [0] "beats\_input\_codec\_plain\_applied"  
Jun 19 13:46:05 atdevxhv03 logstash: ],  
Jun 19 13:46:05 atdevxhv03 logstash: "prospector" =\> {  
Jun 19 13:46:05 atdevxhv03 logstash: "type" =\> "log"  
Jun 19 13:46:05 atdevxhv03 logstash: },  
Jun 19 13:46:05 atdevxhv03 logstash: "source" =\> "/var/log/messages",  
Jun 19 13:46:05 atdevxhv03 logstash: "@version" =\> "1",  
Jun 19 13:46:05 atdevxhv03 logstash: "host" =\> {  
Jun 19 13:46:05 atdevxhv03 logstash: "name" =\> "[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)"  
Jun 19 13:46:05 atdevxhv03 logstash: }  
Jun 19 13:46:05 atdevxhv03 logstash: }  
Jun 19 13:46:05 atdevxhv03 logstash: {  
Jun 19 13:46:05 atdevxhv03 logstash: "message" =\> "Jun 19 13:46:01 atdevxh104 nrpe[21148]: Client request was invalid, bailing out...",  
Jun 19 13:46:05 atdevxhv03 logstash: "@timestamp" =\> 2018-06-19T10:46:05.070Z,  
Jun 19 13:46:05 atdevxhv03 logstash: "offset" =\> 612980,  
Jun 19 13:46:05 atdevxhv03 logstash: "input" =\> {  
Jun 19 13:46:05 atdevxhv03 logstash: "type" =\> "log"  
Jun 19 13:46:05 atdevxhv03 logstash: },  
Jun 19 13:46:05 atdevxhv03 logstash: "beat" =\> {  
Jun 19 13:46:05 atdevxhv03 logstash: "version" =\> "6.3.0",  
Jun 19 13:46:05 atdevxhv03 logstash: "name" =\> "[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)",  
Jun 19 13:46:05 atdevxhv03 logstash: "hostname" =\> "[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)"  
Jun 19 13:46:05 atdevxhv03 logstash: },  
Jun 19 13:46:05 atdevxhv03 logstash: "tags" =\> [  
Jun 19 13:46:05 atdevxhv03 logstash: [0] "beats\_input\_codec\_plain\_applied"  
Jun 19 13:46:05 atdevxhv03 logstash: ],  
Jun 19 13:46:05 atdevxhv03 logstash: "prospector" =\> {  
Jun 19 13:46:05 atdevxhv03 logstash: "type" =\> "log"  
Jun 19 13:46:05 atdevxhv03 logstash: },  
Jun 19 13:46:05 atdevxhv03 logstash: "source" =\> "/var/log/messages",  
Jun 19 13:46:05 atdevxhv03 logstash: "@version" =\> "1",  
Jun 19 13:46:05 atdevxhv03 logstash: "host" =\> {  
Jun 19 13:46:05 atdevxhv03 logstash: "name" =\> "[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)"  
Jun 19 13:46:05 atdevxhv03 logstash: }  
Jun 19 13:46:05 atdevxhv03 logstash: }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 19, 2018, 10:52am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/14 "2018-06-19T10:52:24Z")

</div>

As you can see the `type` field doesn't contain "syslog" (in fact, the `type` field isn't set at all) so your filters aren't used.

---

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 19, 2018, 11:06am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/15 "2018-06-19T11:06:33Z")

</div>

So i have to set type field as "log" and not as "syslog"

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 19, 2018, 1:42pm UTC](https://discuss.elastic.co/t/output-file-configuration/136269/16 "2018-06-19T13:42:49Z")

</div>

You can set the fields to any values you like as long as you're being consistent with the rest of your configuration.

---

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 20, 2018, 6:02am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/17 "2018-06-20T06:02:51Z")

</div>

> [@magnusbaeck](#):
>
> You can set the fields to any values you like as long as you're being consistent with the

So if on my clients filebeat.yml configuration have

- type: log

i have to write on filter  
if [type] == "log"

Do i need the filter??

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 20, 2018, 6:13am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/18 "2018-06-20T06:13:16Z")

</div>

`type: log` in the Filebeat configuration doesn't have anything to do with the `type` field in the Logstash messages. If you want to set the `type` field to e.g. syslog you can do this:

```nohighlight
fields:
  type: syslog
fields_under_root: true

```

See complete example at [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html).

---

<div class="post-metadata">

**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\
**Post date:** [June 20, 2018, 7:30am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/19 "2018-06-20T07:30:22Z")

</div>

Removing just the line if [type] == "syslog" solved my problem.

now i am getting in Kibana this json file from my clients

{  
"\_index": "filebeat-2018.06.20",  
"\_type": "doc",  
"\_id": "f6MPHGQBa73D38yOumhb",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"syslog\_timestamp": "Jun 20 10:18:59",  
"host": {  
"name": "[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)"  
},  
"syslog\_facility": "user-level",  
"source": "/var/log/messages",  
"syslog\_pid": "21558",  
"syslog\_program": "nrpe",  
"offset": 757242,  
"input": {  
"type": "log"  
},  
"received\_from": "{"name":"[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)"}",  
"syslog\_facility\_code": 1,  
"message": "Jun 20 10:18:59 atdevxh104 nrpe[21558]: Error: Request packet type/version was invalid!",  
"received\_at": "2018-06-20T07:19:00.884Z",  
"syslog\_message": "Error: Request packet type/version was invalid!",  
"tags": [  
"beats\_input\_codec\_plain\_applied"  
],  
"syslog\_severity": "notice",  
"@version": "1",  
"beat": {  
"name": "[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)",  
"hostname": "[atdevxh104.emea.nsn-net.net](http://atdevxh104.emea.nsn-net.net)",  
"version": "6.3.0"  
},  
"syslog\_severity\_code": 5,  
"@timestamp": "2018-06-20T07:18:59.000Z",  
"syslog\_hostname": "atdevxh104",  
"prospector": {  
"type": "log"  
}  
},  
"fields": {  
"received\_at": [  
"2018-06-20T07:19:00.884Z"  
],  
"@timestamp": [  
"2018-06-20T07:18:59.000Z"  
]  
},  
"sort": [  
1529479139000  
]

So i can use syslog\_program on my output configuration like this

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}  
file {  
path =\> "/data/Syslog\_Server\_Logs/%{[host][name]}/%{[syslog\_program]}-%{+YYYY-MM-dd}.log"  
codec =\> line { format =\> "%{[source]} - %{[message]}" }  
}  
}

and i am able to write the logs in seperate files using syslog\_program name on the log name like this

kernel-2018-06-20.log nrpe-2018-06-20.log rtvscand-2018-06-20.log

The only problem that i have now is on winlogbeat logs from Windows Eventviewer

The name that stores the winlogbeats logs is like this  
%{[syslog\_program]}-2018-06-20.log

This happens beacause there is not syslog\_program field on winlogbeat logs.

Can i use an if statement to seperate the file output? For example if input.type==log have this file output and if type==wineventlog have a different file output.

How i can use these if statements on my output file configuration?

Thanks a lot for all the replies

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 20, 2018, 7:52am UTC](https://discuss.elastic.co/t/output-file-configuration/136269/20 "2018-06-20T07:52:17Z")

</div>

> Removing just the line if [type] == "syslog" solved my problem.

For now yes, but you don't want to apply syslog filtering for your Winlogbeat messages.

> Can i use an if statement to seperate the file output? For example if input.type==log have this file output and if type==wineventlog have a different file output.
> 
> How i can use these if statements on my output file configuration?

See [Accessing event data and fields | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html) for some examples. That page also describes the notation used for subfields (`[input][log]` not `input.type`).

[Next page](https://discuss.elastic.co/t/output-file-configuration/136269.md?page=2)