# Logstash file plugin

**URL:** <https://discuss.elastic.co/t/logstash-file-plugin/56267>\
**Category:** Logstash\
**Created:** [July 25, 2016, 8:00am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267 "2016-07-25T08:00:08Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kishore](https://avatars.discourse-cdn.com/v4/letter/k/8edcca/32.png) [@Kishore](https://discuss.elastic.co/u/Kishore)\
**Post date:** [July 25, 2016, 8:00am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/1 "2016-07-25T08:00:08Z")

</div>

I am using file Output plugin to store parsed events, is there any option to clear that file for every 24hr?

Below is the Example:

output  
{  
if ( [status] =~ "Invalid credentials" ) and "throttled" not in [tags]  
{  
file {  
path =\> "/home/ec2-user/vpn\_invalid.yml"  
message\_format =\> "%{username}: invalid"  
}  
}  
}

I want to clear the data from **vpn\_invalid.yml** file for every 24hrs.

Kindly help me.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 25, 2016, 8:08am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/2 "2016-07-25T08:08:51Z")

</div>

You would need to write it to a time based file and the handle removal of the old one external to LS.

---

<div class="post-metadata">

**Author:** ![Kishore](https://avatars.discourse-cdn.com/v4/letter/k/8edcca/32.png) [@Kishore](https://discuss.elastic.co/u/Kishore)\
**Post date:** [July 25, 2016, 8:10am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/3 "2016-07-25T08:10:40Z")

</div>

Hi Mark,

Thank you very much for quick response.

Kindly help me in creating time based file.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 25, 2016, 8:19am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/4 "2016-07-25T08:19:17Z")

</div>

Have a read of [https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html#plugins-outputs-file-path](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html#plugins-outputs-file-path)

---

<div class="post-metadata">

**Author:** ![Kishore](https://avatars.discourse-cdn.com/v4/letter/k/8edcca/32.png) [@Kishore](https://discuss.elastic.co/u/Kishore)\
**Post date:** [July 25, 2016, 8:25am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/5 "2016-07-25T08:25:08Z")

</div>

One more query regarding this, this time based file can be used in Translate filter?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 25, 2016, 8:27am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/6 "2016-07-25T08:27:40Z")

</div>

What do you mean, can you give an example?

---

<div class="post-metadata">

**Author:** ![Kishore](https://avatars.discourse-cdn.com/v4/letter/k/8edcca/32.png) [@Kishore](https://discuss.elastic.co/u/Kishore)\
**Post date:** [July 25, 2016, 8:50am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/7 "2016-07-25T08:50:21Z")

</div>

I am using above mentioned file ' **vpn\_invalid.yml**' in the Translate filter to match the data.

Below is the example.

if ( [vpn\_status] =~ "pool returned") {  
translate {  
field =\> "username"  
destination =\> "invalid\_user"  
refresh\_interval =\> 5  
dictionary\_path =\> "/home/ec2-user/vpn\_invalid.yml"  
add\_tag =\> ["invaliduser"]  
}  
}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 25, 2016, 8:52am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/8 "2016-07-25T08:52:07Z")

</div>

You're using the same file in the output and then the filter?

---

<div class="post-metadata">

**Author:** ![Kishore](https://avatars.discourse-cdn.com/v4/letter/k/8edcca/32.png) [@Kishore](https://discuss.elastic.co/u/Kishore)\
**Post date:** [July 25, 2016, 9:08am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/9 "2016-07-25T09:08:37Z")

</div>

Yes, In the output section i am using to write the username filed and in the filter part i am using to match the value.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 25, 2016, 9:13am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/10 "2016-07-25T09:13:30Z")

</div>

Can you show the full config?

---

<div class="post-metadata">

**Author:** ![Kishore](https://avatars.discourse-cdn.com/v4/letter/k/8edcca/32.png) [@Kishore](https://discuss.elastic.co/u/Kishore)\
**Post date:** [July 25, 2016, 9:18am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/11 "2016-07-25T09:18:21Z")

</div>

```
input 
 {
stdin{}
 }

 
filter {
grok {
		match => { "message" => "%{MONTH:month}(?:\s|\s\s)%{MONTHDAY:day}\s(?<time> *[0-9]*[:][0-9]*[:][0-9][0-9])\s%{HOSTNAME:hostname}\s%{WORD:vpn_daemon}\:\suser\s\'%{USER:username}\' %{WORD:status}", "%{MONTH:month}(?:\s|\s\s)%{MONTHDAY:day}\s(?<time> *[0-9]*[:][0-9]*[:][0-9][0-9])\s%{HOSTNAME:hostname}\s%{WORD:vpn_daemon}\:\suser\s\'%{USER:username}\'(?<status> \b\w+\b\s\b\w+\b\s\b\w+\b)", "%{MONTH:month}(?:\s|\s\s)%{MONTHDAY:day}\s(?<time> *[0-9]*[:][0-9]*[:][0-9][0-9])\s%{HOSTNAME:hostname}\s%{WORD:vpn_daemon}\:\s/openvpn.auth-user.php: ERROR! Could not login to server\s%{WORD:server_name} as user %{USER:username}\:(?<status> \b\w+\b\s\b\w+\b)", "%{MONTH:month}(?:\s|\s\s)%{MONTHDAY:day}\s(?<time> *[0-9]*[:][0-9]*[:][0-9][0-9])\s%{HOSTNAME:hostname}\s%{WORD:vpn_daemon}\:\s/openvpn.auth-user.php: ERROR! Either(?<status> \b\w+\b\s\b\w+\b\s\b\w+\b\,\s\b\w+\b\s\b\w+\b\s\b\w+\b\s\b\w+\b\s\b\w+\b)", "%{MONTH:month}(?:\s|\s\s)%{MONTHDAY:day}\s(?<time> *[0-9]*[:][0-9]*[:][0-9][0-9])\s%{HOSTNAME:hostname}\s%{WORD:vpn_daemon}\[%{INT:vpn_id}\]\:\s%{USERNAME:username}/%{IPV4:public_ip}:%{INT:publicip_port} MULTI_sva:\s(?<vpn_status> *\b\w+\b \b\w+\b) IPv4=%{IPV4:vpn_ip}, IPv6=(?<vpn_msg>.*)" }
		add_field => ["received_at", "%{@timestamp}"]
		add_field => ["received_from", "%{host}"]
    }

geoip 
		{
           		add_tag => ["GeoIP"]
           		source => "public_ip"
           		add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
           		add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]

       	}
   		mutate 
		{
      			convert => ["[geoip][coordinates]", "float" ]
   		}

mutate {
            remove_tag => ["_grokparsefailure"]
        }
		
if ([vpn_daemon] !~ "openvpn")
                       { drop{} }		

if ( [status] =~ "Invalid credentials")
{
throttle {
before_count => 3
after_count => 3
period => 300
key => "%{username}"
add_tag => "throttled"
}
}

if ( [vpn_status] =~ "pool returned") {
translate {
    field => "username"
    destination => "invalid_user"
    refresh_interval => 5
    dictionary_path => "/home/ec2-user/vpn_invalid.yml"
    add_tag => ["invaliduser"]
}
}
  
  
  }
 
output
{
if ( [status] =~ "Invalid credentials" ) and "throttled" not in [tags]
{
  file {
        path => "/home/ec2-user/vpn_invalid.yml"
        message_format => "%{username}: invalid"
}
}     

 }
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 25, 2016, 9:32am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/12 "2016-07-25T09:32:35Z")

</div>

You can just do;

```auto
filter {
grok {
		match => { "message" => "PATTERN1", "PATTERN2", etc etc }
}
}

```

Instead of having all those separate groks, just makes it cleaner and easier to read.

---

<div class="post-metadata">

**Author:** ![Kishore](https://avatars.discourse-cdn.com/v4/letter/k/8edcca/32.png) [@Kishore](https://discuss.elastic.co/u/Kishore)\
**Post date:** [July 25, 2016, 9:58am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/13 "2016-07-25T09:58:29Z")

</div>

I did as you mentioned.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:46am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267/14 "2017-07-06T04:46:35Z")

</div>