# EOF in Logstash File Input

**URL:** <https://discuss.elastic.co/t/eof-in-logstash-file-input/251126>\
**Category:** Logstash\
**Created:** [October 6, 2020, 1:00pm UTC](https://discuss.elastic.co/t/eof-in-logstash-file-input/251126 "2020-10-06T13:00:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sfischer](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@sfischer](https://discuss.elastic.co/u/sfischer)\
**Post date:** [October 6, 2020, 1:00pm UTC](https://discuss.elastic.co/t/eof-in-logstash-file-input/251126/1 "2020-10-06T13:00:53Z")

</div>

Hello,

I am using Logstash to aggregate some data out of a log file and store it in elastic search, to get some usage data for a software.

We copy the log files into a directory that logstash checks. We use the file input plugin in read mode and delete the files from the directory afterwards.

```auto
file {
     mode => "read"
     file_completed_action => "delete"
     path => "path/to/directory/with/logs/*"
   }

```

For the data aggreation I would need to know when the file ends. Is there any way in Logstash to detect that EOF was reached?  
For instance, can I automatically generate a EOF event?  
Or detect that an event is the last one in the file?

Thanks!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 6, 2020, 2:32pm UTC](https://discuss.elastic.co/t/eof-in-logstash-file-input/251126/2 "2020-10-06T14:32:05Z")

</div>

> [@sfischer](#):
>
> Is there any way in Logstash to detect that EOF was reached?

Not really. If you generate document\_id yourself then you could use an aggregate filter to track the most recent record from each file and then generate an upsert to tag the last record after a timeout.

---

<div class="post-metadata">

**Author:** ![sfischer](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@sfischer](https://discuss.elastic.co/u/sfischer)\
**Post date:** [October 13, 2020, 12:15pm UTC](https://discuss.elastic.co/t/eof-in-logstash-file-input/251126/3 "2020-10-13T12:15:18Z")

</div>

Thank you for the tip. I am using the timeout event function of the aggregate filter as you suggested.

However, I have an issue that the event is not stored in the correct index.

I define the index in an event like:

```auto
mutate {
    add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}" } 
}

```

In the output I use this as index:

```auto
output {
	elasticsearch {
		hosts => ["http://127.0.0.1:9200/"]
		user => "elastic"
		password => "changeme"
		index => "%{[@metadata][index]}"
	}
	stdout { codec => json }
}

```

For the timeout event I add the same index as for the other events:

```auto
aggregate {
task_id => "%{[observer][id]}"
code => "
	
	map['index'] = event.get('[@metadata][index]')
"
timeout => 120
push_map_as_event_on_timeout => true
timeout_code => "
	
	event.set('[@metadata][index]', map['index'])
	
	#aggregate other data
"
}

```

However, the timeout event is stored with the index "%{[@metadata][index]}" literally and not replaced correctly. The other events are stored in the correct index.  
There are two fields in the event in elasticsearch then:  
\_index: %{[@metadata][index]}  
index: (With the correct value)

So I guess I am addressing something wrong here, but I can not figure it out.  
Does anyone know what is going on there?

Thanks again!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 13, 2020, 4:17pm UTC](https://discuss.elastic.co/t/eof-in-logstash-file-input/251126/4 "2020-10-13T16:17:20Z")

</div>

> [@sfischer](#):
>
> ```auto
> push_map_as_event_on_timeout => true
> timeout_code => "
> 	
> event.set('[@metadata][index]', map['index'])
> 
> ```

When the timeout\_code executes the map has already been converted into an event (in create\_previous\_map\_as\_event, the .shift that removes the map happens before the call to create\_timeout\_event). Try

```
event.set('[@metadata][index]', event.get('index'))

```

---

<div class="post-metadata">

**Author:** ![sfischer](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@sfischer](https://discuss.elastic.co/u/sfischer)\
**Post date:** [October 14, 2020, 8:12am UTC](https://discuss.elastic.co/t/eof-in-logstash-file-input/251126/5 "2020-10-14T08:12:09Z")

</div>

Ahh okay now it all makes sense.  
Works.  
Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 11, 2020, 8:12am UTC](https://discuss.elastic.co/t/eof-in-logstash-file-input/251126/6 "2020-11-11T08:12:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.