# Using Logstash to ingest Topbeat data that is output in file mode

**URL:** https://discuss.elastic.co/t/using-logstash-to-ingest-topbeat-data-that-is-output-in-file-mode/40278
**Category:** Beats
**Created:** [January 27, 2016, 7:20pm UTC](https://discuss.elastic.co/t/using-logstash-to-ingest-topbeat-data-that-is-output-in-file-mode/40278 "2016-01-27T19:20:42Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![redrise](https://avatars.discourse-cdn.com/v4/letter/r/858c86/32.png) [@redrise](https://discuss.elastic.co/u/redrise)
#### Post date: [January 27, 2016, 7:20pm UTC](https://discuss.elastic.co/t/using-logstash-to-ingest-topbeat-data-that-is-output-in-file-mode/40278/1 "2016-01-27T19:20:42Z")

</div>

I've setup Topbeat in a mixed environment where some nodes are configured to talk to Logstash directly using the logstash output mode, however I have some nodes that don't have direct connectivity and I am outputting the Topbeat data using the file output mode. I'm currently unsure how I then get these files into my Elasticsearch cluster via Logstash once I've transferred them to a location with direct connectivity, while also maintaining the beats field formatting?

I'm running Logstash with the following configuration file as per the documentation and I've also loaded the topbeat template into my Elasticsearch cluster:

input {  
beats {  
port =\> 5044  
}  
}

output {  
elasticsearch {  
hosts =\> "localhost:9200"  
sniffing =\> true  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}  
}

Any suggestions would be gratefully received.

Regards,  
Tom

---

<div class="post-metadata">

### Author: ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)
#### Post date: [January 27, 2016, 7:35pm UTC](https://discuss.elastic.co/t/using-logstash-to-ingest-topbeat-data-that-is-output-in-file-mode/40278/2 "2016-01-27T19:35:16Z")

</div>

I think easiest way to load those files would be to use Logstash with the `file` input plugin and the `json` codec. Topbeat outputs in "one json per line" format, which should be compatible with file input + json codec.

---

<div class="post-metadata">

### Author: ![redrise](https://avatars.discourse-cdn.com/v4/letter/r/858c86/32.png) [@redrise](https://discuss.elastic.co/u/redrise)
#### Post date: [January 27, 2016, 8:29pm UTC](https://discuss.elastic.co/t/using-logstash-to-ingest-topbeat-data-that-is-output-in-file-mode/40278/3 "2016-01-27T20:29:47Z")

</div>

Thanks for the speedy reply Tudor. I just tried your suggestion and although I couldn't seem to get it working with the file input using the json codec I've got it working with the json filter plugin as per the below example.

Thanks for your help, much appreciated.

input {  
file {  
path =\> "/topbeat/\*"  
}  
}

filter {  
json {  
source =\> "message"  
}  
}

output {  
elasticsearch {  
hosts =\> "localhost:9200"  
manage\_template =\> false  
index =\> "topbeat-%{+YYYY.MM.dd}"  
}  
}

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 9:56pm UTC](https://discuss.elastic.co/t/using-logstash-to-ingest-topbeat-data-that-is-output-in-file-mode/40278/4 "2017-07-05T21:56:16Z")

</div>