Using Logstash to ingest Topbeat data that is output in file mode

Elastic Stack Beats
1

I've setup Topbeat in a mixed environment where some nodes are configured to talk to Logstash directly using the logstash output mode, however I have some nodes that don't have direct connectivity and I am outputting the Topbeat data using the file output mode. I'm currently unsure how I then get these files into my Elasticsearch cluster via Logstash once I've transferred them to a location with direct connectivity, while also maintaining the beats field formatting?

I'm running Logstash with the following configuration file as per the documentation and I've also loaded the topbeat template into my Elasticsearch cluster:

input {
beats {
port => 5044
}
}

output {
elasticsearch {
hosts => "localhost:9200"
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Any suggestions would be gratefully received.

Regards,
Tom

2

I think easiest way to load those files would be to use Logstash with the file input plugin and the json codec. Topbeat outputs in "one json per line" format, which should be compatible with file input + json codec.

3

Thanks for the speedy reply Tudor. I just tried your suggestion and although I couldn't seem to get it working with the file input using the json codec I've got it working with the json filter plugin as per the below example.

Thanks for your help, much appreciated.

input {
file {
path => "/topbeat/*"
}
}

filter {
json {
source => "message"
}
}

output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "topbeat-%{+YYYY.MM.dd}"
}
}