How to import 'File outputed' Metricbeat data into ES via Logstash

[Patrick_Stevens]

Hi, I am attempting to use Logstash to import Metricbeat 'file output' files into Elasticsearch. I am using file output on Metricbeat because the servers I'm monitoring are on a different domain than my ES, Logstash, and Kibana installations, and direct connectivity is not possible (this is due to my current corporate network security constraints, and won't be changing soon...). I have the Metricbeat file output configured to roll files every 20 mb, and I'll be using Powershell scripts to migrate those completed output files into a staging directory on my Logstash server.

My question is, how can I configure Logstash to consume these files, create indexes based off the event timestamps, and drop the data into the appropriate indexes? The Metricbeat file output page in the documentation said it was possible to consume those files with Logstash, but offered no guidance on how to configure the pipeline.

I am running 5.5.1 for all ELK components. Any help would be greatly appreciated

[leandrojmp (Leandro Pereira)]

I think that the Metricbeat file output is formatted as a json, then you can configure your logstash input to read this file and use the json codec to parse it.

input {
    file {
        path => "/path/to/your/metric/files/*"
        codec => "json"
    }
}

Can you test that? I don't know if you can need the json codec or the json_lines, but testing will show you which of them works, or if it will really work.

[Patrick_Stevens]

Hi Leandro, thank you for your reply. The "json_lines" codec worked for reading the file, but now my logstash appears to be having issues pushing that data to elasticsearch. I get an endless loop of the following console messages after the file has been read. The file size is approximately 20 MB- could that be why I am getting all these messages? Or is it a configuration error?

my pipeline configuration is the following:

input {
file {
path => "C:/Logstash/logstash-5.5.1/data/metricbeat/*"
codec => "json_lines"
start_position => "beginning"
}
}
output {
elasticsearch {
hosts => ["http://myserver:9200"]
template_name => "metricbeat"
}
stdout { codec => rubydebug }
}

Are there any metadata fields that I need to add to the events so that the metricbeat data gets consumed properly? The 'file outputed' data appears to just be the raw data, without any kind of metricbeat specific metadata.

[leandrojmp (Leandro Pereira)]

Hello,

Well, this look like the normal behaviour, it's reading the file in a periodic interval.

Is the file getting written with new metrics from metricbeat? What do you have in elasticsearch?

Can you post an event of the metricbeat output file (the logstash input) and the logstash output?

[Patrick_Stevens]

Hey Leandro, thanks for the help but I figured it out. I had a few config errors in filebeat and the ES ingest pipeline that I was able to identify with some more reference material reading