Output to HDFS as Parquet files from Filebeat

Elastic Stack Beats
1

Hi,

I can't seem to find any information on this. Is it possible to write the output from filebeat to HDFS as parquet files? I'm currently writing from filebeat to ES, but would like to write to parquet files as well for longer-term storage.

Thank you.

2

Hi @hjazz6, I think there no parquet file output module available from filebeat.

You can try some workarounds -

You can try to send logs from filebeat to logstash and logstash having Webhdfs output plugin, which sends Logstash events into files in HDFS. But not sure if webhdfs support writing to the parquet file.