I am currently index data into Elasticsearch although I'm looking for a simple method to also store logs on an HDFS cluster in parquet format for longer term storage and to allow for search/map reduce jobs over many years. I know logstash has the avro output codec although I specifically want the parquet format because of it's columnar store properties. Are there any plans to create a parquet codec in logstash? I would really like to have a single solution to do data transformation if possible but i could also likely use Kafka streams to do this.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.