I've thousands of records in my Elasticsearch which span across different dates, month and year. I would like to output the data by year, month and date wise using output plugin.
Here is my pipeline. Can someone help on how to achieve by separating the files by year, month and date?
That was a typo in my post. With the above syntax, I was able to get each file created on date. However, I want to create these files separated in folders by year, month and date.
Thank you ! This worked and able to create multiple folders by date and month.
Another question, how do I enforce logstash to use current system date as YYYY, MM & DD is being used from my input @timestamp field which holds another value.
The use path => "%{[@metadata][filePath]}" in the file output. If @timestamp is set before the event hits the pipeline (e.g. by a json codec) then you would have to use ruby. I have not tested it but something like
Thank you very much and I really appreciate quick responses here. I'm new to this technology and can't tell how much I'm loving and learning through this community ! It's a great community !
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.