How to group logstash output files based on incoming input date?

[newelastic (elastic user)]

Hello,

I've thousands of records in my Elasticsearch which span across different dates, month and year. I would like to output the data by year, month and date wise using output plugin.

Here is my pipeline. Can someone help on how to achieve by separating the files by year, month and date?

input { 
	elasticsearch {	
	hosts => "esDNS:9200"
	index => "transactIndex"
	user => "${ES_USER}"
	password => "${ES_PWD}"
	}
}

output {  

	file
	{
		path => "/elasticData/data/%{+YYYY-MM-dd}}.json"		
		codec => "json_lines"
		gzip => false
	}	
}

[warkolm (Mark Walkom)]

@newelastic

path => "/elasticData/data/%{+YYYY-MM-dd}}.json"

You've got an extra } there at the end. Otherwise, does that not work for you?

[newelastic (elastic user)]

That was a typo in my post. With the above syntax, I was able to get each file created on date. However, I want to create these files separated in folders by year, month and date.

[Badger]

@newelastic

However, I want to create these files separated in folders by year, month and date.

Use path => "/tmp/%{+YYYY}/%{+MM}/%{+dd}/foo.json". The file output will create the directories if they do not exist.

[newelastic (elastic user)]

@Badger

path => "/tmp/%{+YYYY}/%{+MM}/%{+dd}/foo.json"

Thank you ! This worked and able to create multiple folders by date and month.

Another question, how do I enforce logstash to use current system date as YYYY, MM & DD is being used from my input @timestamp field which holds another value.

[Badger]

At the very start of the filter section (assuming @timestamp has not been set) do

mutate { add_field => { "[@metadata][filePath]" => "/tmp/%{+YYYY}/%{+MM}/%{+dd}/foo.json" } }

The use path => "%{[@metadata][filePath]}" in the file output. If @timestamp is set before the event hits the pipeline (e.g. by a json codec) then you would have to use ruby. I have not tested it but something like

ruby { code => 'event.set("[@metadata][filePath]", DateTime.now.strftime("/tmp/%y/%y/%d/foo.json"))' }

[newelastic (elastic user)]

Thank you very much and I really appreciate quick responses here. I'm new to this technology and can't tell how much I'm loving and learning through this community ! It's a great community !