# How to group logstash output files based on incoming input date?

**URL:** <https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275>\
**Category:** Logstash\
**Created:** [June 3, 2022, 12:58am UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275 "2022-06-03T00:58:38Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![newelastic](https://avatars.discourse-cdn.com/v4/letter/n/b9bd4f/32.png) [@newelastic](https://discuss.elastic.co/u/newelastic)\
**Post date:** [June 3, 2022, 12:58am UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275/1 "2022-06-03T00:58:38Z")

</div>

Hello,

I've thousands of records in my Elasticsearch which span across different dates, month and year. I would like to output the data by year, month and date wise using output plugin.

Here is my pipeline. Can someone help on how to achieve by separating the files by year, month and date?

```auto
input { 
	elasticsearch {	
	hosts => "esDNS:9200"
	index => "transactIndex"
	user => "${ES_USER}"
	password => "${ES_PWD}"
	}
}

output {  

	file
	{
		path => "/elasticData/data/%{+YYYY-MM-dd}}.json"		
		codec => "json_lines"
		gzip => false
	}	
}

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 3, 2022, 1:11am UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275/2 "2022-06-03T01:11:29Z")

</div>

> [@newelastic](#):
>
> `path => "/elasticData/data/%{+YYYY-MM-dd}}.json"`

You've got an extra `}` there at the end. Otherwise, does that not work for you?

---

<div class="post-metadata">

**Author:** ![newelastic](https://avatars.discourse-cdn.com/v4/letter/n/b9bd4f/32.png) [@newelastic](https://discuss.elastic.co/u/newelastic)\
**Post date:** [June 3, 2022, 5:53pm UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275/3 "2022-06-03T17:53:22Z")

</div>

That was a typo in my post. With the above syntax, I was able to get each file created on date. However, I want to create these files separated in folders by year, month and date.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 3, 2022, 6:14pm UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275/4 "2022-06-03T18:14:31Z")

</div>

> [@newelastic](#):
>
> However, I want to create these files separated in folders by year, month and date.

Use `path => "/tmp/%{+YYYY}/%{+MM}/%{+dd}/foo.json"`. The file output will [create](https://github.com/logstash-plugins/logstash-output-file/blob/70f77b9fb89c7c4628647c4b3902a4bc4088d62e/lib/logstash/outputs/file.rb#L260) the directories if they do not exist.

---

<div class="post-metadata">

**Author:** ![newelastic](https://avatars.discourse-cdn.com/v4/letter/n/b9bd4f/32.png) [@newelastic](https://discuss.elastic.co/u/newelastic)\
**Post date:** [June 7, 2022, 12:29am UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275/5 "2022-06-07T00:29:12Z")

</div>

> [@Badger](#):
>
> path =\> "/tmp/%{+YYYY}/%{+MM}/%{+dd}/foo.json"

Thank you ! This worked and able to create multiple folders by date and month.

Another question, how do I enforce logstash to use current system date as YYYY, MM & DD is being used from my input @timestamp field which holds another value.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 7, 2022, 12:50am UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275/6 "2022-06-07T00:50:27Z")

</div>

At the very start of the filter section (assuming @timestamp has not been set) do

```
mutate { add_field => { "[@metadata][filePath]" => "/tmp/%{+YYYY}/%{+MM}/%{+dd}/foo.json" } }

```

The use `path => "%{[@metadata][filePath]}"` in the file output. If @timestamp is set before the event hits the pipeline (e.g. by a json codec) then you would have to use ruby. I have not tested it but something like

```
ruby { code => 'event.set("[@metadata][filePath]", DateTime.now.strftime("/tmp/%y/%y/%d/foo.json"))' }
```

---

<div class="post-metadata">

**Author:** ![newelastic](https://avatars.discourse-cdn.com/v4/letter/n/b9bd4f/32.png) [@newelastic](https://discuss.elastic.co/u/newelastic)\
**Post date:** [June 7, 2022, 10:56pm UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275/7 "2022-06-07T22:56:20Z")

</div>

Thank you very much and I really appreciate quick responses here. I'm new to this technology and can't tell how much I'm loving and learning through this community ! It's a great community !

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2022, 10:57pm UTC](https://discuss.elastic.co/t/how-to-group-logstash-output-files-based-on-incoming-input-date/306275/8 "2022-07-05T22:57:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.