# Dumping logs to a file based on timestamp field

**URL:** <https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165>\
**Category:** Logstash\
**Created:** [September 12, 2015, 6:46am UTC](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165 "2015-09-12T06:46:17Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Deb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deb/32/46162_2.png) [@Deb](https://discuss.elastic.co/u/Deb)\
**Post date:** [September 12, 2015, 6:46am UTC](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165/1 "2015-09-12T06:46:17Z")

</div>

I am trying to dump log events to a file having name of the format, `hello-YYYY-MM-DD.log` using the file output plugin. I want to get YYYY-MM-DD from a field named `timestamp` contained in the log event (not `@timestamp` field). `timestamp` contains date in the form `2015-09-12T00:00:01.919+05:30`.

I tried to add a new field so that I can use it the file path name using the mutate filter like below. But that did not succeed

```
mutate {
      add_field => { "app_log_time" => "%{timestamp}.%{+YYYY-MM-dd}" }
  }

```

Also tried using ruby code filter to add a new filter but that did not succeed as well.

Can someone let me know how can I achieve this ( preferably without adding a new field )?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 12, 2015, 7:10am UTC](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165/2 "2015-09-12T07:10:17Z")

</div>

You _will_ have to add a new field, but if you add it as a subfield of `@metadata` it won't become part of the message that's sent to the outputs. You can use a grok filter to extract the date from the `timestamp` field.

```
grok {
  match => [
    "timestamp",
    "^(?<[@metadata][app_log_time]>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
  ]
}

```

Then use that field in your output filename:

```
file {
  path => "/foo/bar/hello-%{[@metadata][app_log_time]}.log"
}
```

---

<div class="post-metadata">

**Author:** ![Deb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deb/32/46162_2.png) [@Deb](https://discuss.elastic.co/u/Deb)\
**Post date:** [September 13, 2015, 7:28am UTC](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165/3 "2015-09-13T07:28:13Z")

</div>

Thanks again @magnusbaeck.

But If I use the below grok pattern:-

```
grok {
      match => [
        "timestamp", "^(?<[@metadata][app_log_time]>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
      ]
    }

```

I am seeing the below error in console:-

```
The error reported is: 
  invalid char in group name <[@metadata][app_log_time]>: /^(?<[@metadata][app_log_time]>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])))/m

```

However doing `--configtest` reports everything as OK.

If I place just `app_log_time` in the grok pattern as shown below it works as expected and I see a `app_log_time` in the event

```
grok {
      match => [
        "timestamp", "^(?<app_log_time>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
      ]
 }
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 13, 2015, 8:55am UTC](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165/4 "2015-09-13T08:55:06Z")

</div>

Okay. I thought you could use the subfield notation in named capture groups but apparently not. If you don't want the `app_log_time` field to show up in the output events you can always rename it to `[@metadata][app_log_time]` after the grok filter.

---

<div class="post-metadata">

**Author:** ![Deb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deb/32/46162_2.png) [@Deb](https://discuss.elastic.co/u/Deb)\
**Post date:** [September 13, 2015, 12:45pm UTC](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165/5 "2015-09-13T12:45:51Z")

</div>

Yeah @magnusbaeck I am doing the same. Can we make `--configtest` more robust to catch these errors?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 13, 2015, 2:36pm UTC](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165/6 "2015-09-13T14:36:44Z")

</div>

`--configtest` isn't really meant to catch _all_ configuration errors. For example, it doesn't peek into the semantics of each parameter. Feel free to file a bug and we'll see what the maintainers say, but it may not be a trivial endeavor to fix this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:29am UTC](https://discuss.elastic.co/t/dumping-logs-to-a-file-based-on-timestamp-field/29165/7 "2017-07-06T05:29:15Z")

</div>