Dumping logs to a file based on timestamp field

Elastic Stack Logstash
1

I am trying to dump log events to a file having name of the format, hello-YYYY-MM-DD.log using the file output plugin. I want to get YYYY-MM-DD from a field named timestamp contained in the log event (not @timestamp field). timestamp contains date in the form 2015-09-12T00:00:01.919+05:30.

I tried to add a new field so that I can use it the file path name using the mutate filter like below. But that did not succeed

mutate {
      add_field => { "app_log_time" => "%{timestamp}.%{+YYYY-MM-dd}" }
  }

Also tried using ruby code filter to add a new filter but that did not succeed as well.

Can someone let me know how can I achieve this ( preferably without adding a new field )?

2

You will have to add a new field, but if you add it as a subfield of @metadata it won't become part of the message that's sent to the outputs. You can use a grok filter to extract the date from the timestamp field.

grok {
  match => [
    "timestamp",
    "^(?<[@metadata][app_log_time]>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
  ]
}

Then use that field in your output filename:

file {
  path => "/foo/bar/hello-%{[@metadata][app_log_time]}.log"
}
3

Thanks again @magnusbaeck.

But If I use the below grok pattern:-

grok {
      match => [
        "timestamp", "^(?<[@metadata][app_log_time]>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
      ]
    }

I am seeing the below error in console:-

The error reported is: 
  invalid char in group name <[@metadata][app_log_time]>: /^(?<[@metadata][app_log_time]>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])))/m

However doing --configtest reports everything as OK.

If I place just app_log_time in the grok pattern as shown below it works as expected and I see a app_log_time in the event

grok {
      match => [
        "timestamp", "^(?<app_log_time>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
      ]
 }
4

Okay. I thought you could use the subfield notation in named capture groups but apparently not. If you don't want the app_log_time field to show up in the output events you can always rename it to [@metadata][app_log_time] after the grok filter.

5

Yeah @magnusbaeck I am doing the same. Can we make --configtest more robust to catch these errors?

6

--configtest isn't really meant to catch all configuration errors. For example, it doesn't peek into the semantics of each parameter. Feel free to file a bug and we'll see what the maintainers say, but it may not be a trivial endeavor to fix this.