Logstash output is not generating file with current date when timestamp field not available

[sagarpatel]

Hi,

I am trying to create output file using logstash file output plugin but when @timestamp field is not available then it is not genearting file with current date.

Below is working example where i have added date filter and generated @timestamp field. this will generate successfully file name as test-2022-04-06.log

input {
  http_poller {
    urls => {
      login => {
        method => post
        url => "URL"
        headers => {
          "Content-Type" => "application/x-www-form-urlencoded"
        }
        body => "username=uname&password=pwd"
     }
    }
    request_timeout => 120
    schedule => { "every" => "30s"}
    codec => "json"
	cookies => false
  }
}

filter {
  ruby {
    code => "event.set('now_ms', Time.now.to_i * 1000)"
  }

  prune {
    whitelist_names => [ "now_ms", "access_token" ]
  }
  
   mutate {
    convert => {
      "now_ms" => "integer"
    }
  }

  date {
    match => [ "now_ms", "UNIX_MS" ]
	target => "@timestamp"
  }

  prune {
    whitelist_names => [ "@timestamp","now_ms", "access_token"]
  }
}

output {
  file {
	path => "test-%{+YYYY-MM-dd}.log"
  }
}

As using http_poller, I am not expecting @timestamp field in response. So if i removed date filter then it will not generate file name with date and it looks like test-.log

Not working Example

input {
  http_poller {
    urls => {
      login => {
        method => post
        url => "URL"
        headers => {
          "Content-Type" => "application/x-www-form-urlencoded"
        }
        body => "username=uname&password=pwd"
     }
    }
    request_timeout => 120
    schedule => { "every" => "30s"}
    codec => "json"
	cookies => false
  }
}

filter {
  ruby {
    code => "event.set('now_ms', Time.now.to_i * 1000)"
  }

  prune {
    whitelist_names => [ "now_ms", "access_token" ]
  }
  
}

output {
  file {
	path => "test-%{+YYYY-MM-dd}.log"
  }
}

Question:

is @timestamp field is required for generating file name with current date ? or am i missing something here.

[leandrojmp (Leandro Pereira)]

Yes, you will need the @timestamp field in your event, the values of the %{+YYYY-MM-dd} are extracted from the @timestamp field.

If your original event does not have a date field that you can parse into the @timestamp field, logstash will generate a @timestamp field when the event enters the filter section and this field will be used for the values of YYYY-MM-dd.

You should edit your prune filter and allow the @timestamp field if you want to create date based files.

[sagarpatel]

@leandrojmp Thanks for quick reply and help. it is working fine after adding @timestamp field in prune. I have removed date filter as well.