Ignore date offset in logstash

yishain11 · March 11, 2018, 9:03am

HI, I have the following date format:
"2017-12-01 04:39:34-04:000"

Currently, I use this configuration:

date{
                match => ["[documentsList][0][commandTimestamp]","yyyy-MM-dd HH:mm:ssZZ'0'"]
                target => ["command_timestamp"]
                remove_field => ["%{timestamp}"]
        }

Now I want to ignore the date offset, i.e. ignore the "-04:000".
When I try to change the configuration to:

date{
                match => ["[documentsList][0][commandTimestamp]","yyyy-MM-dd HH:mm:ss"]
                target => ["command_timestamp"]
                remove_field => ["%{timestamp}"]
        }

I get date parse failure.
Why is this happening? How can I fix it?

immavalls · March 11, 2018, 9:25am

This might help: Want to capture timeStamp without milliseconds.

yishain11 · March 11, 2018, 12:25pm

HI, Thanks for the answer.
I tried:

    mutate{
            copy => {"[documentsList][0][commandTimestamp]" => "[documentsList][0][noffset_commandTimestamp]"}

            gsub => ["[documentsList][0][noffset_commandTimestamp]","/.{8}$/g",""]
    }

    date{
            match => ["[documentsList][0][noffset_commandTimestamp]","yyyy-MM-dd HH:mm:ss"]
            target => ["no_offset_command_timestamp"]
    }

and the no offset timestamp remain unchanged.
I tried also in the gsub line:
/-\d{2}+:\d{3}/g

to no avial.
I tried also without the "/" on the start and "/g" in the end. Same result.

Any advice?

magnusbaeck · March 11, 2018, 7:34pm

Don't assume that the options to a mutate filter execute in the order given. They always execute in a fixed order:

github.com

logstash-plugins/logstash-filter-mutate/blob/v3.3.1/lib/logstash/filters/mutate.rb#L247-L265


def filter(event)
  coerce(event) if @coerce
  rename(event) if @rename
  update(event) if @update
  replace(event) if @replace
  convert(event) if @convert
  gsub(event) if @gsub
  uppercase(event) if @uppercase
  capitalize(event) if @capitalize
  lowercase(event) if @lowercase
  strip(event) if @strip
  remove(event) if @remove
  split(event) if @split
  join(event) if @join
  merge(event) if @merge
  copy(event) if @copy


  filter_matched(event)
end

So, gsub runs before copy. Split your mutate filter in two. Secondly, the gsub expression is wrong. Try this:

gsub => ["[documentsList][0][noffset_commandTimestamp]", ".{8}$", ""]

system · April 8, 2018, 7:34pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.