Date conversion with Logstash

Elastic Stack Logstash
1

Hello, I currently have a problem for date conversion with Logstash.

I receive logs including dates in epoche format (UNIX), so I added the following filters to modify them. Namely, I have multiple date fields, as you can see.

My configuration :

filter {
  json {
    source => "message"
  }
  date {
     match => [ "startDate","UNIX_MS" ]
     target => "startDate"
     timezone => "UTC"
  }
  date {
     match => [ "endDate","UNIX_MS" ]
     target => "endDate"
     timezone => "UTC"
  }
  date {
     match => [ "updatedAt","UNIX_MS" ]
     target => "updatedAt"
     timezone => "UTC"
  }
  date {
     match => [ "createdAt","UNIX_MS" ]
     target => "createdAt"
     timezone => "UTC"
  }
}

In this case, I get, in Elasticsearch, the dates: 2,022 for every fields of date.

On the other hand, if I remove the target in one of the dates, this modifies the @timestamp with the correct date, but each date must keep its field name...

Any ideas ?

2

Most likely your date format coming in isn't UNIX_MS. Can you post a sample of your data?

3

Yes of course. An example of date in json lines :

"startDate":1649751840000
4

Looks like your pipeline works so the next step would be to look at the mapping for those fields.

Conf

input {
  generator {
      lines => [ '{"startDate":1649751840000}' ]
      codec => json
      count => 1
  }
}
filter {
  date {
    match => [ "startDate","UNIX_MS" ]
    target => "startDate"
    timezone => "UTC"
  }  
}
output {
  stdout { codec => json_lines }
}

Output

"startDate": "2022-04-12T08:24:00.000Z"
5

Ok well finally, I chose to take into account only the "startDate" field and convert it with the "@timestamp" target. I don't need the other dates.

However, yes it is indeed strange...

Thanks !

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.