Multiple Date fields in logstash

Elastic Stack Logstash
1

Hi,

I have a small query regarding date field.

In my data file there are two date fields like
submit_date : 2012-12-22 12:53:30.000
release_date: 2012-12-31 12:53:30.000

for this I tried using the following in the logstash configuration file.

date {
locale => "en"
match => [ "submit_date", "YYYY-MM-dd HH:mm:ss.SSS"]
}

date {
locale => "en"
match => [ "release_date", "YYYY-MM-dd HH:mm:ss.SSS"]
}

But still it is considering only submit date and release_date is showing as string.

I want to take both submit_date and release_date in the date format.

Please help

Thanks & Regards,
Sanjay Reddy

2

By default the date filter stores the result of the parsing in the @timestamp field. If you want it stored elsewhere you need to set the filter's target parameter.

3

@magnusbaeck Can you please give an example to set the filter's target parameter for date

4 
date {
  match => ["submit_date", "YYYY-MM-dd HH:mm:ss.SSS"]
  target => "submit_date"
}
5

Thanks @magnusbaeck that worked...
Thankyou very much :smile:

6

@magnusbaeck

As suggested I have added another date field with target parameter in the logstash.conf file.

If there is null in the "ReleaseDate" those logs are not indexing. Because of that only few records are indexing and the remaining records are not indexing. But I want all the records to be indexed.

Can you please help in this.

7

If certain messages are dropped I doubt it's related to a "null" ReleaseDate field (do you mean a missing field?), but you can use a conditional to only use the date filter when the field is set.

if [ReleaseDate] {
  date {
    ...
  }
}

If you still have problems please provide a complete example that exhibits your problem.

8

I have a very similar question about having multiple dates in my data.

I have a Start_Date_Time and an End_Date_Time. If I use the date plugin to shove both of them into the @timestamp so I can convert them both with the target option to store them as date/times in the data instead of strings (thanks for that tip, since it was what I was looking for), which @timestamp goes into ElasticSearch or do they both go in as two different @timestamps? I'm assuming the last one in the code would be the one stored as typically code overwrites previous code; however, want to be sure before changing the logstash files that I've inherited since this could really muck up my data if I get it wrong. Thanks in advance!

9

@L2W, please post unrelated questions in new topics.