@timestamp field not matching with the Actual log field

Elastic Stack Logstash
1

Hi,

I'm new to ELK, Please help me in getting out of this issue.
I have logs having a "AdmitDate" field. @timestamp field is showing the date and time which I have uploaded the logs.The requirement is, @timestamp field must contain the date and time same as "AdmitDate" field.
For this I have changed the logstash.conf file as below:
input
{
file
{
type => "csv"
tags => "csv"
path =>["Q:/sanjay/Splunk/DB/out_hemas_test.csv"]
start_position => "beginning"
}
}
filter
{
csv
{
columns => ["AdmitDate","IPID","AdmissionNumber","PatientID","RegCode","FirstName","Middlename","PatientName","Age","Gender"]
separator => ","
}

date
{ locale => "en"
match => [ "AdmitDate", "YYYY-MM-DD HH:mm:ss.SSS" ]
}
}
output
{
elasticsearch_http
{
host => "localhost"
index => "logstash-%{+YYYY.MM.dd}"
manage_template => false
template_name => "logstash"
}
stdout{codec => json}
}

Is there any modifications required in logstash.conf file??

Please help.

Thanks

2

That looks right to me... are you sure the date format matches?

3

Thanks for the quick response,

I have changed the date format in KIBANA according to the date format I have in the logs.

Don't know what to do:(

4

I have logs having a "AdmitDate" field. @timestamp field is showing the date and time which I have uploaded the logs.The requirement is, @timestamp field must contain the date and time same as "AdmitDate" field.

Please give an example input line.

5

@magnusbaeck

please find the screenshot of the input logs below

6

@magnusbaeck

The input line from the logs is given below:

AdmitDate, IPID , AdmissionNumber, PatientID , RegCode , FirstName , Middlename
2014-08-09 17:20:43.530 , 381385, OPD.00144616 , 220946, WTID.0000095414 , MANOJA, UDAYANGANI

What should I do to get the @timestamp field same as the "AdmitDate"

7

I think the pattern is failing as 'DD' represents day of the year while 'dd' corresponds to day of the month. Try using the pattern 'YYYY-MM-dd HH:mm:ss.SSS' instead and see if that helps.

Christian

8

Hi @Christian_Dahlqvist

Even after changing the format to 'YYYY-MM-dd HH:mm:ss.SSS', facing the same problem. While uploading the logs the following error is popping up

9

The AdmitDate field appears to have a trailing space.

Please copy/paste text when you can instead of inserting screenshots.

10

Hi @magnusbaeck

please find the sample log file.

AdmitDate , IPID , AdmissionNumber , PatientID , RegCode
2014-08-09 17:20:43.530 , 381385 , OPD.00144616 , 220946, WTID.0000095414
2014-08-09 17:22:40.820 , 381388, OPD.00144619, 71303, WTID.0000011478
2014-08-09 17:23:28.113 , 381391 , OPD.00144620 , 217521, WTID.0000093720
2014-08-09 17:30:39.657 , 381409 , OPD.00144631 , 54810, WTID.0000000270

I'm not able to insert all the fields due to space constraint :frowning:

11

And what if you remove the trailing space with e.g. the mutate filter's gsub option?

1 Like
12

@magnusbaeck please let me know, what should I replace with the trailing space in the AdmitDate

13

You should delete the trailing space.

mutate {
  gsub => ["AdmitDate", "\s+$", ""]
}
1 Like
14

@magnusbaeck

I have done the same. But the following error came while uploading

←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:24:20.667 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:24:20.667 ", :level=>:warn}←[0m
←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:31:31.140 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:31:31.140 ", :level=>:warn}←[0m
←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:38:50.040 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:38:50.040 ", :level=>:warn}←[0m
←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:39:36.393 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:39:36.393 ", :level=>:warn}←[0m
←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:52:30.603 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:52:30.603 ", :level=>:warn}←[0m

Logs are uploaded successfully, but the @timestamp remains same... :frowning:a

15

But you still have the trailing whitespace, but now I see you also have leading whitespace in the AdmitDate field. I suggest you hand-edit one line in the source file and trim the whitespace and retry. If we can confirm that it fixes the problem with can dig into the whitespace again.

16

Hi,
my problem solved by using the following grok command.
grok
{
match => ["AdmitDate", "%{YEAR:al_year}-%{MONTHNUM:al_month}-%{MONTHDAY:al_monthday}
}

thanks @Christian_Dahlqvist @magnusbaeck for the help :smile:

1 Like
17

Hi All,
I am also facing the same problem Logstash is not considering my log date rather making the date at which I am forwarding the logs as @timeStamp field.

@sanju1323 could you please help me in generating the filter as you have did for your date

currently I am using

date {
match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS Z" ]

    }
18

Please start a new topic when asking a new question.

19

Hi @magnusbaeck ,

Will start new discussion next time onwards but as I am facing the similar problem as @sanju1323 So I have pasted my problem on this thread.

20

Solution for this Issue is as shown in below Image.