@timestamp field not matching with the Actual log field


(Sanjay Reddy) #1

Hi,

I'm new to ELK, Please help me in getting out of this issue.
I have logs having a "AdmitDate" field. @timestamp field is showing the date and time which I have uploaded the logs.The requirement is, @timestamp field must contain the date and time same as "AdmitDate" field.
For this I have changed the logstash.conf file as below:
input
{
file
{
type => "csv"
tags => "csv"
path =>["Q:/sanjay/Splunk/DB/out_hemas_test.csv"]
start_position => "beginning"
}
}
filter
{
csv
{
columns => ["AdmitDate","IPID","AdmissionNumber","PatientID","RegCode","FirstName","Middlename","PatientName","Age","Gender"]
separator => ","
}

date
{ locale => "en"
match => [ "AdmitDate", "YYYY-MM-DD HH:mm:ss.SSS" ]
}
}
output
{
elasticsearch_http
{
host => "localhost"
index => "logstash-%{+YYYY.MM.dd}"
manage_template => false
template_name => "logstash"
}
stdout{codec => json}
}

Is there any modifications required in logstash.conf file??

Please help.

Thanks


@timestamp field not matching with the Actual log field in Kibana
Graph time series data (Beginner)
@timestamo and Date in the logs are not matching
(Antonio Bonuccelli) #2

That looks right to me... are you sure the date format matches?


(Sanjay Reddy) #3

Thanks for the quick response,

I have changed the date format in KIBANA according to the date format I have in the logs.

Don't know what to do:(


(Magnus Bäck) #4

I have logs having a "AdmitDate" field. @timestamp field is showing the date and time which I have uploaded the logs.The requirement is, @timestamp field must contain the date and time same as "AdmitDate" field.

Please give an example input line.


(Sanjay Reddy) #5

@magnusbaeck

please find the screenshot of the input logs below


(Sanjay Reddy) #6

@magnusbaeck

The input line from the logs is given below:

AdmitDate, IPID , AdmissionNumber, PatientID , RegCode , FirstName , Middlename
2014-08-09 17:20:43.530 , 381385, OPD.00144616 , 220946, WTID.0000095414 , MANOJA, UDAYANGANI

What should I do to get the @timestamp field same as the "AdmitDate"


(Christian Dahlqvist) #7

I think the pattern is failing as 'DD' represents day of the year while 'dd' corresponds to day of the month. Try using the pattern 'YYYY-MM-dd HH:mm:ss.SSS' instead and see if that helps.

Christian


(Sanjay Reddy) #8

Hi @Christian_Dahlqvist

Even after changing the format to 'YYYY-MM-dd HH:mm:ss.SSS', facing the same problem. While uploading the logs the following error is popping up


(Magnus Bäck) #9

The AdmitDate field appears to have a trailing space.

Please copy/paste text when you can instead of inserting screenshots.


(Sanjay Reddy) #10

Hi @magnusbaeck

please find the sample log file.

AdmitDate , IPID , AdmissionNumber , PatientID , RegCode
2014-08-09 17:20:43.530 , 381385 , OPD.00144616 , 220946, WTID.0000095414
2014-08-09 17:22:40.820 , 381388, OPD.00144619, 71303, WTID.0000011478
2014-08-09 17:23:28.113 , 381391 , OPD.00144620 , 217521, WTID.0000093720
2014-08-09 17:30:39.657 , 381409 , OPD.00144631 , 54810, WTID.0000000270

I'm not able to insert all the fields due to space constraint :frowning:


(Magnus Bäck) #11

And what if you remove the trailing space with e.g. the mutate filter's gsub option?


(Sanjay Reddy) #12

@magnusbaeck please let me know, what should I replace with the trailing space in the AdmitDate


(Magnus Bäck) #13

You should delete the trailing space.

mutate {
  gsub => ["AdmitDate", "\s+$", ""]
}

(Sanjay Reddy) #14

@magnusbaeck

I have done the same. But the following error came while uploading

←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:24:20.667 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:24:20.667 ", :level=>:warn}←[0m
←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:31:31.140 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:31:31.140 ", :level=>:warn}←[0m
←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:38:50.040 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:38:50.040 ", :level=>:warn}←[0m
←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:39:36.393 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:39:36.393 ", :level=>:warn}←[0m
←[33mFailed parsing date from field {:field=>"AdmitDate", :value=>" 2014-08-09 1
8:52:30.603 ", :exception=>java.lang.IllegalArgumentException: Invalid format: "
2014-08-09 18:52:30.603 ", :level=>:warn}←[0m

Logs are uploaded successfully, but the @timestamp remains same... :frowning:a


(Magnus Bäck) #15

But you still have the trailing whitespace, but now I see you also have leading whitespace in the AdmitDate field. I suggest you hand-edit one line in the source file and trim the whitespace and retry. If we can confirm that it fixes the problem with can dig into the whitespace again.


(Sanjay Reddy) #16

Hi,
my problem solved by using the following grok command.
grok
{
match => ["AdmitDate", "%{YEAR:al_year}-%{MONTHNUM:al_month}-%{MONTHDAY:al_monthday}
}

thanks @Christian_Dahlqvist @magnusbaeck for the help :smile:


(Yashpal Singh) #17

Hi All,
I am also facing the same problem Logstash is not considering my log date rather making the date at which I am forwarding the logs as @timeStamp field.

@sanju1323 could you please help me in generating the filter as you have did for your date

currently I am using

date {
match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS Z" ]

    }

(Magnus Bäck) #18

Please start a new topic when asking a new question.


(Yashpal Singh) #19

Hi @magnusbaeck ,

Will start new discussion next time onwards but as I am facing the similar problem as @sanju1323 So I have pasted my problem on this thread.


(Mahesh Dn) #20

Solution for this Issue is as shown in below Image.