Logstash Timestamp date field filter not parsing properly

Elastic Stack Logstash
1

Hello,

I'm unable to parse the Timestamp field from logstash configuration,

I didn't see any errors while running configuration but the date field is not matching my index field same as in IIS logs.

All my logs are appending into today's date as mentioned below but not IIS log date.

For Ex.

My Logstash Lumberjack Outconf file:

output {

stdout { codec => rubydebug }
 
if [URI] != ""{
  elasticsearch {
   host => "XXXX.XXXX.com"
   cluster => "SSTLogs"
   protocol=> "http"
   index => "weblogs-%{+YYYY.MM.dd}"
   manage_template => "false"
  }
 }

Logstash Filter Conf:

filter {
	grok {

	match => {
		'message' => '\A%{TIMESTAMP_ISO8601:DateTime}%{SPACE}%{URIPATHPARAM:URI}%{SPACE}%{INT:Status}%{SPACE}%{QUOTEDSTRING:ComputerName}%{SPACE}%{QUOTEDSTRING:Referer}%{SPACE}%{INT:Win32Status}%{SPACE}%{NUMBER:BytesSent}%{SPACE}%{NUMBER:BytesReceived}%{SPACE}%{QUOTEDSTRING:UserAgent}%{SPACE}%{IP:ServerIP}%{SPACE}%{INT:ServerPort}%{SPACE}%{QUOTEDSTRING:Protocol}%{SPACE}%{PROG:Method}%{SPACE}%{IP:ClientIP}%{SPACE}%{NUMBER:TimeTaken}%{SPACE}%{NUMBER:RequestPerSecond}%{SPACE}%{HOSTNAME:WebSiteName}%{SPACE}%{GREEDYDATA:QRY}'

	}
}
    date {

    		locale => "en"
    		match => ["DateTime", "YYYY-MM-dd;HH:mm:ss.SSS"]
    		target => "@timestamp"
    	}

Please help.

2

Please show an example of a log entry that isn't processed correctly.

3

Thanks magnus,

If you see the above screenshot which is showing today's date i mean current date but for me it should show IIS logs date

I was trying to import 1st January 2017 logs so the index should be weblogs-2107.01.01 and also for next day import it should create next index as weblogs-2017.01.02 and so on.

4

If you want help please provide the information I ask for.

5

When I process the 1st January weblogs the log entry is current date weblogs-2017.06.27 which is not supposed to be.

Here is the example of processed weblogs:

am I getting into right direction of your answer.

6

No! I don't care what your index is named or how big it is, I want to see what your logs look like.

7

My IIS log file -Here you go.

#Software: IIS Advanced Logging Module
#Version: 1.0
#Start-Date: 2017-01-01 00:00:00.531
#Fields:  date time cs-uri-stem sc-status s-computername cs(Referer) sc-win32-status sc-bytes cs-bytes cs(User-Agent) s-ip s-port c-protocol cs-method X-Forwarded-For TimeTakenMS RequestsPerSecond cs(Host) cs-uri-query
2017-01-01 00:00:06.942 /404.asp 404 "29WEB01" - 0 3829 406 "Java/1.8.0_31" 10.181.130.191 443 "https" GET - 46  0 "www2.xxx.com" 404;https://www2.xxxx.com:443/wcms/codeReports/SCNR_Nail_strongdrive_specialtyfasteners.html
2017-01-01 00:00:06.942 /wcms/codeReports/SCNR_Nail_strongdrive_specialtyfasteners.html 404 "29WEB01" - -2147024894 3829 406 "Java/1.8.0_31" 10.181.130.191 443 "https" GET - 62  0 "www2.xxxx.com" -
2017-01-01 00:00:07.145 /wcms/loadtables/SCNR_Nail_strongdrive_specialtyfasteners.html 200 "29WEB01" - 0 6869 405 "Java/1.8.0_31" 10.181.130.191 443 "https" GET - 31  0 "www2.xxxx.com" -
2017-01-01 00:00:07.161 /404.asp 404 "29WEB01" - 0 3829 403 "Java/1.8.0_31" 10.181.130.191 443 "https" GET - 46  0 "www2.xxxx.com" 404;https://www2.xxxx.com:443/wcms/drawings/SCNR_Nail_strongdrive_specialtyfasteners.html
2017-01-01 00:00:07.161 /wcms/drawings/SCNR_Nail_strongdrive_specialtyfasteners.html 404 "29WEB01" - -2147024894 3829 403 "Java/1.8.0_31" 10.181.130.191 443 "https" GET - 46  0 "www2.xxxx.com" -
2017-01-01 00:00:10.327 /graphics/highwind2009/diagrams/D1.gif 200 "29WEB01" "https://www.google.com/" 0 39711 380 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" 10.181.130.191 443 "https" GET - 31  0 "www2.xxxx.com" -
2017-01-01 00:00:20.717 /contact_us.asp 200 "29WEB01" - 0 10371 490 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 10.181.130.191 80 "http" GET "66.249.65.104" 15  0 "www.xxxx.com.au" source=topnav
2017-01-01 00:00:45.100 /financials/news.html 200 "29WEB01" - 0 8295 388 "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36" 10.181.130.191 80 "http" GET "178.82.120.26" 46  0 "www.xxxx.com" source=topnav
2017-01-01 00:00:45.880 /financials/reporting.html 200 "29WEB01" - 0 13921 382 "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.60 Safari/537.1" 10.181.130.191 80 "http" GET "23.253.151.250" 62 0.9862 "www.xxxx.com" source=topnav
2017-01-01 00:00:52.603 / 301 "29WEB01" - 0 404 458 "Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)" 10.181.130.191 80 "http" GET "199.58.86.211" 0  0 "www2.xxxx.com" -
2017-01-01 00:00:54.959 /robots.txt 200 "29WEB01" - 0 1077 166 "Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)" 10.181.130.191 443 "https" GET - 46  0 "www2.xxxx.com" -
2017-01-01 00:00:56.909 / 301 "29WEB01" - 0 404 343 "Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)" 10.181.130.191 443 "https" GET - 0  0 "www2.xxxx.com" -

Let me know if you need more information.

8

Your date pattern clearly doesn't match your timestamp so the filter fails (there's no semicolon between the date and the time). When this happens it adds a _dateparsefailure tag to the event. If you look in your Logstash logs the date filter will point you to where the parser fails.

9

I ran the config test and its says Configuration ok and I don't see any error in my logstash.log as well,

Can you please guide me as you mentioned where to change the semicolon exactly as I was new to this elastic search.

thanks.

10

I ran the config test and its says Configuration ok

The configuration test can't possibly catch this kind of problem.

and I don't see any error in my logstash.log as well,

I'm sure there's something in there.

Can you please guide me as you mentioned where to change the semicolon exactly as I was new to this Elasticsearch.

Here's a line from your date filter configuration:

match => ["DateTime", "YYYY-MM-dd;HH:mm:ss.SSS"]

Here's what the timestamp from your log look like:

2017-01-01 00:00:06.942

Remove the semicolon from your date pattern. Over and out.

11

Yes I tried with the below, no luck.

match => ["DateTime", "YYYY-MM-dd HH:mm:ss.SSS"]

still i see the index pattern date with current date weblogs-2017.06.28

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.