Importing old IIS logfiles. Timestamp is today instead of date/time in logfile

Elastic Stack Logstash
1

I must be a complete idiot - I have been trying to get this working for 2 days.

I have load of old IIS log files that I wish to import into elasticsearch-1.7.3 using logstash_1.5.4-1 and view with kibana-4.1.2. I am using ubuntu-14.04.3. All 3 applications in my 'ELK stack' are on the same machine at the moment as Iam just testing.

I have managed to get nxlog-ce-2.8.1248 (on my Windows 2012 R2 IIS server) to sucessfully connect to logstash and I can see the data in Kibana.

However, the dates are today's import date and not the timestamps from the IIS files. So for today I have 60 million 'hits' showing in Kibana and 0 hits for the previous days/months (as I expected)

I am going round and round in circles trying to see what I am doing wrong and it is driving me a little crazy.

my /etc/logstash/conf.d/logstash-iis.conf file is

input {  
	tcp {
		type => "iis"
		port => "3516"
		host => "10.4.10.8"
	}
}

filter {
 
  #ignore log comments
  if [message] =~ "^#" {
    drop {}
  }

grok {
    match => ["@message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referrer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
	}
  
  #Set the Event Timestamp from the log
	date {
		match => [ "timestamp", "YYYY-MM-dd HH:mm:ss" ]
		target => "@timestamp"
		timezone => "Europe/London"
    }	

useragent {
	source=> "useragent"
	prefix=> "browser"
	}
	
mutate {
	remove_field => [ "timestamp"]
    }
}

output {  
    elasticsearch_http {
        host => "10.4.10.8"
        port => "9200"
    }
}

I am using the default IIS logging setup - no additional fields. Sample of the logs:

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2015-07-31 00:00:01
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2015-07-31 00:00:01 10.2.10.11 GET / - 80 - 10.2.10.1 - - 200 0 64 0
2015-07-31 00:00:03 10.2.10.11 GET / - 80 - 10.2.10.2 - - 200 0 64 0
2015-07-31 00:00:06 10.2.10.11 GET / - 80 - 10.2.10.1 - - 200 0 64 0
2015-07-31 00:00:08 10.2.10.11 GET / - 80 - 10.2.10.2 - - 200 0 64 0

Kibana is showing this:

message:2015-07-31 00:00:01 10.2.10.11 GET / - 80 - 10.2.10.1 - - 200 0 64 0 @version:1 @timestamp:October 29th 2015, 15:32:32.144 host:10.4.10.40 type:iis tags:_grokparsefailure browsername:Other browseros:Other browseros_name:Other browserdevice:Other _id:AVC0Op8YFei46efN7GTq _type:iis _index:logstash-2015.10.29

No errors in the nxlog or logstash logfiles.

I would appreciate ANY help as I am googling myself into ever decreasing circles! Please can someone put me out of my misery.

Many Thanks
Martin

2

As evidenced by the _grokparsefailure tag the message gets, your grok filter isn't matching correctly. Start with changing the filter from inspecting the @message field because you have no such field in your messages (it's message).

1 Like
3

Thank you Magnus.

That solved my problem. I never would have found that as I assumed the @ was required!

I am so grateful that you responded.

Have a fab weekend.

Martin.