Filebeat Grok for Date IIS logs

Elastic Stack Beats
1

Hi everyone

I am trying to import IIS Logs, for now literally one field which is the date and time. Here is what my log file looks like:
#Software: IIS Advanced Logging Module
#Version: 1.0
#Start-Date: 2018-09-07 05:33:33.297
#Fields: date-local time-local
2018-09-07 07:33:32.841
2018-09-07 07:37:04.393

As you can see the last 2 lines are the date and time. I tested this pattern:
%{TIMESTAMP_ISO8601:iis_custom.advanced_logs.logtime}
online at http://grokconstructor.appspot.com and it passed using the below date and time in the log:
2018-09-07 07:33:32.841

But when I put it in filebeat I get the following error in Kibana (filebeat, started with the -e -d "*" flags are not giving me any errors. Only in Kibana can I see this error:

Provided Grok expressions do not match field value: [2018-09-07 07:53:14.908]

Below is my full filebeat ingest.json. Any help will greatly be appreciated!

Thank you
{
"description": "Pipeline for parsing custom iis logs",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"%{TIMESTAMP_ISO8601:iis_custom.advanced_logs.logtime}"
],
"ignore_missing": true
}
},{
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "iis_custom.advanced_logs.logtime",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:H:m:s Z"]
}
}, {
"remove": {
"field": "iis_custom.advanced_logs.logtime"
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}

2

The format defined in the date parser does not match the timestamp you provided.

This is the correct format for your timestamp:

"date": {
    "field": "iis_custom.advanced_logs.logtime",
    "target_field": "@timestamp",
    "formats": ["yyyy-MM-dd HH:mm:ss.SSS"]
}
3

Hi Noemi

Thank you so much for the quick reply. I will test it out Monday morning when I'm back at work!

Best regards

4

Have a nice weekend! :slight_smile:

5

Hi Noemi

I did what you suggested, but unfortunately I am getting the same error:

Kibana%20Error
Kibana Error.PNG823×430 25.6 KB

Here is my full ingest.json file and the log file (I removed all other fields and kept only the date and time to make it simple for now). I understand that the lines starting with a # will give an error, but I am OK with that for now.

IIS Logs:
#Software: IIS Advanced Logging Module
#Version: 1.0
#Start-Date: 2018-09-10 09:43:25.684
#Fields: date-local time-local
2018-09-10 11:43:25.243
2018-09-10 11:43:26.834
2018-09-10 11:43:26.850
2018-09-10 11:43:26.912

I had a look at my custom Apache, PHP, and Python ingest.json files, and for some of them I specified the datetime part as DATA. I guess way back I must've had issues as well with them. Their data formats are all like this:

"date": {
"field": "apache2_php.access.time",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:H:m:s Z"]
}

Could it be that the @timestamp format in Kibana expects the date to be in format ["dd/MMM/YYYY:H:m:s Z"], but that my Grok data type is not correct? (PS, I also tried data type DATA; I see for one of my other logs it is DATA and it actually maps correctly the @timestamp field)

My ingest.json file:

"description": "Pipeline for parsing custom iis logs",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"%{TIMESTAMP_ISO8601:iis_custom.advanced_logs.logtime}"
],
"ignore_missing": true
}
},{
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "iis_custom.advanced_logs.logtime",
"target_field": "@timestamp",
"formats": ["yyyy-MM-dd HH:mm:ss.SSS"]
}
}, {
"remove": {
"field": "iis_custom.advanced_logs.logtime"
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}

I will appreciate if you can help out again.

Thank you!

6

How did you upload the pipeline? Are you sure that the pipeline on your Ingest node is the one which has the fixed date parser?

Filebeat does not update pipelines by default. You need to pass a flag to it during setup or running it.
You could run ./filebeat setup --pipelines --modules="iis" --update-pipelines This forces Filebeat to update existing pipelines of IIS.

7

Hi Noemi

Thanks for the quick reply. If I stop and start the filebeat service after updating the file, will that also be OK, meaning, will it read in the new file?

8

No, it's not enough. You need to run the setup command.

9

Ahhhh. Thank you! I just learnt something new, again :slight_smile: OK, I'll do it tomorrow morning first thing then I'll send you an update.

10

Hi Noemi

I ran the command you gave but got an error that --pipelines and --update-pipelines commands do not exist.

However, running the below worked.

D:\filebeat>filebeat setup --modules=iis_custom
Loaded index template
Loaded dashboards
Loaded machine learning job configurations

Unfortunately I am still getting the same error.

I proceeded to make the log file even simpler by just leaving the date (without the time) to make it as simple as possible, but unfortunately, still the same error:
#Software: IIS Advanced Logging Module
#Version: 1.0
#Start-Date: 2018-09-11 05:42:42.924
#Fields: date-local
2018-09-11
2018-09-11
2018-09-11
2018-09-11
2018-09-11

Regards

Leigh

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.