# Filebeat Grok for Date IIS logs

**URL:** <https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 7, 2018, 5:59am UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659 "2018-09-07T05:59:57Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![necrolingus](https://avatars.discourse-cdn.com/v4/letter/n/91b2a8/32.png) [@necrolingus](https://discuss.elastic.co/u/necrolingus)\
**Post date:** [September 7, 2018, 5:59am UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/1 "2018-09-07T05:59:57Z")

</div>

Hi everyone

I am trying to import IIS Logs, for now literally one field which is the date and time. Here is what my log file looks like:  
_#Software: IIS Advanced Logging Module_  
_#Version: 1.0_  
_#Start-Date: 2018-09-07 05:33:33.297_  
_#Fields: date-local time-local_  
_2018-09-07 07:33:32.841_  
_2018-09-07 07:37:04.393_

As you can see the last 2 lines are the date and time. I tested this pattern:  
%{TIMESTAMP\_ISO8601:iis\_custom.advanced\_logs.logtime}  
online at [http://grokconstructor.appspot.com](http://grokconstructor.appspot.com) and it passed using the below date and time in the log:  
2018-09-07 07:33:32.841

But when I put it in filebeat I get the following error in Kibana (filebeat, started with the -e -d "\*" flags are not giving me any errors. Only in Kibana can I see this error:

Provided Grok expressions do not match field value: [2018-09-07 07:53:14.908]

Below is my full filebeat ingest.json. Any help will greatly be appreciated!

Thank you  
{  
"description": "Pipeline for parsing custom iis logs",  
"processors": [{  
"grok": {  
"field": "message",  
"patterns":[  
"%{TIMESTAMP\_ISO8601:iis\_custom.advanced\_logs.logtime}"  
],  
"ignore\_missing": true  
}  
},{  
"remove":{  
"field": "message"  
}  
}, {  
"rename": {  
"field": "@timestamp",  
"target\_field": "read\_timestamp"  
}  
}, {  
"date": {  
"field": "iis\_custom.advanced\_logs.logtime",  
"target\_field": "@timestamp",  
"formats": ["dd/MMM/YYYY:H:m:s Z"]  
}  
}, {  
"remove": {  
"field": "iis\_custom.advanced\_logs.logtime"  
}  
}],  
"on\_failure" : [{  
"set" : {  
"field" : "error.message",  
"value" : "{{ \_ingest.on\_failure\_message }}"  
}  
}]  
}

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 7, 2018, 12:01pm UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/2 "2018-09-07T12:01:41Z")

</div>

The format defined in the date parser does not match the timestamp you provided.

This is the correct format for your timestamp:

```auto
"date": {
    "field": "iis_custom.advanced_logs.logtime",
    "target_field": "@timestamp",
    "formats": ["yyyy-MM-dd HH:mm:ss.SSS"]
}

```

---

<div class="post-metadata">

**Author:** ![necrolingus](https://avatars.discourse-cdn.com/v4/letter/n/91b2a8/32.png) [@necrolingus](https://discuss.elastic.co/u/necrolingus)\
**Post date:** [September 7, 2018, 6:01pm UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/3 "2018-09-07T18:01:50Z")

</div>

Hi Noemi

Thank you so much for the quick reply. I will test it out Monday morning when I'm back at work!

Best regards

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 7, 2018, 6:39pm UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/4 "2018-09-07T18:39:42Z")

</div>

Have a nice weekend! 🙂

---

<div class="post-metadata">

**Author:** ![necrolingus](https://avatars.discourse-cdn.com/v4/letter/n/91b2a8/32.png) [@necrolingus](https://discuss.elastic.co/u/necrolingus)\
**Post date:** [September 10, 2018, 10:04am UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/5 "2018-09-10T10:04:23Z")

</div>

Hi Noemi

I did what you suggested, but unfortunately I am getting the same error:

 ![Kibana%20Error](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a75e4fd421de2f6ecec443937e9baee2906a630b.png)

Here is my full ingest.json file and the log file (I removed all other fields and kept only the date and time to make it simple for now). I understand that the lines starting with a # will give an error, but I am OK with that for now.

**IIS Logs:**  
#Software: IIS Advanced Logging Module  
#Version: 1.0  
#Start-Date: 2018-09-10 09:43:25.684  
#Fields: date-local time-local  
2018-09-10 11:43:25.243  
2018-09-10 11:43:26.834  
2018-09-10 11:43:26.850  
2018-09-10 11:43:26.912

I had a look at my custom Apache, PHP, and Python ingest.json files, and for some of them I specified the datetime part as DATA. I guess way back I must've had issues as well with them. Their data formats are all like this:

"date": {  
"field": "apache2\_php.access.time",  
"target\_field": "@timestamp",  
"formats": ["dd/MMM/YYYY:H:m:s Z"]  
}

Could it be that the @timestamp format in Kibana expects the date to be in format ["dd/MMM/YYYY:H:m:s Z"], but that my Grok data type is not correct? (PS, I also tried data type DATA; I see for one of my other logs it is DATA and it actually maps correctly the @timestamp field)

**My ingest.json file:**

"description": "Pipeline for parsing custom iis logs",  
"processors": [{  
"grok": {  
"field": "message",  
"patterns":[  
"%{TIMESTAMP\_ISO8601:iis\_custom.advanced\_logs.logtime}"  
],  
"ignore\_missing": true  
}  
},{  
"remove":{  
"field": "message"  
}  
}, {  
"rename": {  
"field": "@timestamp",  
"target\_field": "read\_timestamp"  
}  
}, {  
"date": {  
"field": "iis\_custom.advanced\_logs.logtime",  
"target\_field": "@timestamp",  
"formats": ["yyyy-MM-dd HH:mm:ss.SSS"]  
}  
}, {  
"remove": {  
"field": "iis\_custom.advanced\_logs.logtime"  
}  
}],  
"on\_failure" : [{  
"set" : {  
"field" : "error.message",  
"value" : "{{ \_ingest.on\_failure\_message }}"  
}  
}]  
}

I will appreciate if you can help out again.

Thank you!

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 10, 2018, 11:58am UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/6 "2018-09-10T11:58:43Z")

</div>

How did you upload the pipeline? Are you sure that the pipeline on your Ingest node is the one which has the fixed date parser?

Filebeat does not update pipelines by default. You need to pass a flag to it during setup or running it.  
You could run `./filebeat setup --pipelines --modules="iis" --update-pipelines` This forces Filebeat to update existing pipelines of IIS.

---

<div class="post-metadata">

**Author:** ![necrolingus](https://avatars.discourse-cdn.com/v4/letter/n/91b2a8/32.png) [@necrolingus](https://discuss.elastic.co/u/necrolingus)\
**Post date:** [September 10, 2018, 12:30pm UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/7 "2018-09-10T12:30:34Z")

</div>

Hi Noemi

Thanks for the quick reply. If I stop and start the filebeat service after updating the file, will that also be OK, meaning, will it read in the new file?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 10, 2018, 12:46pm UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/8 "2018-09-10T12:46:36Z")

</div>

No, it's not enough. You need to run the `setup` command.

---

<div class="post-metadata">

**Author:** ![necrolingus](https://avatars.discourse-cdn.com/v4/letter/n/91b2a8/32.png) [@necrolingus](https://discuss.elastic.co/u/necrolingus)\
**Post date:** [September 10, 2018, 1:12pm UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/9 "2018-09-10T13:12:40Z")

</div>

Ahhhh. Thank you! I just learnt something new, again 🙂 OK, I'll do it tomorrow morning first thing then I'll send you an update.

---

<div class="post-metadata">

**Author:** ![necrolingus](https://avatars.discourse-cdn.com/v4/letter/n/91b2a8/32.png) [@necrolingus](https://discuss.elastic.co/u/necrolingus)\
**Post date:** [September 11, 2018, 6:19am UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/10 "2018-09-11T06:19:09Z")

</div>

Hi Noemi

I ran the command you gave but got an error that --pipelines and --update-pipelines commands do not exist.

However, running the below worked.

D:\filebeat\> **filebeat setup --modules=iis\_custom**  
Loaded index template  
Loaded dashboards  
Loaded machine learning job configurations

Unfortunately I am still getting the same error.

I proceeded to make the log file even simpler by just leaving the date (without the time) to make it as simple as possible, but unfortunately, still the same error:  
#Software: IIS Advanced Logging Module  
#Version: 1.0  
#Start-Date: 2018-09-11 05:42:42.924  
#Fields: date-local  
2018-09-11  
2018-09-11  
2018-09-11  
2018-09-11  
2018-09-11

Regards

Leigh

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2018, 6:19am UTC](https://discuss.elastic.co/t/filebeat-grok-for-date-iis-logs/147659/11 "2018-10-09T06:19:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.