# Need help with the grok and the date filter for parsing logs **URL:** \ **Category:** Logstash\ **Created:** [May 20, 2017, 11:57am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523 "2017-05-20T11:57:38Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [May 20, 2017, 11:57am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/1 "2017-05-20T11:57:39Z")
I am trying to parse logs from a CSV file . Issue 1\> I am using below mentioned conf file. input { file { path =\> "/tmp/test1.csv" type =\> "core2" start\_position =\> "beginning" } } filter { csv { separator =\> ";" columns =\> ["Time","Source","Status","Severity","Location","ConfigItem","Alert","Message1","Message2"] } } output { elasticsearch { action =\> "index" hosts =\> "localhost" index =\> "stock" workers =\> 1 } stdout {} } Let me know , is this the correct one ? If yes, then how will I get the date option as I tried using the date filter but unable to get through (Please find the sample data): Time=Monday, May 01, 2017 1:48 AM Format for this kind of date ....! Thanks in advance ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 24, 2017, 9:29am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/2 "2017-05-24T09:29:42Z")
Use a date filter to parse the timestamp. The pattern "EEE, MMM dd, YYYY hh:mm aa" probably works. ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [May 24, 2017, 12:08pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/3 "2017-05-24T12:08:06Z")
Hi Magnus, Thanks for replying ! I am able to parse the logs with the help of grok filter : but I am not getting the correct implementation of logs in KIBANA For ex. : When I am parsing "NODEDOWN ALERT" , then while visualization it is showing one bar for "NODEDOWN" and another bar for "ALERT". how I can sort this out ? Regards, Gaurav Singh ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 24, 2017, 8:06pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/4 "2017-05-24T20:06:50Z")
This is because the field in Elasticsearch is analyzed. You need to adjust the index template used so that the field or question (or maybe all string fields?) are non-analyzed string fields (aka keyword fields as of ES 5.0). ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [May 25, 2017, 3:21pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/5 "2017-05-25T15:21:39Z")
I got some thing "Indexes imported from 2.x do not support keyword. Instead they will attempt to downgrade keyword into string. " . Seems like I need to move on to elasticsearch 5.x. ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [May 31, 2017, 12:44pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/6 "2017-05-31T12:44:23Z")
Is there any option or choice we can change the string to keyword as i don't want to move for elasticsearch 5.x ? (Currently using 2.x) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 31, 2017, 1:47pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/7 "2017-05-31T13:47:19Z")
Make sure the string field is set as not\_analyzed. That's equivalent to 5.x's keyword type. ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [May 31, 2017, 2:08pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/8 "2017-05-31T14:08:30Z")
And how can I do that ... I tried to find but got no luck . Any specific option I need to look for this ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 1, 2017, 5:16am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/9 "2017-06-01T05:16:20Z")
See [https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html#\_index\_2](https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html#_index_2) for an example of how to set a string field as not\_analyzed. ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [June 1, 2017, 10:13am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/10 "2017-06-01T10:13:01Z")
I am unable to make grok filter for the below mentioned data as the data values are not consistent Time=Monday, May 01, 2017 3:12 AM;Source=NPM;Status=New;Severity=Critical;Location=SYD-TO;[ConfigItem=elgar-mt.iii.com](http://ConfigItem=elgar-mt.iii.com) aws;Alert=Reboot;Message1=LastBootTimeChanged Time=Monday, May 01, 2017 3:12 AM;Source=NPM;Status=New;Severity=Critical;Location=SYD-TO;[ConfigItem=asia-spacewalk.iii.com](http://ConfigItem=asia-spacewalk.iii.com);Alert=Reboot;Message1=LastBootTimeChanged Time=Monday, May 01, 2017 3:12 AM;Source=NPM;Status=New;Severity=Critical;Location=SIN-TO;[ConfigItem=sinsierra-mt.iii.com](http://ConfigItem=sinsierra-mt.iii.com);Alert=Reboot;Message1=LastBootTimeChanged Time=Monday, May 01, 2017 3:14 AM;Source=NPM;Status=New;Severity=Warning;Location=JEM-PD;ConfigItem=bamboo;Alert=Diskspace;Message1=/;Message2=90 % Time=Monday, May 01, 2017 4:09 AM;Source=NPM;Status=New;Severity=Critical;Location=SYRDC-POL;ConfigItem=scottsdaledc1;Alert=Reboot;Message1=LastBootTimeChanged Time=Monday, May 01, 2017 4:13 AM;Source=NPM;Status=New;Severity=Critical;Location=SYRDC-POL;ConfigItem=sbcldc1;Alert=Reboot;Message1=LastBootTimeChanged Time=Monday, May 01, 2017 4:13 AM;Source=NPM;Status=New;Severity=Critical;Location=SYRDC-POL;ConfigItem=chandlerdc1;Alert=Reboot;Message1=LastBootTimeChanged Time=Monday, May 01, 2017 4:19 AM;Source=NPM;Status=New;Severity=Critical;Location=SYRDC-POL;ConfigItem=tempedc1;Alert=Reboot;Message1=LastBootTimeChanged Time=Monday, May 01, 2017 4:22 AM;Source=NPM;Status=New;Severity=Critical;Location=JEM-PD;ConfigItem=bamboo;Alert=Diskspace;Message1=/;Message2=100 % As you can see message 1 and message 2 values are not consistent ? I am using the mentioned grok filter : Time=%{GREEDYDATA:time};Source=%{DATA:source};Status=%{DATA:status};Severity=%{DATA:severity};Location=%{DATA:location};ConfigItem=%{DATA:configitem};Alert=%{DATA:alert};Message1=%{DATA:message1}(;)?{Message2=%{INT:message2}}? Kindly Suggest. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2017, 10:17am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/11 "2017-06-01T10:17:56Z")
For this type of data, consider using the [kv filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html) with `;` as `field_split` instead of the grok filter. Grok is a very powerful filter, but not necessarily the ideal tool for all types of data. ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [June 1, 2017, 10:37am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/12 "2017-06-01T10:37:26Z")
Is it correct , kindly suggest if it will work or not ! kv { field\_split =\> ";" value\_split =\> "=" } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2017, 10:41am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/13 "2017-06-01T10:41:46Z")
Looks OK to me, but try it to find out for sure. ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [June 1, 2017, 11:53am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/14 "2017-06-01T11:53:52Z")
Able to parse the logs , thanks ! Now, could you please let me know to add the correct timestamp as it is taking the default timestamp not the time stamp from logs . For that do I need to use date filter and if yes then how ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 1, 2017, 1:03pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/15 "2017-06-01T13:03:21Z")
Yes, use the date filter. Please read the documentation and ask if you have any specific problems. ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [June 1, 2017, 3:02pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/16 "2017-06-01T15:02:46Z")
I am using the below filter but still the timestamp issue is same . kv { field\_split =\> ";" value\_split =\> "=" } date { match =\> ["Time" , "EEEE, MMMM dd, yyyy HH:mm a", "EEEE, MMMM dd,yyyy H:mm a"] target =\> "@timestamp" } As per the data shared above .. is it the right approach ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 1, 2017, 8:08pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/17 "2017-06-01T20:08:59Z")
Please show an example event as produced by Logstash. Copy/paste from the JSON tab of Kibana's Discover panel or use a `stdout { codec => rubydebug }` output. Observations: - There's a space missing in your last pattern ("dd,yyyy"). - Don't use H for 12-hour times. Use h instead. See [http://www.joda.org/joda-time/key\_format.html](http://www.joda.org/joda-time/key_format.html). ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [June 2, 2017, 8:27am UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/18 "2017-06-02T08:27:29Z")
Thank you very much ..... everything is perfect now ! I really appreciate the help! ---
**Author:** ![Gaurav\_Singh1](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@Gaurav\_Singh1](https://discuss.elastic.co/u/Gaurav_Singh1)\ **Post date:** [June 2, 2017, 12:26pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/19 "2017-06-02T12:26:57Z")
Well now getting the timezone difference.... though the time is correct . During the elasticsearch testing command : curl -X GET [http://localhost:9200/index/\_search?pretty](http://localhost:9200/index/_search?pretty) It is showing the correct time stamp as what in logs but while visualizing the data in Kibana the timezone is getting changed . Any setting I need to work on ? (By the way , i am working all this with docker just fyi) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 2, 2017, 12:52pm UTC](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523/20 "2017-06-02T12:52:51Z")
ES stores timestamps in UTC and Kibana adjusts them for the browser's timezone. What's the timezone of the timestamps in the logs? Please give an example input string and how it's stored in ES (use the JSON tab in Kibana's Discover panel so we can see the raw JSON document). [Next page](https://discuss.elastic.co/t/need-help-with-the-grok-and-the-date-filter-for-parsing-logs/86523.md?page=2)