# Logstash filter not working as expected **URL:** \ **Category:** Logstash\ **Created:** [May 10, 2017, 5:45am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190 "2017-05-10T05:45:16Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![raghuvarma](https://avatars.discourse-cdn.com/v4/letter/r/8baadc/32.png) [@raghuvarma](https://discuss.elastic.co/u/raghuvarma)\ **Post date:** [May 10, 2017, 5:45am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/1 "2017-05-10T05:45:16Z")
i am new to ELK and trying to learn it. I have given following filter file for logtash to get log details from the application server. these are the paths where i am taking the logs from /opt/tomcat7/apache-tomcat-7.0.77/logs/catalina.out & /var/log/syslog. In kibana i could see the timestamp as when the logstash is pulling the logs from those files. But i wanted to replace it with actual timestamp of the log. please suggest me the changes that need to be done for my filters filter { if [type] == "syslog" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" } # add\_field =\> ["received\_at", "%{@timestamp}"] # add\_field =\> ["received\_from", "%{host}"] } syslog\_pri { } date { match =\> ["timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] target =\> "@timestamp" } } if [type] == "apache\_access" { grok { match =\> { "message" =\> ["%{COMBINEDAPACHELOG}", "%{IPORHOST:clientip} %{NOTSPACE:ident} %{NOTSPACE:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))" %{NOTSPACE:response} (?:%{NOTSPACE:bytes})" ] } } date { match =\> ["timestamp" , "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] target =\> "@timestamp" } } if [type] == "apache\_error" { grok { match =\> { "message" =\> "[(?%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})] [%{WORD:module}:%{LOGLEVEL:loglevel}] [pid %{NUMBER:pid}:tid %{NUMBER:tid}]( (%{POSINT:proxy\_errorcode})%{DATA:proxy\_errormessage}:)?( [client %{IPORHOST:client}:%{POSINT:clientport}])? %{DATA:errorcode}: %{GREEDYDATA:message}" } } date { match =\> ["timestamp" , "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] target =\> "@timestamp" } } if [type] == "apache\_sslrequest" { grok { match =\> { "message" =\> "[%{HTTPDATE:timestamp}] %{IPORHOST:client} %{NOTSPACE:protocol} %{NOTSPACE:cipher} "(%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" } } date { match =\> ["timestamp" , "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] target =\> "@timestamp" } } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 10, 2017, 5:49am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/2 "2017-05-10T05:49:29Z")
The patterns in your date filters are wrong, except maybe in the syslog case. Inspect an example `timestamp` field of each log type and try to adjust the date filter so the pattern matches the data. Right now there are some very obvious mismatches. The Logstash documentation contains examples of both syslog and HTTP log parsing. ---
**Author:** ![raghuvarma](https://avatars.discourse-cdn.com/v4/letter/r/8baadc/32.png) [@raghuvarma](https://discuss.elastic.co/u/raghuvarma)\ **Post date:** [May 10, 2017, 6:26am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/3 "2017-05-10T06:26:18Z")
@magnusbaeck thanks for replying.. even in syslog case i see the same difference. Can you share me the link of the logstash docs. ![](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0e1cafd0c6bfb64ce11a4bcc5202617439971db5.JPG). ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 10, 2017, 6:39am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/4 "2017-05-10T06:39:33Z")
Please show the contents of the `timestamp` field. Please use copy/paste, no screenshots. Use the JSON tab in Kibana. See [https://www.elastic.co/guide/en/logstash/current/config-examples.html](https://www.elastic.co/guide/en/logstash/current/config-examples.html). ---
**Author:** ![raghuvarma](https://avatars.discourse-cdn.com/v4/letter/r/8baadc/32.png) [@raghuvarma](https://discuss.elastic.co/u/raghuvarma)\ **Post date:** [May 10, 2017, 8:43am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/5 "2017-05-10T08:43:44Z")
@magnusbaeck May 10th 2017, 10:26:01.167 ---- is the timestamp i am able to see in kibana May 10, 2017 4:56:00 AM org.apache.catalina.startup.Catalina start ---- here is the actual time when the log genrated along with msg ---
**Author:** ![raghuvarma](https://avatars.discourse-cdn.com/v4/letter/r/8baadc/32.png) [@raghuvarma](https://discuss.elastic.co/u/raghuvarma)\ **Post date:** [May 10, 2017, 8:45am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/6 "2017-05-10T08:45:16Z")
@magnusbaeck { "\_index": "filebeat-2017.05.10", "\_type": "apache\_sslrequest", "\_id": "AVvwt\_eVag6JaBoW8duC", "\_score": null, "\_source": { "message": "May 10, 2017 4:56:00 AM org.apache.catalina.startup.Catalina start", "@version": "1", "@timestamp": "2017-05-10T04:56:01.167Z", "source": "/opt/tomcat7/apache-tomcat-7.0.77/logs/catalina.out", "count": 1, "fields": null, "offset": 230233, "type": "apache\_sslrequest", "input\_type": "log", "beat": { "hostname": "deploymnetvm", "name": "deploymnetvm" }, "host": "deploymnetvm", "tags": [ "beats\_input\_codec\_plain\_applied", "\_grokparsefailure" ] }, "fields": { "@timestamp": [ 1494392161167 ] }, "highlight": { "message": [ "@kibana-highlighted-field@May@/kibana-highlighted-field@ 10, 2017 4:56:00 AM org.apache.catalina.startup.Catalina start" ] }, "sort": [ 1494392161167 ] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 10, 2017, 8:51am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/7 "2017-05-10T08:51:16Z")
Where's the `timestamp` field that you're trying to parse with your date filter? ---
**Author:** ![raghuvarma](https://avatars.discourse-cdn.com/v4/letter/r/8baadc/32.png) [@raghuvarma](https://discuss.elastic.co/u/raghuvarma)\ **Post date:** [May 10, 2017, 9:17am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/8 "2017-05-10T09:17:39Z")
@magnusbaeck I could find the above JSON from one of the log in kibana discover page. u mean from kibana logs in elk server ? I am very new to this. Please help me to understand from where i can share the required timestamp details to you. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 10, 2017, 1:38pm UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/9 "2017-05-10T13:38:31Z")
The idea is that you use the grok filter to extract pieces of a field into fields of their own. That part seems to be working. The resulting fields can then be processed by other filter, like the date filter. If you look at your configuration you'll note that you've configured your date filter(s) to parse a field named `timestamp`, but when looking at an actual event there is no `timestamp` field (or any other field that only contains the timestamp to parse). ---
**Author:** ![raghuvarma](https://avatars.discourse-cdn.com/v4/letter/r/8baadc/32.png) [@raghuvarma](https://discuss.elastic.co/u/raghuvarma)\ **Post date:** [May 11, 2017, 5:21am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/10 "2017-05-11T05:21:57Z")
@magnusbaeck Here is the actual sample log that's being generated in the syslog file. May 10 09:27:59 deploymnetvm ansible-copy: Invoked with src=/home/devopsvm/.ansible/tmp/ansible-tmp-1494408474.42-15265948874711/source directory\_mode=None force=True remote\_src=None unsafe\_writes=None selevel=None seuser=None setype=None group=None content=NOT\_LOGGING\_PARAMETER dest=/opt/tomcat7/apache-tomcat-7.0.77/webapps/ serole=None original\_basename=tudu-dwr.war delimiter=None mode=None regexp=None owner=None follow=False validate=None attributes=None backup=False If this is not the thing that you are looking for, please let me know where i can see and how i can make the changes if any for getting the actual log time in timestamp field of kibana web page. Should this timestamp field to be invoked anywhere to get the correct one of log even. I don't know where i am missing here ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 11, 2017, 5:25am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/11 "2017-05-11T05:25:34Z")
You already have a grok filter to parse syslog events, but it's extracting the timestamp to the `syslog_timestamp` field. As I said earlier that's not the field you've configured the date filter to parse. ---
**Author:** ![raghuvarma](https://avatars.discourse-cdn.com/v4/letter/r/8baadc/32.png) [@raghuvarma](https://discuss.elastic.co/u/raghuvarma)\ **Post date:** [May 11, 2017, 7:42am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/12 "2017-05-11T07:42:17Z")
@magnusbaeck got it. So in the filter also i an changing it to time stamp. Please help and make changes to this, where you feel the changes needed for getting the log timestamp. filter { if [type] == "syslog" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" } # add\_field =\> ["received\_at", "%{@timestamp}"] # add\_field =\> ["received\_from", "%{host}"] } syslog\_pri { } date { match =\> ["timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] target =\> "@timestamp" } } if [type] == "apache\_access" { grok { match =\> { "message" =\> ["%{COMBINEDAPACHELOG}", "%{IPORHOST:clientip} %{NOTSPACE:ident} %{NOTSPACE:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))" %{NOTSPACE:response} (?:%{NOTSPACE:bytes})" ] } } date { match =\> ["timestamp" , "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] target =\> "@timestamp" } } if [type] == "apache\_error" { grok { match =\> { "message" =\> "[(?%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})] [%{WORD:module}:%{LOGLEVEL:loglevel}] [pid %{NUMBER:pid}:tid %{NUMBER:tid}]( (%{POSINT:proxy\_errorcode})%{DATA:proxy\_errormessage}:)?( [client %{IPORHOST:client}:%{POSINT:clientport}])? %{DATA:errorcode}: %{GREEDYDATA:message}" } } date { match =\> ["timestamp" , "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] target =\> "@timestamp" } } if [type] == "apache\_sslrequest" { grok { match =\> { "message" =\> "[%{HTTPDATE:timestamp}] %{IPORHOST:client} %{NOTSPACE:protocol} %{NOTSPACE:cipher} "(%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" } } date { match =\> ["timestamp" , "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] target =\> "@timestamp" } } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 11, 2017, 9:13am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/13 "2017-05-11T09:13:32Z")
I don't have time to fix your configuration for you but I can provide pointers. For each message type, look at the event in Kibana and make sure that a) you are extracting a timestamp field and b) compare it to the pattern you have in the corresponding date filter. It should be quite obvious that they don't match. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 8, 2017, 9:22am UTC](https://discuss.elastic.co/t/logstash-filter-not-working-as-expected/85190/14 "2017-06-08T09:22:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.