# Grok filter is not working properly **URL:** \ **Category:** Logstash\ **Created:** [April 21, 2020, 2:52pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073 "2020-04-21T14:52:12Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Ankit-github-26](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankit-github-26/32/66756_2.png) [@Ankit-github-26](https://discuss.elastic.co/u/Ankit-github-26)\ **Post date:** [April 21, 2020, 2:52pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/1 "2020-04-21T14:52:12Z")
Hello Guys, I have Filebeat-7.1 installed in a Debian server, this Filebeat send data from files in this Debian server to server with Logstash 7.6 , here are the files config Filebeat.yml: #=========================== Filebeat inputs ============================= filebeat.inputs: - type: log - type: log enabled: true paths: - /usr/local/freeswitch/log/freeswitch.log force\_close\_files: true fields: env: dev type: freeswitch.log processors: - drop\_fields: fields: ["agent.ephemeral\_id", "time", "agent.hostname", "agent.id", "agent.type", "agent.version", "ecs.version", "input.type", "log.offset", "@version", "fields.env", "tags"] #----------------------------- Logstash output -------------------------------- output.logstash: hosts: ["35.171.202.75:5044"] --------------------------------logstash.conf----------------------------------------------------------------------------- input.conf input { beats { port =\> 5044 } } filter.conf filter{ if [fields][env] == "dev" { if [source] == "/root/code/cigol/logs/server.log" { json { source =\> "message" } } } else if [source] == "/usr/local/freeswitch/log/freeswitch.log" { grok { match =\> { "message" =\> "%{NOTSPACE:uuid} %{TIMESTAMP\_ISO8601:date} [%{LOGLEVEL:loglevel}] %{GREEDYDATA:message}" } remove\_field =\> ["message"] } } } Output.conf output { elasticsearch { hosts =\> ["127.0.0.1:9200"] index =\> "%{[fields][type]}-%{+YYYY.MM.dd}" } stdout { codec =\> rubydebug } } application logs format 79110982-6d35-4b80-9be7-6ec9772313f9 2020-04-21 14:25:55.001130 [DEBUG] switch\_core\_state\_machine.c:749 (sofia/3clogic\_external/3001@freeswitch-registrar-10x.i3clogic.com:5505) State DESTROY Kibana Output message 79110982-6d35-4b80-9be7-6ec9772313f9 2020-04-21 14:25:55.001130 [DEBUG] mod\_sofia.c:364 sofia/3clogic\_external/3001@freeswitch-registrar-10x.i3clogic.com:5505 SOFIA DESTROY I want to segregate message as below "UUID" = 79110982-6d35-4b80-9be7-6ec9772313f9 "date" = 2020-04-21 14:25:55.001130 "loglevel" = DEBUG "message" = switch\_core\_state\_machine.c:749 (sofia/3clogic\_external/3001@freeswitch-registrar-10x.i3clogic.com:5505) State DESTROY Please help me on this ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 21, 2020, 3:55pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/2 "2020-04-21T15:55:42Z")
I do not believe that the timestamp format you have is ISO8601. See [this](https://discuss.elastic.co/t/help-needed-in-grok/213827/2) for advice on how to develop a grok pattern. ---
**Author:** ![Ankit-github-26](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankit-github-26/32/66756_2.png) [@Ankit-github-26](https://discuss.elastic.co/u/Ankit-github-26)\ **Post date:** [April 21, 2020, 4:14pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/3 "2020-04-21T16:14:10Z")
Still getting the same output in kibana filter{ if [fields][env] == "dev" { if [source] == "/root/code/cigol/logs/server.log" { json { source =\> "message" } } } else if [source] == "/usr/local/freeswitch/log/freeswitch.log" { grok { pattern\_definitions =\> { "MYDATETIME" =\> "%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}" } match =\> { "message" =\> "%{NOTSPACE:uuid} ^%{MYDATETIME:time} [%{LOGLEVEL:loglevel}] %{GREEDYDATA:msg}" } remove\_field =\> ["message"] } } } I am new in logstash, please help me ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 21, 2020, 6:39pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/4 "2020-04-21T18:39:32Z")
The ^ anchors the pattern to the start of the line. You need to remove it, or move it to before %{NOTSPACE:uuid} ---
**Author:** ![Ankit-github-26](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankit-github-26/32/66756_2.png) [@Ankit-github-26](https://discuss.elastic.co/u/Ankit-github-26)\ **Post date:** [April 21, 2020, 6:57pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/5 "2020-04-21T18:57:25Z")
still getting the same output message 043e5c71-30a7-484c-9616-1f0dd3712875 2020-04-21 18:50:48.541132 [DEBUG] switch\_core\_session.c:1726 Session 146 (sofia/3clogic\_external/3001@freeswitch-registrar-10x.i3clogic.com:5505) Locked, Waiting on external entities filter{ if [fields][env] == "dev" { if [source] == "/root/code/cigol/logs/server.log" { json { source =\> "message" } } } else if [source] == "/usr/local/freeswitch/log/freeswitch.log" { grok { pattern\_definitions =\> { "MYDATETIME" =\> "%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}" } match =\> { "message" =\> "^%{NOTSPACE:uuid} %{MYDATETIME:time} [%{LOGLEVEL:loglevel}] %{GREEDYDATA:msg}" } remove\_field =\> ["message"] } } } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 21, 2020, 9:26pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/6 "2020-04-21T21:26:09Z")
I suggest you follow that link I posted above and build your pattern in the way it describes. ---
**Author:** ![Ankit-github-26](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankit-github-26/32/66756_2.png) [@Ankit-github-26](https://discuss.elastic.co/u/Ankit-github-26)\ **Post date:** [April 21, 2020, 9:33pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/7 "2020-04-21T21:33:49Z")
Can you please elaborate what exactly i am doing wrong. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 21, 2020, 9:35pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/8 "2020-04-21T21:35:18Z")
You are trying to write a pattern that matches the entire line. That is not the best approach. A better approach is described in the post I linked to. ---
**Author:** ![Ankit-github-26](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankit-github-26/32/66756_2.png) [@Ankit-github-26](https://discuss.elastic.co/u/Ankit-github-26)\ **Post date:** [April 21, 2020, 9:37pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/9 "2020-04-21T21:37:06Z")
I also tried match =\> { "message" =\> "%{NOTSPACE:uuid}" } but no luck ---
**Author:** ![Ankit-github-26](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankit-github-26/32/66756_2.png) [@Ankit-github-26](https://discuss.elastic.co/u/Ankit-github-26)\ **Post date:** [April 21, 2020, 9:43pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/10 "2020-04-21T21:43:39Z")
Below mentioned Grok syntax is parsing properly in [https://grokdebug.herokuapp.com](https://grokdebug.herokuapp.com) but unfortunately not reflecting same result in Kibana %{NOTSPACE:uuid} %{TIMESTAMP\_ISO8601:date} [%{LOGLEVEL:loglevel}] %{GREEDYDATA:message} ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 19, 2020, 9:43pm UTC](https://discuss.elastic.co/t/grok-filter-is-not-working-properly/229073/11 "2020-05-19T21:43:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.