# Grok parser not working as expected with GREEDYDATA **URL:** \ **Category:** Logstash\ **Created:** [March 28, 2018, 4:27am UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849 "2018-03-28T04:27:19Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![sachinvy](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Post date:** [March 28, 2018, 4:27am UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/1 "2018-03-28T04:27:19Z")
Here is an example: Data in file: 03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY\_LOCAL\_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0 03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY\_LOCAL\_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0 03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY\_LOCAL\_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0 Here is my grok parser: filter { if [source] =~ "eft\*.log" { if [message] =~ /^\s\*$/ { drop { } } grok{ match =\> {"message" =\> "%{DATESTAMP:date},%{NUMBER:msgnum} [%{NOTSPACE:session}] %{LOGLEVEL:src} %{GREEDYDATA:message}"} } } } Output: { "path" =\> "E:/elastic/logstash/incoming/eft\_test.log", "@version" =\> "1", "host" =\> "SVYAHALKAR2012R2", "message" =\> "03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY\_LOCAL\_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0\r", "@timestamp" =\> 2018-03-28T00:06:15.545Z } Basically, it is not splitting data into different fields and showing up all in message field which is GREEDYDATA. Parsing works in Grok Debugger tool. Can you please help? Thanks, Sachin ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 28, 2018, 6:23am UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/2 "2018-03-28T06:23:12Z")
> if [source] =~ "eft\*.log" { But your event doesn't have a `source` field so the grok filter never gets to process anything (however, there is a `path` field). Also, the regexp is wrong. You'll probably want to use `eft.*\.log$` instead. ---
**Author:** ![sachinvy](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Post date:** [March 28, 2018, 3:18pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/3 "2018-03-28T15:18:09Z")
Thank you very much Magnus. That worked. Sorry I am newbie to ELK stack. I am getting following output instead of GREEDYDATA in one message field. { "path" =\> "/Users/svyahalkar/elastic/logstash/incoming/eft\_test.log", "src" =\> "DEBUG", "host" =\> "[vlau6400.eagleinvsys.com](http://vlau6400.eagleinvsys.com)", "msgnum" =\> "563", "@timestamp" =\> 2018-03-28T15:14:43.226Z, "session" =\> "1452", "date" =\> "03-27-18 09:32:10", "@version" =\> "1", "message" =\> [ [0] "03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY\_LOCAL\_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0\r", [1] "AdvancedProperties - No value in: HKEY\_LOCAL\_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0\r" ] } ---
**Author:** ![sachinvy](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Post date:** [March 28, 2018, 3:19pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/4 "2018-03-28T15:19:04Z")
Only one change I made filter { if [path] =~ "eft" { if [message] =~ /^\s\*$/ { drop { } } grok{ match =\> {"message" =\> "%{DATESTAMP:date},%{NUMBER:msgnum} [%{NOTSPACE:session}] %{LOGLEVEL:src} %{GREEDYDATA:message}"} } } } ---
**Author:** ![sachinvy](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Post date:** [March 28, 2018, 9:11pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/5 "2018-03-28T21:11:19Z")
No worries, I found the problem. I was giving same name to GREEDYDATA as message. I changed the name of field and it worked. Thanks again for your immense help. I would like to know how I learn logstash programs, can you please point me to any resource where I can read more about it. Thanks, Sachin ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 28, 2018, 10:06pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/6 "2018-03-28T22:06:31Z")
> [@sachinvy](#): > > I was giving same name to GREEDYDATA as message. I changed the name of field and it worked. That's one option. If for some reason it was important to keep the name then you could have added this to the grok configuration ```auto overwrite => ["message"] ``` ---
**Author:** ![sachinvy](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Post date:** [March 28, 2018, 10:22pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/7 "2018-03-28T22:22:58Z")
Thanks Badger. One last issue I have is with my filters. I have two log files eft\_test.log and u\_extest.log and I want to process these files differently under filter. So I have following filter but it picks up only eft\_test.log file. I am newbie to ELK, can you please help me here? Also point me to resource/documentation where I can learn more about [path] [source] in logstash filters. I am not able to find any documents. filter { if [path] =~ "eft._.log$" { mutate { replace =\> { type =\> "eft\_log" } } if [message] =~ /^\s_$/ { drop { } } grok{ match =\> {"message" =\> "%{DATESTAMP:date},%{NUMBER:msgnum} [%{NOTSPACE:session}] %{LOGLEVEL:src} %{GREEDYDATA:msgdata}"} } } else if [path] =~ "u\_ex.\*.log$" { mutate { replace =\> { type =\> "ftp\_log" } } grok{ match =\> {"message" =\> "%{TIMESTAMP\_ISO8601:date} %{IP:c-ip} %{NOTSPACE:c-port} %{NOTSPACE:cs-username} [%{NUMBER:session}]%{NOTSPACE:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:sc-status} %{NOTSPACE:sc-bytes} %{NOTSPACE:cs-bytes} %{NOTSPACE:s-name} %{NOTSPACE:sc-port}"} } ruby { code =\> " hash = event.to\_has has.each do |k,v| if v== '-' event.remove(k) end end " } } else { mutate { replace =\> { type =\> "random\_logs" } } } } Thanks again for all your help. Thanks, Sachin ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 28, 2018, 10:46pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/8 "2018-03-28T22:46:18Z")
You say you have a file called eft\_test.log, and say it gets processed, but the test you have for that ([path] =~ "eft..log$") would not match it. There are other things in that config which would clearly cause errors if you ran it. Can you paste the actual configuration, preceded and followed by lines just containing three ` (back-tick). I think that should stop some of the mangling. ---
**Author:** ![sachinvy](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Post date:** [March 29, 2018, 6:20am UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/9 "2018-03-29T06:20:22Z")
Here is my filter in config file. ``` filter { if [path] =~ "eft.*\.log$" { mutate { replace => { type => "eft_log" } } if [message] =~ /^\s*$/ { drop { } } grok{ match => {"message" => "%{DATESTAMP:date}\,%{NUMBER:msgnum} \[%{NOTSPACE:session}\] %{LOGLEVEL:src} %{GREEDYDATA:message}"} overwrite => ["message"] } } else if [path] =~ "u_ex.*\.log$" { mutate { replace => { type => "ftp_log" } } grok{ match => {"message" => "%{TIMESTAMP_ISO8601:date} %{IP:c-ip} %{NOTSPACE:c-port} %{NOTSPACE:cs-username} \[%{NUMBER:session}\]%{NOTSPACE:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:sc-status} %{NOTSPACE:sc-bytes} %{NOTSPACE:cs-bytes} %{NOTSPACE:s-name} %{NOTSPACE:sc-port}"} } ruby { code => " hash = event.to_has has.each do |k,v| if v== '-' event.remove(k) end end " } } else { mutate { replace => { type => "random_logs" } } } } ``` Here are sample lines from eft\_test.log 03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY\_LOCAL\_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0 03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY\_LOCAL\_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0 Thanks Badger for quick reply. Thanks, Sachin ---
**Author:** ![sachinvy](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Post date:** [March 29, 2018, 6:38am UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/10 "2018-03-29T06:38:29Z")
Now I am getting following exception for eft log file. ``` [2018-03-28T23:36:44,146][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `to_has' for # Did you mean? to_hash to_s ``` ---
**Author:** ![sachinvy](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Post date:** [March 29, 2018, 6:35pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/11 "2018-03-29T18:35:12Z")
What am I doing wrong here? Input files eft\_test.log and u\_extest.log eft\_test.log ``` 03-27-18 09:32:10,590 [2184] TRACE SFTP <> - [139C9318] msg: 2024502473 Sending SSH_MSG_KEX_31 (139 bytes, seq nr 1) 03-27-18 09:32:10,591 [2184] TRACE SFTP <> - [139C9318] msg: 2024502476 Received SSH_MSG_KEX_32 (134 bytes, seq nr 2) 03-27-18 09:32:10,592 [1452] TRACE Events.Server.Pittsburgh_Non-Prod - Dispatch file system event; event type: 20491; PPath: \\Pnpfs01\np-ftp\FTPData\Usr\trpd001\CMW\control\; VPath: /Usr/trpd001/CMW/control/ ``` u\_extest.log ``` 2018-03-25 04:00:00 xx.xx.xxx.xxx - - [179863]user DISTST4FTP1_XD3W - 331 - - - 21 2018-03-25 04:00:00 xx.xx.xxx.xxx - - [179863]pass ****** - 530 - - - 21 2018-03-25 04:00:00 xx.xx.xxx.xxx - - [179864]user TRPD003FTP1_E4PK - 331 - - - 21 2018-03-25 04:00:00 xx.xx.xxx.xxx- EADMZ\TRPD003FTP1_E4PK [179864]pass ****** - 230 - - - 21 ``` Filter: ``` input { #stdin {} file { path => ["/Users/svyahalkar/elastic/logstash/incoming/eft*.log","/Users/svyahalkar/elastic/logstash/incoming/u_ex*.log"] start_position => "beginning" sincedb_path => "/dev/null" } } filter { if [path] =~ "eft" { mutate { replace => { type => "eft_log" } } if [message] =~ /^\s*$/ { drop { } } grok{ match => {"message" => "%{DATESTAMP:date}\,%{NUMBER:msgnum} \[%{NOTSPACE:session}\] %{LOGLEVEL:src} %{GREEDYDATA:msgdata}"} #overwrite => ["message"] } } else if [path] =~ "u_ex" { mutate { replace => { type => "ftp_log" } } grok{ match => {"message" => "%{TIMESTAMP_ISO8601:date} %{IP:c-ip} %{NOTSPACE:c-port} %{NOTSPACE:cs-username} \[%{NUMBER:session}\]%{NOTSPACE:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:sc-status} %{NOTSPACE:sc-bytes} %{NOTSPACE:cs-bytes} %{NOTSPACE:s-name} %{NOTSPACE:sc-port}"} } ruby { code => " hash = event.to_has has.each do |k,v| if v== '-' event.remove(k) end end " } } else { mutate { replace => { type => "random_logs" } } } } ``` I am getting following error ``` [2018-03-29T11:27:30,375][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `to_has' for # Did you mean? to_hash to_s [2018-03-29T11:27:30,396][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `to_has' for # Did you mean? to_hash to_s { "type" => "eft_log", "src" => "TRACE", "path" => "/Users/svyahalkar/elastic/logstash/incoming/eft_test.log", "@version" => "1", "date" => "03-27-18 09:32:10", "msgdata" => "SFTP <> - [139C9318] msg: 2024502476 Received SSH_MSG_KEX_32 (134 bytes, seq nr 2)\r", "msgnum" => "591", "message" => "03-27-18 09:32:10,591 [2184] TRACE SFTP <> - [139C9318] msg: 2024502476 Received SSH_MSG_KEX_32 (134 bytes, seq nr 2)\r", "@timestamp" => 2018-03-29T18:27:29.983Z, "host" => "svyahalkar-mbp.local", "session" => "2184" } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 29, 2018, 7:12pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/12 "2018-03-29T19:12:27Z")
> [@sachinvy](#): > > hash = event.to\_has > has.each do |k,v| That should be to\_hash and hash.each ---
**Author:** ![sachinvy](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Post date:** [March 29, 2018, 7:31pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/13 "2018-03-29T19:31:20Z")
Sorry to bother you for such a trivial issue. I feel embarrassed. 😔 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 26, 2018, 7:31pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849/14 "2018-04-26T19:31:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.