Grok parser not working as expected with GREEDYDATA

sachinvy · March 28, 2018, 4:27am

Here is an example:

Data in file:

03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY_LOCAL_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0

Here is my grok parser:
filter {
if [source] =~ "eft*.log" {
if [message] =~ /^\s*$/ {
drop { }
}
grok{
match => {"message" => "%{DATESTAMP:date},%{NUMBER:msgnum} [%{NOTSPACE:session}] %{LOGLEVEL:src} %{GREEDYDATA:message}"}
}
}
}

Output:

{
"path" => "E:/elastic/logstash/incoming/eft_test.log",
"@version" => "1",
"host" => "SVYAHALKAR2012R2",
"message" => "03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY_LOCAL_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0\r",
"@timestamp" => 2018-03-28T00:06:15.545Z
}

Basically, it is not splitting data into different fields and showing up all in message field which is GREEDYDATA. Parsing works in Grok Debugger tool.

Can you please help?

Thanks,
Sachin

magnusbaeck · March 28, 2018, 6:23am

if [source] =~ "eft*.log" {

But your event doesn't have a source field so the grok filter never gets to process anything (however, there is a path field).

Also, the regexp is wrong. You'll probably want to use eft.*\.log$ instead.

sachinvy · March 28, 2018, 3:18pm

Thank you very much Magnus. That worked. Sorry I am newbie to ELK stack.

I am getting following output instead of GREEDYDATA in one message field.

{
"path" => "/Users/svyahalkar/elastic/logstash/incoming/eft_test.log",
"src" => "DEBUG",
"host" => "vlau6400.eagleinvsys.com",
"msgnum" => "563",
"@timestamp" => 2018-03-28T15:14:43.226Z,
"session" => "1452",
"date" => "03-27-18 09:32:10",
"@version" => "1",
"message" => [
[0] "03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY_LOCAL_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0\r",
[1] "AdvancedProperties - No value in: HKEY_LOCAL_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0\r"
]
}

sachinvy · March 28, 2018, 3:19pm

Only one change I made

filter {
if [path] =~ "eft" {
if [message] =~ /^\s*$/ {
drop { }
}
grok{
match => {"message" => "%{DATESTAMP:date},%{NUMBER:msgnum} [%{NOTSPACE:session}] %{LOGLEVEL:src} %{GREEDYDATA:message}"}
}
}
}

sachinvy · March 28, 2018, 9:11pm

No worries, I found the problem. I was giving same name to GREEDYDATA as message. I changed the name of field and it worked. Thanks again for your immense help.

I would like to know how I learn logstash programs, can you please point me to any resource where I can read more about it.

Thanks,
Sachin

Badger · March 28, 2018, 10:06pm

That's one option. If for some reason it was important to keep the name then you could have added this to the grok configuration

overwrite => [ "message" ]

sachinvy · March 28, 2018, 10:22pm

Thanks Badger. One last issue I have is with my filters. I have two log files eft_test.log and u_extest.log and I want to process these files differently under filter. So I have following filter but it picks up only eft_test.log file. I am newbie to ELK, can you please help me here? Also point me to resource/documentation where I can learn more about [path] [source] in logstash filters. I am not able to find any documents.

filter {
if [path] =~ "eft..log$" {
mutate { replace => { type => "eft_log" } }
if [message] =~ /^\s$/ {
drop { }
}
grok{
match => {"message" => "%{DATESTAMP:date},%{NUMBER:msgnum} [%{NOTSPACE:session}] %{LOGLEVEL:src} %{GREEDYDATA:msgdata}"}
}
} else if [path] =~ "u_ex.*.log$" {
mutate { replace => { type => "ftp_log" } }
grok{
match => {"message" => "%{TIMESTAMP_ISO8601:date} %{IP:c-ip} %{NOTSPACE:c-port} %{NOTSPACE:cs-username} [%{NUMBER:session}]%{NOTSPACE:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:sc-status} %{NOTSPACE:sc-bytes} %{NOTSPACE:cs-bytes} %{NOTSPACE:s-name} %{NOTSPACE:sc-port}"}
}
ruby {
code => "
hash = event.to_has
has.each do |k,v|
if v== '-'
event.remove(k)
end
end
"
}
} else {
mutate { replace => { type => "random_logs" } }
}
}

Thanks again for all your help.

Thanks,
Sachin

Badger · March 28, 2018, 10:46pm

You say you have a file called eft_test.log, and say it gets processed, but the test you have for that ([path] =~ "eft..log$") would not match it. There are other things in that config which would clearly cause errors if you ran it. Can you paste the actual configuration, preceded and followed by lines just containing three ` (back-tick). I think that should stop some of the mangling.

sachinvy · March 29, 2018, 6:20am

Here is my filter in config file.

 filter {
    if [path] =~ "eft.*\.log$" {
        mutate { replace => { type => "eft_log" } }
        if [message] =~ /^\s*$/ {
            drop { }
        }
        grok{
            match => {"message" => "%{DATESTAMP:date}\,%{NUMBER:msgnum} \[%{NOTSPACE:session}\] %{LOGLEVEL:src} %{GREEDYDATA:message}"}
            overwrite => ["message"]
        }        
    } else if [path] =~ "u_ex.*\.log$" {
        mutate { replace => { type => "ftp_log" } }
        grok{
            match => {"message" => "%{TIMESTAMP_ISO8601:date} %{IP:c-ip} %{NOTSPACE:c-port} %{NOTSPACE:cs-username} \[%{NUMBER:session}\]%{NOTSPACE:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:sc-status} %{NOTSPACE:sc-bytes} %{NOTSPACE:cs-bytes} %{NOTSPACE:s-name} %{NOTSPACE:sc-port}"}                 
        }
        ruby {
            code => "
                hash = event.to_has
                has.each do |k,v|
                    if v== '-'
                        event.remove(k)
                    end
                end            
            "
        }
    } else {
        mutate { replace => { type => "random_logs" } }
    } 
}

Here are sample lines from eft_test.log

03-27-18 09:32:10,563 [1452] DEBUG AdvancedProperties - No value in: HKEY_LOCAL_MACHINE\SOFTWARE\GlobalSCAPE Inc.\EFT Server 7.0\ReplaceBackslashWithSlashInPathsForFTP, using default: 0

Thanks Badger for quick reply.

Thanks,
Sachin

sachinvy · March 29, 2018, 6:38am

Now I am getting following exception for eft log file.

[2018-03-28T23:36:44,146][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `to_has' for #<LogStash::Event:0xac98283>
Did you mean?  to_hash
               to_s

sachinvy · March 29, 2018, 6:35pm

What am I doing wrong here?

Input files eft_test.log and u_extest.log

eft_test.log

03-27-18 09:32:10,590 [2184] TRACE SFTP <> - [139C9318] msg: 2024502473 Sending SSH_MSG_KEX_31 (139 bytes, seq nr 1)

03-27-18 09:32:10,591 [2184] TRACE SFTP <> - [139C9318] msg: 2024502476 Received SSH_MSG_KEX_32 (134 bytes, seq nr 2)

03-27-18 09:32:10,592 [1452] TRACE Events.Server.Pittsburgh_Non-Prod <Parse FTP Command> - Dispatch file system event; event type: 20491; PPath: \\Pnpfs01\np-ftp\FTPData\Usr\trpd001\CMW\control\; VPath: /Usr/trpd001/CMW/control/

u_extest.log

2018-03-25 04:00:00 xx.xx.xxx.xxx - - [179863]user DISTST4FTP1_XD3W - 331 - - - 21 
2018-03-25 04:00:00 xx.xx.xxx.xxx - - [179863]pass ****** - 530 - - - 21 
2018-03-25 04:00:00 xx.xx.xxx.xxx - - [179864]user TRPD003FTP1_E4PK - 331 - - - 21 
2018-03-25 04:00:00 xx.xx.xxx.xxx- EADMZ\TRPD003FTP1_E4PK [179864]pass ****** - 230 - - - 21

Filter:

input {
  #stdin {}
  file {
    path => ["/Users/svyahalkar/elastic/logstash/incoming/eft*.log","/Users/svyahalkar/elastic/logstash/incoming/u_ex*.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
    if [path] =~ "eft" {
        mutate { replace => { type => "eft_log" } }
        if [message] =~ /^\s*$/ {
            drop { }
        }
        grok{
            match => {"message" => "%{DATESTAMP:date}\,%{NUMBER:msgnum} \[%{NOTSPACE:session}\] %{LOGLEVEL:src} %{GREEDYDATA:msgdata}"}
            #overwrite => ["message"]
        }        
    } else if [path] =~ "u_ex" {
        mutate { replace => { type => "ftp_log" } }
        grok{
            match => {"message" => "%{TIMESTAMP_ISO8601:date} %{IP:c-ip} %{NOTSPACE:c-port} %{NOTSPACE:cs-username} \[%{NUMBER:session}\]%{NOTSPACE:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:sc-status} %{NOTSPACE:sc-bytes} %{NOTSPACE:cs-bytes} %{NOTSPACE:s-name} %{NOTSPACE:sc-port}"}                 
        }
        ruby {
            code => "
                hash = event.to_has
                has.each do |k,v|
                    if v== '-'
                        event.remove(k)
                    end
                end            
            "
        }
    } else {
        mutate { replace => { type => "random_logs" } }
    } 
}

I am getting following error

[2018-03-29T11:27:30,375][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `to_has' for #<LogStash::Event:0x5c4d547>
Did you mean?  to_hash
               to_s
[2018-03-29T11:27:30,396][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `to_has' for #<LogStash::Event:0x666aa4f9>
Did you mean?  to_hash
               to_s
{
          "type" => "eft_log",
           "src" => "TRACE",
          "path" => "/Users/svyahalkar/elastic/logstash/incoming/eft_test.log",
      "@version" => "1",
          "date" => "03-27-18 09:32:10",
       "msgdata" => "SFTP <> - [139C9318] msg: 2024502476 Received SSH_MSG_KEX_32 (134 bytes, seq nr 2)\r",
        "msgnum" => "591",
       "message" => "03-27-18 09:32:10,591 [2184] TRACE SFTP <> - [139C9318] msg: 2024502476 Received SSH_MSG_KEX_32 (134 bytes, seq nr 2)\r",
    "@timestamp" => 2018-03-29T18:27:29.983Z,
          "host" => "svyahalkar-mbp.local",
       "session" => "2184"
}

Badger · March 29, 2018, 7:12pm

That should be to_hash and hash.each

sachinvy · March 29, 2018, 7:31pm

Sorry to bother you for such a trivial issue. I feel embarrassed.

system · April 26, 2018, 7:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.