Source field parse not working

Elastic Stack Logstash
1

Input:

"source": "E:\\Program Files\\GlobalSCAPE\\EFT Enterprise\\Config\\Logs\\u_ex180419.log"

Filter:

if [source] =~ "^.*u_ex.*$"

This is not working. Can someone please help me here?

2

I want to pick up files like u_ex*.log and eft*.log (with case insensitive). Can someone please help?

3

Here is eft source

E:\Program Files\GlobalSCAPE\EFT Enterprise\Config\EFT.log

4

Could you post your full configuration? The regex looks right, so maybe there is another error.

5

Here it is:

Input:

input {
  beats {
    port => 5045
  }
}

Filter:

filter {
    mutate{
        lowercase => ["source"]
    }
    if [source] =~ "^.*eft.*$" {
        mutate { 
          replace => { type => "eftlog" }
          add_field => {"[@metadata][indexname]" => "eftlog" } 
        }
        if [message] =~ /^\s*$/ {
            drop { }
        }
        grok{
            match => {"message" => "%{DATESTAMP:date}\,%{NUMBER:msgnum} \[%{NOTSPACE:session}\] %{LOGLEVEL:src} %{GREEDYDATA:msgdata}"}
            #overwrite => ["message"]            
        }
        
        date {
        	match => [ "date", "MM-dd-yy HH:mm:ss"]
            locale => "en-US"
        	timezone => "America/New_York" 
            target => "@timestamp"                  	        	
      	}
              
        mutate{
            convert => {"msgnum" => "integer"}
            convert => {"session" => "integer"}
        }        
    } else if [source] =~ "^.*u_ex.*$" {
        mutate { 
          replace => { type => "ftplog" } 
          add_field => {"[@metadata][indexname]" => "ftplog" }
        }
        grok{
            match => {"message" => "%{TIMESTAMP_ISO8601:date} %{IP:c-ip} %{NOTSPACE:c-port} %{NOTSPACE:cs-username} \[%{NUMBER:session}\]%{NOTSPACE:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:sc-status} %{NOTSPACE:sc-bytes} %{NOTSPACE:cs-bytes} %{NOTSPACE:s-name} %{NOTSPACE:sc-port}"}                 
        }
        
        ruby {
            code => "
                hash = event.to_hash
                hash.each do |k,v|
                    if v== '-'
                        event.set(k,0)
                    end
                end            
            "
        }
        date {
        	match => [ "date", "yyyy-MM-dd HH:mm:ss"]
            locale => "en-US"
        	timezone => "America/New_York" 
            target => "@timestamp"                  	        	
      	}	    
        mutate{
            convert => {"session" => "integer"}
            convert => {"c-port" => "integer"}
            convert => {"sc-bytes" => "integer"}
            convert => {"cs-bytes" => "integer"}
            convert => {"sc-port" => "integer"}
            convert => {"sc-status" => "integer"}            
        }
    } else {
        mutate { replace => { type => "random_logs" } }
    } 
}

Output:

output {
  stdout {codec => rubydebug}
}
6

I want to include "*.log" so that my eft regex will not get confuse with u_ex's eft source. All data from u_ex are also going under eft index.

7

I still cannot see what might be wrong. Maybe you could post the full input and output?

8

I am looking for simple regex expression where if input files are like following and I want to have my if else statement based on filename. But I am not able to get regex working and all my output going to random_logs instead of eft or ftp logs.

Filenames like these:

/user/globalscape/eft/config/logs/eft_20180301.log
/user/globalscape/eft/config/logs/eft_20180302.log
/user/globalscape/eft/config/logs/u_extest_20180301.log
/user/globalscape/eft/config/logs/u_extest_20180302.log

So basically I want eft*.log files to go under eft index and u_extest*.log go under ftp index.

Thanks

9

I do know what you want, so please don't just ignore requests for further information. I wanted to see the output because I had hoped that this helps us find the error.

I can see one problem: Your 'u_ex' logs are in a folder with 'eft' in its name. So they would be considered 'eft' logs. You could change that by turning around the two conditions or you have to make the regular expressions more specific.

But I don't see a reason for them to go to 'random_logs'.

10

Yes you are spot on Jenni. Since we can't change file location, I had to make changes to regex to fix the issue.

Thanks again for your help Jenni.

Thanks

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.