Ruby exception occurred: undefined method `split' for nil:NilClass

Elastic Stack Logstash
1

This is my config for logstash (filter file )

filter {
if "ETL_log_enriched" in [tags] {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:log_timestamp} (%{LOGLEVEL:loglevel}|%{WORD:loglevel}) (?[^]]+)- %{GREEDYDATA:msgbody}"
}
}
mutate
{
add_field => ["event_read_time", "%{@timestamp}"]
}
date {
match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss"]
}
grok {
match => {
"msgbody" => "(?<log_category>\w+(?=-->))"
}
}
grok{
match => {
"msgbody" => "(?<log_value>(?<=-->)(.+))"
}
}
grok{
match => {
"msgbody" => "(?<Loggable_Message>(?<=ETL -)(.+))"
}
}

    if [log_category] == "KPI" or [log_category] == "EXCEPTION" {
        ruby {
            code => "
                log_value=event.get('log_value')
                entities=log_value.split('||',5)
                event.set('appName',entities[0])
                event.set('repoName',entities[1])
                event.set('resource_name',entities[2])
                event.set('resource_type',entities[3])
                kpi_list_string=entities[4]
                kpi_list_string[0]=''
                kpi_list_string[kpi_list_string.length-1]=''
                kpi_list=kpi_list_string.split('||')
                kpi_list.each do |kv_pair|
                    key=kv_pair.split(':',2)[0]
                    value=kv_pair.split(':',2)[1]
                    is_integer=value.match(/\A[+-]?\d+?(\.\d+)?\Z/) == nil ? false : true
                    if(is_integer==true)
                        value = value.to_i
                    end
                    event.set(key,value)
                end
            "
        }

    } 
    else if [log_category] == "STATUS"{
        ruby {
            code => "
                log_value=event.get('log_value')
                entities=log_value.split('||',4)
                event.set('appName',entities[0])
                event.set('repoName',entities[1])
                event.set('resource_name',entities[2])
                event.set('resource_status',entities[3])
            "
        }
    else if [loglevel] == "KPI" or [loglevel] == "ERROR" or [loglevel] == "INFO" {
    if [Loggable_Message] != "" {
        ruby {
            code => "
                Loggable_Message=event.get('Loggable_Message')
                entities=Loggable_Message.split('||',3)
                event.set('appName',entities[1])
                event.set('repoName',entities[0])
                kpi_list_string=entities[2]
                kpi_list_string[0]=''
                kpi_list_string[kpi_list_string.length-1]=''
                kpi_list=kpi_list_string.split('||')
                kpi_list.each do |kv_pair|
                    key=kv_pair.split(':',2)[0]
                    value=kv_pair.split(':',2)[1]
                    is_integer=value.match(/\A[+-]?\d+?(\.\d+)?\Z/) == nil ? false : true
                    if(is_integer==true)
                        value = value.to_i
                    end
                    event.set(key,value)
                end
            "
           }
       }
    }
    mutate {
        remove_field => ["log_value","msgbody","Loggable_Message]
    }

}

}

Input (logs)
2019-06-24 14:54:20 INFO etl - KPI-->Spark ETL Pipeline For Unstructured Repo||PANGAEA||classification/PANGAEA/056050050048056056046065069065071078065080:PANGAEA.json||STORE||(timeTakenToStoreInS3:85)
2019-06-24 14:54:20 KPI ETL - effr||Spark ETL Pipeline For Unstructured Repo||ResourceType:PIPE||pipeName:storeClassification||run_time:96||containerS3Key:pangaea/2019-03-06/PANGAEA.880228/

I am getting Ruby exception occurred: undefined method `split' for nil:NilClass.

2

I get compilation errors for that configuration. Please post the configuration that actually gets a ruby error. Select the configuration and click on </> in the toolbar above the edit pane so that it is block quoted, and verify in the preview pane that what is displayed matches your configuration.

3 
filter {
if "ETL_log_enriched"  in [tags] {
    grok {
    match => {
    "message" => "%{TIMESTAMP_ISO8601:log_timestamp} (%{LOGLEVEL:loglevel}|%{WORD:loglevel}) (?<classname>[^\]]+)- %{GREEDYDATA:msgbody}"
        }
    }
    mutate
    {
        add_field => ["event_read_time", "%{@timestamp}"]
    }
    date {
            match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss"]
            }
    grok {
    match => {
    "msgbody" => "(?<log_category>\w+(?=-->))"
        }
    }
    grok{
        match => {
        "msgbody" => "(?<log_value>(?<=-->)(.+))"
        }
    }
    grok{
        match => {  
        "msgbody" => "(?<Loggable_Message>(?<=ETL -)(.+))"
        }
    }

    if [log_category] == "KPI" or [log_category] == "EXCEPTION" {
        ruby {
            code => "
                log_value=event.get('log_value')
                entities=log_value.split('||',5)
                event.set('appName',entities[0])
                event.set('repoName',entities[1])
                event.set('resource_name',entities[2])
                event.set('resource_type',entities[3])
                kpi_list_string=entities[4]
                kpi_list_string[0]=''
                kpi_list_string[kpi_list_string.length-1]=''
                kpi_list=kpi_list_string.split('||')
                kpi_list.each do |kv_pair|
                    key=kv_pair.split(':',2)[0]
                    value=kv_pair.split(':',2)[1]
                    is_integer=value.match(/\A[+-]?\d+?(\.\d+)?\Z/) == nil ? false : true
                    if(is_integer==true)
                        value = value.to_i
                    end
                    event.set(key,value)
                end
            "
        }

    } 
    else if [log_category] == "STATUS"{
        ruby {
            code => "
                log_value=event.get('log_value')
                entities=log_value.split('||',4)
                event.set('appName',entities[0])
                event.set('repoName',entities[1])
                event.set('resource_name',entities[2])
                event.set('resource_status',entities[3])
            "
        }
}
    else if [loglevel] == "KPI" or [loglevel] == "ERROR" or [loglevel] == "INFO" {
    if [Loggable_Message] != "" {
        ruby {
            code => "
                Loggable_Message=event.get('Loggable_Message')
                entities=Loggable_Message.split('||',3)
                event.set('appName',entities[1])
                event.set('repoName',entities[0])
                kpi_list_string=entities[2]
                kpi_list=kpi_list_string.split('||')
                kpi_list.each do |kv_pair|
                    key=kv_pair.split(':',2)[0]
                    value=kv_pair.split(':',2)[1]
                    is_integer=value.match(/\A[+-]?\d+?(\.\d+)?\Z/) == nil ? false : true
                    if(is_integer==true)
                        value = value.to_i
                    end
                    event.set(key,value)
                end
            "
           }
       }
    }
    mutate {
        remove_field => ["log_value","msgbody","Loggable_Message"]
    }
  }
}
4

I've encountered NilClass errors when the field doesn't have contents. It seems without content, it doesn't "type" it, so it can't use the method.

5

@rugenl how can we check if a field has value or not ?

i.e if [Loggable_Message] != ""

6

The first message gets a _grokparsefailure, but not a _rubyexception. The second one gets both a _grokparsefailure and a _rubyexception. It has [loglevel] == "KPI" so the ruby filter does a split on [Loggable_Message], but that field does not exist.

I would always check that fetched fields exist in the ruby filter

        code => "
            Loggable_Message=event.get('Loggable_Message')
            if Loggable_Message
                entities=Loggable_Message.split('||',3)
                 [...]
            end
        "

If you want to test it in the filter section then the way to do that is

if [Loggable_Message] {

Your test does not work since nil is not equal to "".

Lastly, object names that start with a capital letter, like Loggable_Message are used for constants in ruby. It will likely produce warnings and confuse other people if you use that.

8

OK, so you are doing a positive lookbehind assertion in order to grab whatever follows "ETL -". That would work if you were matching against [message], but [msgbody] has already had that stripped off.

Also, a positive lookbehind is rather obscure. It would be simpler to do

grok { match => { "message" => "ETL - %{GREEDYDATA:loggableMessage}" } }
11

Does it do what you want? If not, what do you want to change?

13

I am getting _grokparsefailure

26 09:04:14 ip-10-95-36-47 logstash: [2] "_grokparsefailure"

14

Thanks Guys!!!

Now it is working fine

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.