Logstash handling of field names with all caps


(Phil) #1

Logstash version is 5.4.3.

I have been attempting to perform a grok on a field, however no matter what I do, the grok never occurs (it is wrapped in an if statement).

My field name is "EVENT_CLICK" and no matter what I do, I can't seem to perform any operations on that field name.

If I execute a "mutate { add_field => { "event_click" => "%{EVENT_CLICK}" } }" my event_click field only ever says %{EVENT_CLICK}.

Here is my current code (note I never see the "yesyes" field created, despite valid matching data being present in the field):

if [collection] == "analytics" {
      mutate { rename => { "EVENT_CLICK" => "eventclick" } }  
      if [eventclick] and [eventclick] != "" and [eventclick] =~ /home/ {
                grok {
                                 match => [ "eventclick", 'click_home:id/(?<event_click_normalized>[^/]+)' ]
                }
               mutate { add_field => { "yesyes" => "grok checked" } }
}

Is there something I'm missing? No matter what else I've tried, I cannot seem to make anything happen to that field other than pass it through to ES (which lets me see/filter on it without issue).

Suggestions?


(Magnus Bäck) #2

Please show your full configuration and use a stdout { codec => rubydebug } output to dump the raw events produced by Logstash.


(Phil) #3

I have added the output, the input line I use to test, and the full config I use to prove it's not creating a new field (in this case I went back to add_field at it's easiest to show).

Here is the output I get:

{
            "metrictype" => [
        [0] "DATA",
        [1] "analytics"
    ],
                "hw_rev" => "2",
                  "type" => "analytics",
        "kafka2es-proxy" => "analytics-es54-scus-4",
            "eventclick" => "%{EVENT_CLICK}",
                   "tid" => "a8d6c049-f751-412c-9152-745d691ac50c",
                  "host" => "orion-logf-1",
                 "msgid" => "fcdfc934-cbda-4044-a6b5-0fb1e0971eff",
            "collection" => "analytics",
           "hw_part_nbr" => "1_0",
        "event_category" => "config_at_boot_time",
          "product_name" => "system1",
            "@timestamp" => 2017-08-14T20:48:00.626Z,
    "kafka2es_timestamp" => "2017-08-14T20:48:00.673Z",
              "sw_build" => 1000,
           "EVENT_CLICK" => "button_home:id/welcomeFragment",
              "category" => "INFO",
                   "did" => "4ea7ec19-78e3-43f2-994d-d24d06759f9f",
                "sw_ver" => "0_9_0"
}

Here is my test line I am inputting (instead of rerunning from Kafka, results are the same):

{"metrictype" : "DATA", "hw_rev" : "2", "type" : "analytics", "tid" : "a8d6c049-f751-412c-9152-745d691ac50c", "msgid" : "fcdfc934-cbda-4044-a6b5-0fb1e0971eff", "hw_part_nbr" : "1.0", "collection" : "analytics", "event_category" : "config_at_boot_time", "product_name" : "system1", "arrivalTime" : "2017-07-12T19:33:32.630Z", "uploadTime" : "2017-07-12T19:33:32.630Z", "event_time" : "2017-07-12T19:33:32.630Z", "sw_build" : 1000, "category" : "INFO", "did" : "4ea7ec19-78e3-43f2-994d-d24d06759f9f", "sw_ver": "0.9.0", "EVENT_CLICK": "button_home:id/welcomeFragment" }

input {

     stdin { add_field => { "collection" => "analytics" } }

}

filter {

        if [collection] == "analytics" {

                                                mutate { add_field => { "eventclick" => "%{EVENT_CLICK}" } }
                                                mutate { add_field => { "btype" => "%{type}" } }
                                                date { match => [ "arrivalTime", "ISO8601" ] target => "server-arrival-time" tag_on_failure => ["_arrivaltimefailure"]}
                                                date { match => [ "uploadTime", "ISO8601" ] target => "client-upload-time" tag_on_failure => ["_uploadtimefailure"]}
                                                date { match => [ "event_time", "ISO8601", "YYYY-MM-dd'T'HH:mm:ss'UTC'", "YYYY-MM-dd'T'HH:mm:ss'.GMT+8Z'"] tag_on_failure => ["_eventtimefailure"] }

                                                mutate { lowercase => [ "tid" ] }
                                                mutate { add_field => { "client-message" => "%{message}" } }
                                                mutate { add_field => { "msgid" => "%{id}" } }
                                                if [eventclick] and [eventclick] != "" and [eventclick] =~ /home/ {
                                                        grok {
                                                                match => [ "eventclick", 'button_home:id/(?<event_click_normalized>[^/]+)' ]
                                                        }
                                                        mutate { add_field => { "yesyes" => "grok checked" } }
                                                }

                                                if [client-message] != "%{message}" { 
                                                                                        # remove any dots that might be present in field names
                                                                                        mutate {
                                                                                                gsub => [
                                                                                                          "client-message", "\.", "_"
                                                                                                ]
                                                                                        }

                                                                                        json { source => "client-message" }
                                                } else { 
                                                                                        mutate { remove_field => [ "client-message"] }  
                                                }
                                                mutate { convert => { "value" => string "sw_build" => integer "id" => string } }
                                                mutate { add_field => { "configvarvalue" => "%{config_var_value}" } }
                                                if [configvarvalue] != "%{config_var_value}" {
                                                                                        json { source => "configvarvalue" }
                                                } else {
                                                                                        mutate { remove_field => [ "configvarvalue"] }
                                                }
 
 

#                                                if [tid] == "" { mutate { replace => { "tid" => "no-tenant-id" } } }
                                                if ![tid] { mutate { replace => { "tid" => "no-tenant-id" } } }
# check to see if something with a type field came in and overwrote the existing type specifier

                                                if [type] {
                                                        if [type] != [btype]  {
                                                                mutate { add_field => { "metrictype" => "%{type}" } }
                                                                mutate { replace => { "type" => "analytics" } }
                                                        }
                                                } else {
                                                        mutate { add_field => { "type" => "%{collection}" } }
                                                }
        } 

}

output {

        stdout { codec => rubydebug }
        }
}

(Magnus Bäck) #4

The input data already contains "collection": "analytics", so the add_field in your input block makes the collection field an array. That causes the first conditional ([collection] == "analytics") to become false. If I comment out the add_field line it works as expected.


(Phil) #5

Ok, so I removed that first conditional check (so no if check on collection) to avoid any issues at all.

When I send in any line of data with a field called EVENT_CLICK, I can never touch that field. All the other fields are converted properly. I never get an "eventclick" field with anything other than "%{EVENT_CLICK}".

I also tried using ruby:

                                           ruby {
                                                   code => 'event.set("eventclick", event.get("EVENT_CLICK"))'
                                           }

And only ever get this:

"eventclick" => nil,

for an output.

I never get a "works as expected" condition.


(Phil) #6

If I reduce it to the one thing I am trying to get working:

input {
             stdin {}
}

    filter {

                                                    mutate { add_field => { "eventclick" => "%{EVENT_CLICK}" } }

    }

output {
             stdout { codec => rubydebug }
}

I get the completely wrong output:

03:45:07.687 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
{"EVENT_CLICK": "button_home:id/welcomeFragment" }
{
"@timestamp" => 2017-08-15T10:45:28.807Z,
"@version" => "1",
"host" => "orion-logf-1",
"message" => "{"EVENT_CLICK": "button_home:id/welcomeFragment" }",
"eventclick" => "%{EVENT_CLICK}"
}

I should have a field called "eventclick" with a value of "button_home:id/welcomeFragment", but as you can see I get the field name, not the data from the field.


(Magnus Bäck) #7

In your example you don't actually have an EVENT_CLICK field so it's no wonder that %{EVENT_CLICK} doesn't resolve. Try using a json codec for your input or a json filter so that the JSON string in the message field is deserialized.


(Phil) #8

Full text of the test (where EVENT_CLICK is the ONLY field):

[root@orion-logf-1 logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/lskakfa.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
03:45:07.592 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
03:45:07.646 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
03:45:07.687 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
{"EVENT_CLICK": "button_home:id/welcomeFragment" }
{
"@timestamp" => 2017-08-15T10:45:28.807Z,
"@version" => "1",
"host" => "orion-logf-1",
"message" => "{"EVENT_CLICK": "button_home:id/welcomeFragment" }",
"eventclick" => "%{EVENT_CLICK}"
}
^C03:46:33.197 [SIGINT handler] WARN logstash.runner - SIGINT received. Shutting down the agent.
03:46:33.202 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>"main"}


(Phil) #9

Oh nevermind, you're right. I totally forgot I'm pulling things out of "message" and they aren't fields until parsed.

Some days I'm just a nitwit.


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.