# Logstash handling of field names with all caps **URL:** \ **Category:** Logstash\ **Created:** [August 14, 2017, 8:05pm UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017 "2017-08-14T20:05:20Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![crazyphil](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@crazyphil](https://discuss.elastic.co/u/crazyphil)\ **Post date:** [August 14, 2017, 8:05pm UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/1 "2017-08-14T20:05:20Z")
Logstash version is 5.4.3. I have been attempting to perform a grok on a field, however no matter what I do, the grok never occurs (it is wrapped in an if statement). My field name is "EVENT\_CLICK" and no matter what I do, I can't seem to perform any operations on that field name. If I execute a "mutate { add\_field =\> { "event\_click" =\> "%{EVENT\_CLICK}" } }" my event\_click field only ever says %{EVENT\_CLICK}. Here is my current code (note I never see the "yesyes" field created, despite valid matching data being present in the field): ``` if [collection] == "analytics" { mutate { rename => { "EVENT_CLICK" => "eventclick" } } if [eventclick] and [eventclick] != "" and [eventclick] =~ /home/ { grok { match => ["eventclick", 'click_home:id/(?[^/]+)' ] } mutate { add_field => { "yesyes" => "grok checked" } } } ``` Is there something I'm missing? No matter what else I've tried, I cannot seem to make anything happen to that field other than pass it through to ES (which lets me see/filter on it without issue). Suggestions? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 14, 2017, 8:14pm UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/2 "2017-08-14T20:14:19Z")
Please show your full configuration and use a `stdout { codec => rubydebug }` output to dump the raw events produced by Logstash. ---
**Author:** ![crazyphil](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@crazyphil](https://discuss.elastic.co/u/crazyphil)\ **Post date:** [August 14, 2017, 8:55pm UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/3 "2017-08-14T20:55:40Z")
I have added the output, the input line I use to test, and the full config I use to prove it's not creating a new field (in this case I went back to add\_field at it's easiest to show). Here is the output I get: ``` { "metrictype" => [ [0] "DATA", [1] "analytics" ], "hw_rev" => "2", "type" => "analytics", "kafka2es-proxy" => "analytics-es54-scus-4", "eventclick" => "%{EVENT_CLICK}", "tid" => "a8d6c049-f751-412c-9152-745d691ac50c", "host" => "orion-logf-1", "msgid" => "fcdfc934-cbda-4044-a6b5-0fb1e0971eff", "collection" => "analytics", "hw_part_nbr" => "1_0", "event_category" => "config_at_boot_time", "product_name" => "system1", "@timestamp" => 2017-08-14T20:48:00.626Z, "kafka2es_timestamp" => "2017-08-14T20:48:00.673Z", "sw_build" => 1000, "EVENT_CLICK" => "button_home:id/welcomeFragment", "category" => "INFO", "did" => "4ea7ec19-78e3-43f2-994d-d24d06759f9f", "sw_ver" => "0_9_0" } ``` Here is my test line I am inputting (instead of rerunning from Kafka, results are the same): `{"metrictype" : "DATA", "hw_rev" : "2", "type" : "analytics", "tid" : "a8d6c049-f751-412c-9152-745d691ac50c", "msgid" : "fcdfc934-cbda-4044-a6b5-0fb1e0971eff", "hw_part_nbr" : "1.0", "collection" : "analytics", "event_category" : "config_at_boot_time", "product_name" : "system1", "arrivalTime" : "2017-07-12T19:33:32.630Z", "uploadTime" : "2017-07-12T19:33:32.630Z", "event_time" : "2017-07-12T19:33:32.630Z", "sw_build" : 1000, "category" : "INFO", "did" : "4ea7ec19-78e3-43f2-994d-d24d06759f9f", "sw_ver": "0.9.0", "EVENT_CLICK": "button_home:id/welcomeFragment" }` input { ``` stdin { add_field => { "collection" => "analytics" } } } filter { if [collection] == "analytics" { mutate { add_field => { "eventclick" => "%{EVENT_CLICK}" } } mutate { add_field => { "btype" => "%{type}" } } date { match => ["arrivalTime", "ISO8601"] target => "server-arrival-time" tag_on_failure => ["_arrivaltimefailure"]} date { match => ["uploadTime", "ISO8601"] target => "client-upload-time" tag_on_failure => ["_uploadtimefailure"]} date { match => ["event_time", "ISO8601", "YYYY-MM-dd'T'HH:mm:ss'UTC'", "YYYY-MM-dd'T'HH:mm:ss'.GMT+8Z'"] tag_on_failure => ["_eventtimefailure"] } mutate { lowercase => ["tid"] } mutate { add_field => { "client-message" => "%{message}" } } mutate { add_field => { "msgid" => "%{id}" } } if [eventclick] and [eventclick] != "" and [eventclick] =~ /home/ { grok { match => ["eventclick", 'button_home:id/(?[^/]+)' ] } mutate { add_field => { "yesyes" => "grok checked" } } } if [client-message] != "%{message}" { # remove any dots that might be present in field names mutate { gsub => [ "client-message", "\.", "_" ] } json { source => "client-message" } } else { mutate { remove_field => ["client-message"] } } mutate { convert => { "value" => string "sw_build" => integer "id" => string } } mutate { add_field => { "configvarvalue" => "%{config_var_value}" } } if [configvarvalue] != "%{config_var_value}" { json { source => "configvarvalue" } } else { mutate { remove_field => ["configvarvalue"] } } # if [tid] == "" { mutate { replace => { "tid" => "no-tenant-id" } } } if ![tid] { mutate { replace => { "tid" => "no-tenant-id" } } } # check to see if something with a type field came in and overwrote the existing type specifier if [type] { if [type] != [btype] { mutate { add_field => { "metrictype" => "%{type}" } } mutate { replace => { "type" => "analytics" } } } } else { mutate { add_field => { "type" => "%{collection}" } } } } } output { stdout { codec => rubydebug } } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 15, 2017, 5:21am UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/4 "2017-08-15T05:21:22Z")
The input data already contains `"collection": "analytics"`, so the `add_field` in your input block makes the `collection` field an array. That causes the first conditional (`[collection] == "analytics"`) to become false. If I comment out the `add_field` line it works as expected. ---
**Author:** ![crazyphil](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@crazyphil](https://discuss.elastic.co/u/crazyphil)\ **Post date:** [August 15, 2017, 10:35am UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/5 "2017-08-15T10:35:54Z")
Ok, so I removed that first conditional check (so no if check on collection) to avoid any issues at all. When I send in any line of data with a field called EVENT\_CLICK, I can never touch that field. All the other fields are converted properly. I never get an "eventclick" field with anything other than "%{EVENT\_CLICK}". I also tried using ruby: ``` ruby { code => 'event.set("eventclick", event.get("EVENT_CLICK"))' } ``` And only ever get this: "eventclick" =\> nil, for an output. I never get a "works as expected" condition. ---
**Author:** ![crazyphil](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@crazyphil](https://discuss.elastic.co/u/crazyphil)\ **Post date:** [August 15, 2017, 10:46am UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/6 "2017-08-15T10:46:10Z")
If I reduce it to the one thing I am trying to get working: ``` input { stdin {} } filter { mutate { add_field => { "eventclick" => "%{EVENT_CLICK}" } } } output { stdout { codec => rubydebug } } ``` I get the completely wrong output: 03:45:07.687 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=\>9600} {"EVENT\_CLICK": "button\_home:id/welcomeFragment" } { "@timestamp" =\> 2017-08-15T10:45:28.807Z, "@version" =\> "1", "host" =\> "orion-logf-1", "message" =\> "{"EVENT\_CLICK": "button\_home:id/welcomeFragment" }", "eventclick" =\> "%{EVENT\_CLICK}" } I should have a field called "eventclick" with a value of "button\_home:id/welcomeFragment", but as you can see I get the field name, not the data from the field. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 15, 2017, 11:00am UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/7 "2017-08-15T11:00:56Z")
In your example you don't actually have an `EVENT_CLICK` field so it's no wonder that `%{EVENT_CLICK}` doesn't resolve. Try using a json codec for your input or a json filter so that the JSON string in the `message` field is deserialized. ---
**Author:** ![crazyphil](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@crazyphil](https://discuss.elastic.co/u/crazyphil)\ **Post date:** [August 15, 2017, 12:21pm UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/8 "2017-08-15T12:21:00Z")
Full text of the test (where EVENT\_CLICK is the ONLY field): [root@orion-logf-1 logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/lskakfa.conf WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console 03:45:07.592 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500} 03:45:07.646 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started The stdin plugin is now waiting for input: 03:45:07.687 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=\>9600} {"EVENT\_CLICK": "button\_home:id/welcomeFragment" } { "@timestamp" =\> 2017-08-15T10:45:28.807Z, "@version" =\> "1", "host" =\> "orion-logf-1", "message" =\> "{"EVENT\_CLICK": "button\_home:id/welcomeFragment" }", "eventclick" =\> "%{EVENT\_CLICK}" } ^C03:46:33.197 [SIGINT handler] WARN logstash.runner - SIGINT received. Shutting down the agent. 03:46:33.202 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=\>"main"} ---
**Author:** ![crazyphil](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@crazyphil](https://discuss.elastic.co/u/crazyphil)\ **Post date:** [August 15, 2017, 12:28pm UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/9 "2017-08-15T12:28:38Z")
Oh nevermind, you're right. I totally forgot I'm pulling things out of "message" and they aren't fields until parsed. Some days I'm just a nitwit. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 12, 2017, 12:28pm UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017/10 "2017-09-12T12:28:38Z")
Closed on September 12, 2017, 12:28pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.