# Logstash mutating/renaming with glob **URL:** \ **Category:** Logstash\ **Created:** [January 26, 2019, 8:30pm UTC](https://discuss.elastic.co/t/logstash-mutating-renaming-with-glob/165865 "2019-01-26T20:30:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![tszyro](https://avatars.discourse-cdn.com/v4/letter/t/9fc29f/32.png) [@tszyro](https://discuss.elastic.co/u/tszyro)\ **Post date:** [January 26, 2019, 8:30pm UTC](https://discuss.elastic.co/t/logstash-mutating-renaming-with-glob/165865/1 "2019-01-26T20:30:56Z")
I try to efficiently rename pattern in logstash. So if the json field comes as `instrument_network_a` it will be renamed to `network.labels.a`, it it comes as `instrument_network_b` it will be renamed to `network.labels.b` and so on. Really what I thought it would be some `mutate { gsub { } }` expression but can't figure out how to make it to work with glob (`*`). So I try ruby filter instead. I tried to wrap `gsub` inside the event to hash conversion but I have a problems to make it executing. Below are three cases of different approaches: Case 1 taken from [StackOverflow](https://stackoverflow.com/questions/31068169/how-to-remove-underscores-in-field-names-with-logstash/37617214#37617214) but it causes logstash to break execution. ``` 1 - pipeline.id: amq 2 config.string: | 3 input { stdin { } } 4 5 filter { 6 json { source => "message" } 7 ruby { code => ' event.to_hash.each_pair { |k, v| 8 if !!(/instrument_network_*/ =~ k) 9 newkey = k.gsub(/^instrument_network_/, 'network.labels.') 10 event.set(newkey, v); 11 event.remove(k) 12 end }' 13 } 14 } 15 16 output { stdout { codec => rubydebug } } ``` When I run it it just stops: ``` $ /usr/share/logstash/bin/logstash --path.settings . --path.data data -l . --log.level debug Sending Logstash's logs to . which is now configured via log4j2.properties $ ``` I logstash-plain.log the error shows as: ``` [2019-01-26T20:14:26,015][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:amq, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 7, column 62 (byte 229) after filter {\n json { source => \"message\" }\n ruby { code => ' event.to_hash.each_pair { |k, v|\n if !!(/instrument_network_*/ =~ k)\n newkey = k.gsub(/^instrument_network_/, '", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]} ``` which I don't really see any clue why it does not work. I have built another parsing solution base on string ``` 1 - pipeline.id: amq 2 config.string: | 3 input { stdin { } } 4 5 filter { 6 json { source => "message" } 7 ruby { code => ' event.to_hash.each_pair { |k, v| 8 if !!(/instrument_network_*/ =~ k) 9 newkey = "#{["network.labels", k.split("_", 3)[2]].join(".")}" 10 event.set(newkey, v); 11 event.remove(k) 12 end }' 13 } 14 } 15 16 output { stdout { codec => rubydebug } } ``` with the output: ``` $ /usr/share/logstash/bin/logstash --path.settings . --path.data data -l . --log.level debug Sending Logstash's logs to . which is now configured via log4j2.properties The stdin plugin is now waiting for input: {"instrument_network_a":false, "instrument_network_b":true, "instrument_network":false} { "network.labels.a" => false, "@version" => "1", "@timestamp" => 2019-01-26T20:08:52.692Z, "network.labels.b" => true, "network.labels." => false, "host" => "***", "message" => "{\"instrument_network_a\":false, \"instrument_network_b\":true, \"instrument_network\":false}" } ``` but it feels neither clear to read not efficient to work 1. What would be the most efficient way to built "mutate" filter using glob? 2. Is there any way to measure or time execution of the filter? 3. Why `rsub` does not work in this case. As far as I read the docs and check syntax, I understand it should work. 4. Ideally I would like to eliminate `network.labels` without the name of the instrument. So `network.labels.a` is accepted but `network.labels.` really is not. Is there a simple/efficient way to incorporate such behaviour? All advise well appreciated. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 26, 2019, 9:30pm UTC](https://discuss.elastic.co/t/logstash-mutating-renaming-with-glob/165865/2 "2019-01-26T21:30:05Z")
> [@tszyro](#): > > "Expected one of #, {, } at line 7, column 62 (byte 229) after filter {\n json { source =\> "message" }\n ruby { code =\> ' event.to\_hash.each\_pair { |k, v|\n if !!(/instrument\_network\_\*/ =~ k)\n newkey = k.gsub(/^instrument\_network\_/, '" OK, so it blows up when it hits the single quote used to quote 'network.labels.'. You are using single quotes for code =\> '...', so either use double quotes around "network.labels." or escape the single quotes. Do not use period in field names. It works most of the time, right up to the point where things break. ---
**Author:** ![tszyro](https://avatars.discourse-cdn.com/v4/letter/t/9fc29f/32.png) [@tszyro](https://discuss.elastic.co/u/tszyro)\ **Post date:** [January 26, 2019, 9:52pm UTC](https://discuss.elastic.co/t/logstash-mutating-renaming-with-glob/165865/4 "2019-01-26T21:52:02Z")
the escape does not seems to work: ``` [ERROR][logstash.agent] Failed to execute action {:id=>:amq, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil} [2019-01-26T21:49:50,518][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SyntaxError) (ruby filter code):4: syntax error, unexpected null newkey = k.gsub(/^instrument_network_/, \'network.labels.\') ``` ---
**Author:** ![tszyro](https://avatars.discourse-cdn.com/v4/letter/t/9fc29f/32.png) [@tszyro](https://discuss.elastic.co/u/tszyro)\ **Post date:** [January 26, 2019, 9:59pm UTC](https://discuss.elastic.co/t/logstash-mutating-renaming-with-glob/165865/5 "2019-01-26T21:59:24Z")
Looks like double quote `"` does the job although I read that really single one should be used in ruby gsub. Maybe I am confused now. Hope it will not cause problems in production. ---
**Author:** ![tszyro](https://avatars.discourse-cdn.com/v4/letter/t/9fc29f/32.png) [@tszyro](https://discuss.elastic.co/u/tszyro)\ **Post date:** [January 28, 2019, 3:46pm UTC](https://discuss.elastic.co/t/logstash-mutating-renaming-with-glob/165865/6 "2019-01-28T15:46:16Z")
Thanks @Badger. When you write "do not use period in field names ... things break " what do you mean by that? I see you have good knowledge of logstash and and honestly your comment made me worry. I follow elastic [ECS](https://github.com/elastic/ecs) and all of it is built on periods in the names. What should I be prepared for? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 28, 2019, 4:04pm UTC](https://discuss.elastic.co/t/logstash-mutating-renaming-with-glob/165865/7 "2019-01-28T16:04:17Z")
In 5.0 or later dots are allowed in field names. However, I know I've seen things break when you use them. For example, as I recall it, sometimes machine learning would replace a field name containing dots with something like "field#17". ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 25, 2019, 4:04pm UTC](https://discuss.elastic.co/t/logstash-mutating-renaming-with-glob/165865/8 "2019-02-25T16:04:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.