# Logstash - How to Dynamic Parse Log's value **URL:** \ **Category:** Logstash\ **Created:** [September 15, 2023, 10:00am UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125 "2023-09-15T10:00:12Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Huy\_Hoang\_Le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huy_hoang_le/32/125447_2.png) [@Huy\_Hoang\_Le](https://discuss.elastic.co/u/Huy_Hoang_Le)\ **Post date:** [September 15, 2023, 10:00am UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/1 "2023-09-15T10:00:12Z")
Hi I have this sample Document ```auto [Thread-13][2023-09-15 09:32:35][INFO]:{'[Sub]0-BaseTransformer]': '0.0004', '[Sub]1-NGINX Feature Extractor Service]': '0.0135', '[Dataloader][#0.-PutToQueue]': '0.0005', '[Sub][#1.EMA_FPS|CURRENT_FPS]': '183.77|69.58', '[Sub][#2.FRAMEID': 143, '[Sub][#3.DataNum': 2, '[Sub][#4.QueueSize': 0} ``` How can I dynamically parse, extract and get the value of each subfield after curly braces . For example . 0.004, 183.77|69.58, 0.0135 .v.v.v I have used so many filter but it doesn't work somehow. here is one that doesn't work: ```auto input { beats { port => 5044 } } filter { grok { match => { "message" => "\[%{DATA:thread}\]\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{WORD:log_level}\]:%{GREEDYDATA:log_data}" } } kv { source => "log_data" field_split => "," value_split => ":" } } output { elasticsearch { hosts => ["localhost:9200"] index => "test" } } ``` ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [September 15, 2023, 12:33pm UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/2 "2023-09-15T12:33:33Z")
Looks like JSON ---
**Author:** ![Huy\_Hoang\_Le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huy_hoang_le/32/125447_2.png) [@Huy\_Hoang\_Le](https://discuss.elastic.co/u/Huy_Hoang_Le)\ **Post date:** [September 15, 2023, 1:55pm UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/3 "2023-09-15T13:55:44Z")
Hi there, can you tell me if I’m right or wrong ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 15, 2023, 6:28pm UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/4 "2023-09-15T18:28:32Z")
You could start with ``` dissect { mapping => { "message" => "[%{thread}][%{[@metadata][ts]}][%{loglevel}]:%{[@metadata][restOfLine]}" } } mutate { gsub => ["[@metadata][ts]", " ", "T" ] } date { match => ["[@metadata][ts]", "ISO8601" ] } mutate { gsub => ["[@metadata][restOfLine]", "'", '"', "[@metadata][restOfLine]", "\[", "{", "[@metadata][restOfLine]", "]", "}" ] } json { source => "[@metadata][restOfLine]" } ``` which will get you ``` "{Sub}{#2.FRAMEID" => 143, "{Dataloader}{#0.-PutToQueue}" => "0.0005", "{Sub}{#4.QueueSize" => 0, "{Sub}{#3.DataNum" => 2 ``` etc. ---
**Author:** ![Huy\_Hoang\_Le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huy_hoang_le/32/125447_2.png) [@Huy\_Hoang\_Le](https://discuss.elastic.co/u/Huy_Hoang_Le)\ **Post date:** [September 18, 2023, 2:24am UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/5 "2023-09-18T02:24:40Z")
Hi, thank for your respone, btw How can I split the value of these to two? For example: > EMA\_FPS : 183.77 > CURRENT\_FPS : 69.58 > [@Huy\_Hoang\_Le](#): > > `'[Sub][#1.EMA_FPS|CURRENT_FPS]': '183.77|69.58'` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 18, 2023, 5:30pm UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/6 "2023-09-18T17:30:39Z")
``` ruby { code => ' event.to_hash.each { |k, v| if k =~ /\|/ and v.to_s =~ /\|/ k = k.sub(/.*\./, "").sub(/}$/, "").split(/\|/) v = v.split(/\|/) k.each_index { |x| event.set(k[x], v[x]) } end } ' } ``` works for that example, but has no error handling and is fairly fragile with respect to the data format. ---
**Author:** ![Huy\_Hoang\_Le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huy_hoang_le/32/125447_2.png) [@Huy\_Hoang\_Le](https://discuss.elastic.co/u/Huy_Hoang_Le)\ **Post date:** [September 19, 2023, 2:57am UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/7 "2023-09-19T02:57:43Z")
Thank you Badger, final question is there anyway I can replace '{' and '}' with '[' and ']' in the final output of log, it's for nicer look for exp: > original key-value output =\> "{Dataloader}{#0.-PutToQueue}" =\> "0.0005" > Modified: "{Dataloader}{#0.-PutToQueue}" replace { } with [] =\> [Dataloader][#0.-PutToQueue] ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 19, 2023, 3:27am UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/8 "2023-09-19T03:27:12Z")
You could, using a mutate+gsub to reverse the effects of the second and third triplets in this filter. ``` mutate { gsub => ["[@metadata][restOfLine]", "'", '"', "[@metadata][restOfLine]", "\[", "{", "[@metadata][restOfLine]", "]", "}" ] } ``` But I recommended this filter because if you have field names like "[foo][bar" then logstash may object when you try to reference them in some ways. If you do it as the last filter then it may well work. I do not think elasticsearch will object to unbalanced square brackets in a field name. ---
**Author:** ![Huy\_Hoang\_Le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huy_hoang_le/32/125447_2.png) [@Huy\_Hoang\_Le](https://discuss.elastic.co/u/Huy_Hoang_Le)\ **Post date:** [September 19, 2023, 3:36am UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/9 "2023-09-19T03:36:25Z")
ah I got your point, you meant I need to write another filter to reverse the effects of the second and third triplets in that filter? And actually I changed the log format a little bit. ` [Thread-13][2023-09-15 09:32:35][INFO]:{'[Sub][0-BaseTransformer]': '0.0004', '[Sub][1-NGINX Feature Extractor Service]': '0.0135', '[Dataloader][#0.-PutToQueue]': '0.0005', '[Sub][#1.EMA_FPS|CURRENT_FPS]': '183.77|69.58', '[Sub][#2.FRAMEID]': 143, '[Sub][#3.DataNum]': 2, '[Sub][#4.QueueSize]': 0}` =\> I changed for exp : > [Sub]#2.FRAMEID] =\> [Sub][#2.FRAMEID] for more consistent. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 17, 2023, 3:37am UTC](https://discuss.elastic.co/t/logstash-how-to-dynamic-parse-logs-value/343125/10 "2023-10-17T03:37:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.