# Logstash splitting and tagging events **URL:** \ **Category:** Logstash\ **Created:** [November 24, 2020, 8:09pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571 "2020-11-24T20:09:50Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![John\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_smith/32/79638_2.png) [@John\_Smith](https://discuss.elastic.co/u/John_Smith)\ **Post date:** [November 24, 2020, 8:09pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571/1 "2020-11-24T20:09:51Z")
I need to ingest following json as separated events ``` { "test1": {some nested json here}, "test2": {some nested json here}, "test3": {some nested json here}, "test4": {some nested json here} } ``` I have 3 problems: - When i try to clone: I got json inside, but as nested json ``` { "test1": { .... } } ``` - Second one is with output: Now i can ingest with: I can do same for all cloned items, but i guess there is more clever way to do it. - Last one is question related to identifying events - something like: if format is { "test1":{},"test2":{},"test3":{},"test4":{} } then do something else do something different I guess this should be done with grok, but I'll play whit that after manage to fix first 2 issues. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 24, 2020, 8:53pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571/2 "2020-11-24T20:53:25Z")
Do you want to discard "test1" and just keep the contents of "{some nested json here}" as the top-level fields in the event? ---
**Author:** ![John\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_smith/32/79638_2.png) [@John\_Smith](https://discuss.elastic.co/u/John_Smith)\ **Post date:** [November 24, 2020, 9:00pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571/3 "2020-11-24T21:00:33Z")
Yes, all json elements from test1 should go to root json. General idea is to ingest content of nested jsons as separate events - maybe there is better idea ... Source json: { "test1": { "element1": "value1" "element2": "value2" "element3": "value3" } "test2": { "element21": "value21" "element22": "value22" "element23": "value23" } "test3": { "element31": "value31" "element32": "value32" "element33": "value33" } } Expected output: event 1 { "element1": "value1" "element2": "value2" "element3": "value3" } event 2 { "element21": "value21" "element22": "value22" "element23": "value23" } event 3 { "element31": "value31" "element32": "value32" "element33": "value33" } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 24, 2020, 9:05pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571/4 "2020-11-24T21:05:06Z")
You could use clone and mutate and so on, but I would do it in ruby ``` input { generator { count => 1 lines => ['{ "test1": { "foo": 1, "bar": 8 }, "test2": { "foo": 2 }, "test3": { "foo": 3 }, "test4": { "foo": 4 } }'] } } filter { json { source => "message" target => "data" remove_field => ["message"] } ruby { code => ' data = event.get("data") if data event.remove("data") data.each { |k, v| newEvent = event.clone v.each { |k, v| newEvent.set(k, v) new_event_block.call(newEvent) } } event.cancel end ' } ``` which gets me ``` { "@timestamp" => 2020-11-24T21:02:25.876Z, "host" => "....", "foo" => 2, "sequence" => 0, "@version" => "1" } { "@timestamp" => 2020-11-24T21:02:25.876Z, "foo" => 1, "sequence" => 0, "@version" => "1", "host" => "...", "bar" => 8 } ``` ---
**Author:** ![John\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_smith/32/79638_2.png) [@John\_Smith](https://discuss.elastic.co/u/John_Smith)\ **Post date:** [November 24, 2020, 9:08pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571/5 "2020-11-24T21:08:47Z")
Problem here is unknown amount of elements inside nested json. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 24, 2020, 10:11pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571/6 "2020-11-24T22:11:49Z")
> [@John\_Smith](#): > > Problem here is unknown amount of elements inside nested json. Why do you think that is a problem? My example demonstrates that it handles arbitrary elements in the nested JSON. ---
**Author:** ![John\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_smith/32/79638_2.png) [@John\_Smith](https://discuss.elastic.co/u/John_Smith)\ **Post date:** [November 24, 2020, 10:25pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571/7 "2020-11-24T22:25:11Z")
You're right, not skilled naf event to read ruby code. Should check and adapt it a bit as not it's generated 3000+ lines output - somehow multiplied events ~15 times What about identifying specific format ? Developers are migrating to this new format and for sometime I'll get mix old events (what is inside test1, test2 etc) as single events and new type of events, where I've this combination of nested jsons ? if somehow i manage to tag new events, i can put this ruby filter. if it's not, then process with old filters. Of course that's just theory. ---
**Author:** ![John\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_smith/32/79638_2.png) [@John\_Smith](https://discuss.elastic.co/u/John_Smith)\ **Post date:** [December 2, 2020, 12:02pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571/8 "2020-12-02T12:02:15Z")
Hello @Badger, As ruby solution is not acceptable for our support guys, i've requested developers to change format of logging to: { "result": [{ "field1": "valuex", "field2": "valuey" ``` }, { "field123": "valuexyd", "field243": "valueyzd" }, { "field444": "valueccy", "field234": "valueyzc" }, { "field444": "valueccy", "field234": "valueyyc" }, .......... ,{ "field4674": "valueddy", "field2134": "valuezyd" }] ``` } Idea is to generate X amount of single events, based on nested jsons. Importunately when i use split it's generating separate events, but again inside nested json ``` { "result" : { "field444": "valueccy", "field234": "valueyzc" } } ``` What i need is event like: ``` { "field444": "valueccy", "field234": "valueyzc" } ``` My test filter is: ``` filter { dissect { mapping => { "message" => "%{garbage}Info: %{msg}" } } json { source => "msg" remove_field => ["garbage", "message" ,"msg"] } split { field => "result" id => "SPLIT" remove_field => ["msg"] } } ``` Thanks in advance for your support. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 30, 2020, 12:02pm UTC](https://discuss.elastic.co/t/logstash-splitting-and-tagging-events/256571/9 "2020-12-30T12:02:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.