# Split Document into multiple document based on condition **URL:** \ **Category:** Logstash\ **Created:** [October 17, 2022, 7:30am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737 "2022-10-17T07:30:01Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 17, 2022, 7:30am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/1 "2022-10-17T07:30:01Z")
Hi there, i plan to split field into multiple document based on OID. So, i have configured snmp pipeline for my network device. I managed to monitor the cpu and memory usage but i'm facing problem when i want to monitor network traffic on this device. in my pipeline configuration, i already add config like below: > walk =\> ["1.3.6.1.2.1.2.2.1.2", "1.3.6.1.2.1.2.2.1.10", "1.3.6.1.2.1.2.2.1.16"] 1.3.6.1.2.1.2.2.1.2 is OID for interface name 1.3.6.1.2.1.2.2.1.10 is OID for inbound traffic of the interface 1.3.6.1.2.1.2.2.1.16 is OID for outbound traffic of the interface After i apply configuration above, the log at elastic look like this: ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/a/aabfd60b21d0b139042f22c336c876c5c4a8ef28.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/9/091040d5629d0ed7986d250e447ffa35577acc71.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/3/434857cb6b97a27be75a100db4c83ec53835ed50.png) from the picture above, is it possible if I group the logs by Child OID ? my goal is group the field that have same child into different document. for example: > Field `iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifDescr.1` will be in the same document as `iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets.1` and `iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable .ifEntry.ifOutOctets.1` but will be in different document with `iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifDescr.10` so that way, I can display it in grafana. Can you help me please? Thank you if you have another idea, please don't hesitate to give it to me. FYI, there are 3 device that i want to monitor ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 17, 2022, 4:59pm UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/2 "2022-10-17T16:59:25Z")
I think you would have to use a ruby filter. You could start with something like ``` ruby { code => ' a = [] event.to_hash.each { |k, v| k.match(/sysOREntry\.(\w+)\.(\d+)$/) { |m| i = m[2].to_i a[i] ||= {} a[i][m[1]] = v } } event.set("data", a) ' } ``` which will convert a large set of fields like ``` "system.sysORTable.sysOREntry.sysORDescr.7" => "The MIB module for managing UDP implementations", "system.sysORTable.sysOREntry.sysORUpTime.1" => 4, "system.sysORTable.sysOREntry.sysORUpTime.6" => 4, "system.sysORTable.sysOREntry.sysORID.2" => "1.3.6.1.6.3.15.2.1.1", ``` into ``` "data" => [ [0] nil, [1] { "sysORUpTime" => 4, "sysORID" => "1.3.6.1.6.3.11.3.1.1", "sysORDescr" => "The MIB for Message Processing and Dispatching." }, [2] { "sysORUpTime" => 4, "sysORID" => "1.3.6.1.6.3.15.2.1.1", "sysORDescr" => "The management information definitions for the SNMP User-based Security Model." }, [3] { .... ``` You will likely need to add error handling and additional filtering. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 20, 2022, 8:37am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/4 "2022-10-20T08:37:15Z")
OK, in advance I'm very grateful to you because now the data is in accordance with what I want. but this ruby filter, causes my logstash to get OOM and it's quite often. ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/3/736be30b6f78a8ced241c121d9e8090841bfcb4b.png) can this ruby ​​filter still be tuned? So that way, my logstash won't get OOM again. Thank you ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 20, 2022, 4:05pm UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/5 "2022-10-20T16:05:53Z")
I cannot imagine why that ruby filter would cause an OOM error. See [this](https://discuss.elastic.co/t/exception-in-pipeline-worker-and-java-out-of-memory-errors/169916/2) thread for how to deal with that. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 22, 2022, 2:38am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/6 "2022-10-22T02:38:22Z")
I already apply filter like you mentioned before, but it's just drop the log if it has \_rubyexception in tags. it doesn't solve OOM error. after i apply that filter, i can still see error log like this: ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/5/85fd8764aa70ef51ea2af6d4998d74964bbada42.png) maybe for your information, My network devices that will be monitored increase to 64 and some of them have 52 interfaces. and my logstash has a 12 gigabyte jvm. do you think i must increase the spec of the logstash? or is there anything else I can do in this pipeline? Thank you ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 22, 2022, 3:55am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/7 "2022-10-22T03:55:22Z")
> [@yuswanul](#): > > it doesn't solve OOM error Nobody on the internet can solve that problem for you. You need to run the analysis. If you can provide the output of a heap dump analyzer then it is possible someone my be able to help you review it. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 22, 2022, 9:55am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/9 "2022-10-22T09:55:46Z")
from Memory analyzer tool (leak suspect) i got these 3 suspect ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/5/95bd68e2cbe065d8fca19532e1ac3ca89185c34c.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bfdb6238eb7876712bc976329298020637addbe4.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/f/ef4277f1481e6dc4dcd8ed4c2c9f5e649d7d197e.png) All the suspects refer to ruby. and this is some details from suspect 1. suspect 3 have a similar information like suspect 1 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4c5116d735d3d3a4c274eb8b1e6b01fbd99c3c00.png) And this is details from suspect 2 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fc73105c1cd98e479a6f6cac0041d19c087e8607.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/5/55af4a2e810bb53d690bb4a2ebf68d1a30e46b38.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/6/0617dcda1d873c24456f5726971a12d172f3529a.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/2/c2674aa6785fb53dab022207722ac1084b590c8d.png) Thank you ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 22, 2022, 1:10pm UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/10 "2022-10-22T13:10:41Z")
> [@Badger](#): > > You need to run the analysis. If you can provide the output of a heap dump analyzer [Here](https://drive.google.com/drive/folders/1eJJnhYiaLusuFmaH43cqeDDlI5Bp81tu?usp=sharing) is if you want the file ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 22, 2022, 5:26pm UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/11 "2022-10-22T17:26:14Z")
If you approve the request for access that I submitted then I will take a look at the dump. The MAT output does not have enough detail to diagnose the problem. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 22, 2022, 11:56pm UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/12 "2022-10-22T23:56:29Z")
> [@yuswanul](#): > > [Here](https://drive.google.com/drive/folders/1eJJnhYiaLusuFmaH43cqeDDlI5Bp81tu?usp=sharing) is if you want the file sorry, can you request it again to this link? i didn't receive a request access to this folder. i already upload MAT report and the heap dump file there. Thank you ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 23, 2022, 12:17am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/13 "2022-10-23T00:17:32Z")
I already update the permission. Technically, you can download it without request access. Thank you Pada tanggal Min, 23 Okt 2022 00.36, Badger via Discuss the Elastic Stack \<[notifications@elastic.discoursemail.com](mailto:notifications@elastic.discoursemail.com)\> menulis: ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 23, 2022, 12:40am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/14 "2022-10-23T00:40:13Z")
> [@yuswanul](#): > > Technically, you can download it without request access. Thank you Yeah, but in the post you deleted there was a different link which required me to request access. I downloaded it and MAT got an error trying to parse it. Is there any chance you could reproduce the problem with a smaller heap size? A 12 GB heap dump may be more than I can handle. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 23, 2022, 2:23am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/15 "2022-10-23T02:23:46Z")
Maybe i will send you a MAT report of the heap dump file later. If you have any instruction or tutorial how to download it. Please share it to me or you can try to change the -Xmx configuration on MemoryAnalyzer.ini file inside mat folder. if your computer or laptop has 16GB of RAM, you can change -Xmx configuration to 11g just like i did before to generate MAT report Pada tanggal Min, 23 Okt 2022 07.50, Badger via Discuss the Elastic Stack \<[notifications@elastic.discoursemail.com](mailto:notifications@elastic.discoursemail.com)\> menulis: ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 23, 2022, 2:29am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/16 "2022-10-23T02:29:41Z")
Or you can try this [link](http://54.176.184.79:8080/my-heap-report.jsp?p=L2hvbWUvZWMyLXVzZXIveWNyYXNoLTFfMTIvLi95Y3Jhc2gvWUNTdXBwb3J0LzIwMjIvMTAvMjMvMTcyLjMxLjUuODUveWMvMDAtNDEtMTcvMTcyLjMxLjUuODUtMDAtNDEtMTcteWMteWMuYmluLnppcC5qc29u). I got this from heaphero team. I already contact them before to help me to analyzing my heap dump file and they just sent the report today ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 23, 2022, 2:54am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/17 "2022-10-23T02:54:52Z")
> [@yuswanul](#): > > or you can try to change the -Xmx configuration on MemoryAnalyzer.ini file inside mat folder. if your computer or laptop has 16GB of RAM, you can change -Xmx configuration to 11g I didn't really want to do that since it resulted in the JVM growing the memory usage to 15.9 GB and swapping every other program to disk, so that they take 30+ seconds to respond. That said, it worked and many of those programs will get swapped back into RAM overnight. I'll be back with an analysis in about 14 hours. MAT only needs about 4 GB to store the analysis, so that's no problem. I was amazed by the level of compression that zip achieved on the heap dump. 150 MB to contain a 12 GB dump. Probably lots and lots of zeroes. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 23, 2022, 3:09am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/18 "2022-10-23T03:09:09Z")
Ok, thank you @Badger . Alternatively, you can use the link that i just share to you from heaphero team ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 23, 2022, 6:18pm UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/19 "2022-10-23T18:18:02Z")
There are two worker threads, each of which has hundreds of millions of references to the same nil object. Each reference retains 40 bytes on the heap. ![clip0](https://us1.discourse-cdn.com/elastic/original/3X/d/f/df5de2fed2b65eece8a5d6d8ba6a195c117b6b49.png) ![clip1](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7cf838cae1ee603bc7a0b3609bc8fc4714b954e2.png) I have absolutely no clue as to how the ruby filter I suggested could cause that. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 25, 2022, 4:13am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/20 "2022-10-25T04:13:09Z")
is it possible if this is caused due to too much data and insufficient memory? because of the network devices that I monitor, some of them have up to 52 interfaces and I also monitor the inbound and outbound traffic from each interface. so there's a lot of data being sent to elastic and maybe logstash is overwhelmed with organizing that data ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 25, 2022, 4:23am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/21 "2022-10-25T04:23:18Z")
> [@yuswanul](#): > > is it possible if this is caused due to too much data and insufficient memory? I don't see it. The threads have accumulated around half a billion nil objects. I have no idea how that could have happened. ---
**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Post date:** [October 25, 2022, 6:54am UTC](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737/22 "2022-10-25T06:54:47Z")
ok, thanks for your help [Next page](https://discuss.elastic.co/t/split-document-into-multiple-document-based-on-condition/316737.md?page=2)