# Help ingesting Data **URL:** \ **Category:** Logstash\ **Created:** [July 17, 2023, 3:18pm UTC](https://discuss.elastic.co/t/help-ingesting-data/338580 "2023-07-17T15:18:35Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Tom\_Dixon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_dixon/32/123545_2.png) [@Tom\_Dixon](https://discuss.elastic.co/u/Tom_Dixon)\ **Post date:** [July 17, 2023, 3:18pm UTC](https://discuss.elastic.co/t/help-ingesting-data/338580/1 "2023-07-17T15:18:35Z")
Hi all, I'm new to Elastic and Logstash. I have a source of event data which I'm having problems ingesting. I think it is because the data itself, but being new to Logstash it could also be me, so I'm not sure where the issue is. I've tried so many different things I've lost track. I created a HTTP input on port 9999 and also turned off security due to certificate issues. The source is sending each event in its own packet in a steam in the form of: {"time": 1689541912.454, "event": {"nodeId": "nWXoUZ4CNTRL", "start": "2023-07-16T21:11:50.979186Z", "end": "2023-07-1621:11:53.929365 z", "virtualTraffic": [{"proto": 17, "src" : " [fd7a: 115c: ale0: ab12:4843: cd96:624:9181]:55993", "dst":" [fd7a: 115c: ale0::]:80", "txPkts" :1, "txBytes" :50}]3, "fields": {"recorded": "2023-07-1621:11:55.945733486Z"}} But there is no comma between the events and it doesn't appear they are being put in an array either. My output appears like this: {"time":1689606809.419,"event":{"nodeId":"nWXoUZ4CNTRL","start":"2023-07-17T15:13:27.94877Z","end":"2023-07-17T15:13:30.889983Z","virtualTraffic":[{"proto":17,"src":"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:62498","dst":"[fd7a:115c:a1e0::]:80","txPkts":1,"txBytes":50}]},"fields":{"recorded":"2023-07-17T15:13:32.936285253Z"}}{"time":1689606814.419,"event":{"nodeId":"nWXoUZ4CNTRL","start":"2023-07-17T15:13:32.950684Z","end":"2023-07-17T15:13:35.889057Z","virtualTraffic":[{"proto":17,"src":"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:56573","dst":"[fd7a:115c:a1e0::]:80","txPkts":1,"txBytes":50}]},"fields":{"recorded":"2023-07-17T15:13:37.935786873Z"}}{"time":1689606819.414,"event":{"nodeId":"nWXoUZ4CNTRL","start":"2023-07-17T15:13:37.938953Z","end":"2023-07-17T15:13:40.889584Z","virtualTraffic":[{"proto":17,"src":"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:59613","dst":"[fd7a:115c:a1e0::]:80","txPkts":1,"txBytes":50}]},"fields":{"recorded":"2023-07-17T15:13:42.937654623Z"}}{"time":1689606824.418,"event":{"nodeId":"nWXoUZ4CNTRL","start":"2023-07-17T15:13:42.947124Z","end":"2023-07-17T15:13:45.890088Z","virtualTraffic":[{"proto":17,"src":"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:63773","dst":"[fd7a:115c:a1e0::]:80","txPkts":1,"txBytes":50}]},"fields":{"recorded":"2023-07-17T15:13:47.93908194Z"}}{"time":1689606829.421,"event":{"nodeId":"nWXoUZ4CNTRL","start":"2023-07-17T15:13:47.95292Z","end":"2023-07-17T15:13:50.890138Z","virtualTraffic":[{"proto":17,"src":"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:64044","dst":"[fd7a:115c:a1e0::]:80","txPkts":1,"txBytes":50}]},"fields":{"recorded":"2023-07-17T15:13:52.933831386Z"}}{"time":1689606834.413,"event":{"nodeId":"nWXoUZ4CNTRL","start":"2023-07-17T15:13:52.938176Z","end":"2023-07-17T15:13:55.88964Z","virtualTraffic":[{"proto":17,"src":"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:53806","dst":"[fd7a:115c:a1e0::]:80","txPkts":1,"txBytes":50}]},"fields":{"recorded":"2023-07-17T15:13:57.935319203Z"}}{"time":1689606839.428,"event":{"nodeId":"nWXoUZ4CNTRL","start":"2023-07-17T15:13:57.967046Z","end":"2023-07-17T15:14:00.890223Z","virtualTraffic":[{"proto":17,"src":"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:51981","dst":"[fd7a:115c:a1e0::]:80","txPkts":1,"txBytes":50}]},"fields":{"recorded":"2023-07-17T15:14:02.936707269Z"}}{"time":1689606844.418,"event":{"nodeId":"nWXoUZ4CNTRL","start":"2023-07-17T15:14:02.944812Z","end":"2023-07-17T15:14:05.891353Z","virtualTraffic":[{"proto":17,"src":"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:57377","dst":"[fd7a:115c:a1e0::]:80","txPkts":1,"txBytes":50}]},"fields":{"recorded":"2023-07-17T15:14:07.937676143Z"}}{"time":1689606849.423,"event":{"nodeId":"nWXoUZ4CNTRL","start":"2023-07-17T15:14:07.955742Z","end":"2023-07-17T15:14:10.890294Z","virtualTraffic":[{"proto":17,"src":"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:50102","dst":"[fd7a:115c:a1e0::]:80","txPkts":1,"txBytes":50}]},"fields":{"recorded":"2023-07-17T15:14:12.935710252Z"}} I've tried many filters with splits and mutates. I do have a support ticket open with the vendor sending the logs and I'm waiting on a response, but figured I'd ask here as well. Any help you can provide me will be hugely appreciated. Let me know any info you may need and I'll happily provide it. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 17, 2023, 3:56pm UTC](https://discuss.elastic.co/t/help-ingesting-data/338580/2 "2023-07-17T15:56:07Z")
> [@Tom\_Dixon](#): > > I do have a support ticket open with the vendor sending the logs and I'm waiting on a response, but figured I'd ask here as well. The best solution would be to fix this in the source so it sends one json per line or at least an array where each item is a json document. But while you wait for a response you can use the following filters to fix your message: ```auto filter { mutate { gsub => ["message", '}{"time', '}||{"time'] } split { field => "message" terminator => "||" } json { source => "message" } } ``` The `gsub` will replace every `}{"time"` in your message with `}||{"time`, the `||` is a string sequence that will be used as an event separator. The `split` filter will split the field `message` using the `||` as a separator, it will create a new event for every item, so if you have this: ```auto {a json event}||{another json event} ``` You will end up with 2 events. And in the last part the `json` filter will parse your message. In the end you will have an output like this: ```auto { "@version" => "1", "time" => 1689606849.423, "@timestamp" => 2023-07-17T15:50:16.471551807Z, "event" => { "nodeId" => "nWXoUZ4CNTRL", "end" => "2023-07-17T15:14:10.890294Z", "virtualTraffic" => [ [0] { "proto" => 17, "dst" => "[fd7a:115c:a1e0::]:80", "src" => "[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:50102", "txBytes" => 50, "txPkts" => 1 } ], "start" => "2023-07-17T15:14:07.955742Z" }, "host" => "lab", "message" => "{\"time\":1689606849.423,\"event\":{\"nodeId\":\"nWXoUZ4CNTRL\",\"start\":\"2023-07-17T15:14:07.955742Z\",\"end\":\"2023-07-17T15:14:10.890294Z\",\"virtualTraffic\":[{\"proto\":17,\"src\":\"[fd7a:115c:a1e0:ab12:4843:cd96:624a:9181]:50102\",\"dst\":\"[fd7a:115c:a1e0::]:80\",\"txPkts\":1,\"txBytes\":50}]},\"fields\":{\"recorded\":\"2023-07-17T15:14:12.935710252Z\"}}", "fields" => { "recorded" => "2023-07-17T15:14:12.935710252Z" } } ``` ---
**Author:** ![Tom\_Dixon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_dixon/32/123545_2.png) [@Tom\_Dixon](https://discuss.elastic.co/u/Tom_Dixon)\ **Post date:** [July 17, 2023, 4:14pm UTC](https://discuss.elastic.co/t/help-ingesting-data/338580/3 "2023-07-17T16:14:42Z")
You are fantastic thank you! This has solved the problem (for now). I'll ask the Vendor to make the changes, I had previously tried to do both the gsub and split, but independently, I didn't think to do them together! I also have to use `additional_codecs =>{"application/json-seq"=>"json_lines" }` in the input to make the split work as well. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 14, 2023, 4:15pm UTC](https://discuss.elastic.co/t/help-ingesting-data/338580/4 "2023-08-14T16:15:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.