# Parse/process json array with filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 19, 2024, 3:45pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455 "2024-01-19T15:45:50Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![andrejcoliveira](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@andrejcoliveira](https://discuss.elastic.co/u/andrejcoliveira)\ **Post date:** [January 19, 2024, 3:45pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/1 "2024-01-19T15:45:50Z")
Hi everyone, at my company we're trying to load our Cucumber logs to Elastic. Currently i want to check if it is possible to do it just using filebeat. The main problem to execute this task is that our logs are single line json object arrays like: ```auto [{json object}, {json_object}] ``` At this moment, my configuration only creates one event with all json code in "message" field. Thanks in advance for your help ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 22, 2024, 6:19pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/2 "2024-01-22T18:19:26Z")
Are you hoping that each json object becomes its own document? If so, I don't believe that Filebeat can open a JSON array and split the objects into multiple documents. You'd like want to preprocess the log or use something like Logstash to split it into multiple documents. ---
**Author:** ![andrejcoliveira](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@andrejcoliveira](https://discuss.elastic.co/u/andrejcoliveira)\ **Post date:** [January 23, 2024, 10:55am UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/3 "2024-01-23T10:55:20Z")
I would like have each json object from the array as an event. In the same document but with different identifiers (timestamp, id, etc). I've tested to manually put one object per line in the log file and it worked as expected. For one log, it created distinct events in the same document. I want to check if Filebeat can identify the distinct objects if the log have just one line ( + newline) ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 23, 2024, 2:06pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/4 "2024-01-23T14:06:18Z")
Yeah sorry, I was using "document" to mean event in elasticsearch. You may be able to play with the delimiter settings on the log input but it's not JSON aware so you run the risk of a couple of issues. But I'm not aware of a way you could have a json array on one line and have filebeat split each object into multiple event for Elasticsearch. ---
**Author:** ![andrejcoliveira](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@andrejcoliveira](https://discuss.elastic.co/u/andrejcoliveira)\ **Post date:** [January 23, 2024, 3:27pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/5 "2024-01-23T15:27:39Z")
Do you think that logstash would be better to do this type of processing? ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 23, 2024, 3:59pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/6 "2024-01-23T15:59:05Z")
Yes, this is relatively easy to do in logstash. Depending on the format of the file you may be able to have Filebeat read the file and use a logstash pipeline to do the splitting. ---
**Author:** ![andrejcoliveira](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@andrejcoliveira](https://discuss.elastic.co/u/andrejcoliveira)\ **Post date:** [January 23, 2024, 4:17pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/7 "2024-01-23T16:17:49Z")
Can you give me an example of how to do it? I don´t know if you´re familiar with Cucumber log format. But basically, each object from the array represents a feature, each feature have several scenarios in the "elements" field. I want to know if it is possible to create individual events to each scenario so it isn't direct, it´s an array inside another array. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 23, 2024, 4:22pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/8 "2024-01-23T16:22:56Z")
I am not familiar with a Cucumber log but here is the documentation for the split filter in Logstash that you'd likely use for this [Split filter plugin | Logstash Reference [8.12] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html) ```auto { field1: ..., results: [ { result ... }, { result ... }, { result ... }, ... ] } ``` The split filter can be used on the above data to create separate events for each value of results field ```auto filter { split { field => "results" } } ``` Logstash would then split the one event into three independent events to write to Elasticsearch ```auto event 1: { result ... } event 2: { result ... } event 3: { result ... } ``` Would you be able to share a sanitized cucumber log in this thread? ---
**Author:** ![andrejcoliveira](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@andrejcoliveira](https://discuss.elastic.co/u/andrejcoliveira)\ **Post date:** [January 23, 2024, 4:32pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/9 "2024-01-23T16:32:54Z")
Sure: ```auto [ { "line": 2, "elements": [ { "line": 4, "name": "", "description": "", "type": "", "keyword": "", "steps": [ { "result": { "duration": "", "status": "" }, "line": 5, "name": "", "match": { "arguments": [ { "val": "", "offset": 2 } ], "location": "" }, "keyword": " " } ] }, { "start_timestamp": "", "before": [ { "result": { "duration": 561992900, "status": "" }, "match": { "location": "" } } ], "line": 8, "name": "", "description": "", "id": "", "after": [ { "result": { "duration": 1327737100, "status": "" }, "match": { "location": "" } } ], "type": "", "keyword": "", "steps": [ { "result": { "duration": 2546265500, "status": "" }, "line": 9, "name": "", "match": { "location": "" }, "keyword": " " }, { "embeddings": [ { "data": "", "mime_type": "", "name": "" } ], "result": { "duration": 8510044800, "status": "" }, "line": 10, "name": "", "match": { "arguments": [ { "val": "", "offset": 25 }, { "val": "", "offset": 44 }, { "val": "", "offset": 52 } ], "location": "" }, "keyword": " " }, { "result": { "duration": 24000600, "status": "" }, "line": 11, "name": "", "match": { "location": "" }, "keyword": " " } ], "tags": [ { "name": "" }, { "name": "" }, { "name": "" }, { "name": "" }, { "name": "" } ] }, { "line": 4, "name": "", "description": "", "type": "", "keyword": "", "steps": [ { "result": { "duration": 853000700, "status": "" }, "line": 5, "name": "", "match": { "arguments": [ { "val": "", "offset": 2 } ], "location": "" }, "keyword": "" } ] }, { "start_timestamp": "", "before": [ { "result": { "duration": 1000500, "status": "" }, "match": { "location": "" } } ], "line": 15, "name": "", "description": "", "id": "", "after": [ { "result": { "duration": 128004000, "status": "" }, "match": { "location": "" } } ], "type": "", "keyword": "", "steps": [ { "result": { "duration": 1852995400, "status": "" }, "line": 16, "name": "", "match": { "location": "" }, "keyword": " " }, { "embeddings": [ { "data": "", "mime_type": "", "name": "" } ], "result": { "duration": 3733347800, "status": "" }, "line": 17, "name": "", "match": { "arguments": [ { "val": "", "offset": 25 }, { "val": "", "offset": 44 }, { "val": "", "offset": 52 } ], "location": "" }, "keyword": " " }, { "result": { "duration": 9997000, "status": "" }, "line": 18, "name": "", "match": { "location": "" }, "keyword": " " } ], "tags": [ { "name": "" }, { "name": "" }, { "name": "" }, { "name": "" }, { "name": "" } ] } ], "name": "", "description": "", "id": "", "keyword": "", "uri": "", "tags": [ { "name": "", "type": "", "location": { "line": 1, "column": 1 } }, { "name": "", "type": "", "location": { "line": 1, "column": 8 } } ] } ] ``` Sorry if it is too long but my json log has more than 17000 lines, this is just a sample of one feature. What i want is one event for each element of each json object in the array. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [January 23, 2024, 4:42pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/10 "2024-01-23T16:42:38Z")
And the actual format of the file is that this is all on one line without any line breaks? or is it pretty printed like this? or something in between? Could you maybe share a screenshot of the file with word wrap off, in its original format, to give me a better idea? ---
**Author:** ![andrejcoliveira](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@andrejcoliveira](https://discuss.elastic.co/u/andrejcoliveira)\ **Post date:** [January 23, 2024, 4:48pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/11 "2024-01-23T16:48:44Z")
```auto [{ "line": 2, "elements": [ { "line": 4, "name": "", "description": "", "type": "", "keyword": "", "steps": [ { "result": { "duration": "", "status": "" }, "line": 5, "name": "", "match": { "arguments": [ { "val": "", "offset": 2 }], "location": "" }, "keyword": " " } ] }, { "start_timestamp": "", "before": [{ "result": { "duration": 561992900, "status": "" }, "match": { "location": "" } }], "line": 8, "name": "", "description": "", "id": "", "after": [{ "result": { "duration": 1327737100, "status": "" }, "match": { "location": "" } }], "type": "", "keyword": "", "steps": [{ "result": { "duration": 2546265500, "status": "" }, "line": 9, "name": "", "match": { "location": "" }, "keyword": " " }, { "embeddings": [ { "data": "", "mime_type": "", "name": "" }], "result": { "duration": 8510044800, "status": "" }, "line": 10, "name": "", "match": { "arguments": [{ "val": "", "offset": 25 }, { "val": "", "offset": 44 }, { "val": "", "offset": 52 }], "location": "" }, "keyword": " " }, { "result": { "duration": 24000600, "status": "" }, "line": 11, "name": "", "match": { "location": "" }, "keyword": " " } ], "tags": [{ "name": "" }, { "name": "" }, { "name": "" }, { "name": "" }, { "name": "" }] }, { "line": 4, "name": "", "description": "", "type": "", "keyword": "", "steps": [{ "result": { "duration": 853000700, "status": "" }, "line": 5, "name": "", "match": { "arguments": [ { "val": "", "offset": 2 }], "location": "" }, "keyword": "" } ] }, { "start_timestamp": "", "before": [{ "result": { "duration": 1000500, "status": "" }, "match": { "location": "" } }], "line": 15, "name": "", "description": "", "id": "", "after": [{ "result": { "duration": 128004000, "status": "" }, "match": { "location": "" } }], "type": "", "keyword": "", "steps": [{ "result": { "duration": 1852995400, "status": "" }, "line": 16, "name": "", "match": { "location": "" }, "keyword": " " }, { "embeddings": [ { "data": "", "mime_type": "", "name": "" }], "result": { "duration": 3733347800, "status": "" }, "line": 17, "name": "", "match": { "arguments": [{ "val": "", "offset": 25 }, { "val": "", "offset": 44 }, { "val": "", "offset": 52 }], "location": "" }, "keyword": " " }, { "result": { "duration": 9997000, "status": "" }, "line": 18, "name": "", "match": { "location": "" }, "keyword": " " } ], "tags": [{ "name": "" }, { "name": "" }, { "name": "" }, { "name": "" }, { "name": "" }] } ], "name": "", "description": "", "id": "", "keyword": "", "uri": "", "tags": [{ "name": "", "type": "", "location": { "line": 1, "column": 1 } }, { "name": "", "type": "", "location": { "line": 1, "column": 8 } }] } ] ``` This is a sample of the log in its original form. As i said before, one line of information and one blank line. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 20, 2024, 6:48pm UTC](https://discuss.elastic.co/t/parse-process-json-array-with-filebeat/351455/12 "2024-02-20T18:48:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.