Logstash - Parsing fields with duplicate names

Elastic Stack Logstash
1

If I receive a log in that looks like this, how do I deal with the fact that the subfields under "records" are identical? Is there a concept of [records][operationName][0] and [1], for example?
{
"records": [
{
"time": "2023-10-31T11:21:11.0067970Z",
"tenantId": "9ea725b4-0569-4102-9774-55555555",
"operationName": "Publish",
"category": "AdvancedHunting-DeviceProcessEvents",
"Tenant": "DefaultTenant"
},
{
"time": "2023-10-31T11:21:11.0068016Z",
"tenantId": "9ea725b4-0569-4102-9774-55555555",
"operationName": "Publish",
"category": "AdvancedHunting-DeviceProcessEvents",
"Tenant": "DefaultTenant"
},
{
"time": "2023-10-31T11:21:11.0068068Z",
"tenantId": "9ea725b4-0569-4102-9774-55555555",
"operationName": "Publish",
"category": "AdvancedHunting-DeviceProcessEvents",
"Tenant": "DefaultTenant"
},

2

It would be [records][0][operationName] for example, but the best approach when you have an array of events like this is to split on this array and have one event per item.

You would have this in your Logstash configuration:

split {
    field => "records"
}

Then you will end up with one event for each item in the array and can access the fields with [records][operationName] for example.

3

The weird thing is, and I just remembered, that I already do that. Most of my records work fine. I need to add a tag on failure to see if the split is failing.

split {
    field => ["records"]
}
4

@leandrojmp
I don't undestand why some of my logs split fine and others do not. I don't get any split errors.
Could the size of the log impact this? The log I am looking at is 2130 lines long, which should probably be split into about 40 separate records but its happening.

5

Not sure, you didn't share what your pipeline configuration looks like, what is your input or any error logs.

But if Logstash cannot split on the field records it will generate a log error.

Do you have documents where the field records wasn't splitted on your Elasticsearch? If so, please please share the entire json document that you can get on Kibana Discover.

6

My bad, split is working. All is well, thanks.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.