# Splitting Logstash message

**URL:** <https://discuss.elastic.co/t/splitting-logstash-message/345597>\
**Category:** Logstash\
**Created:** [October 23, 2023, 8:31pm UTC](https://discuss.elastic.co/t/splitting-logstash-message/345597 "2023-10-23T20:31:31Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![joecarter](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@joecarter](https://discuss.elastic.co/u/joecarter)\
**Post date:** [October 23, 2023, 8:31pm UTC](https://discuss.elastic.co/t/splitting-logstash-message/345597/1 "2023-10-23T20:31:31Z")

</div>

I am pulling events from an Azure Event Hub, but some of the events are being grouped into a single message containing an array of "records", which I want to be processed as individual messages. The format is:

```auto
{
timestamp,
message
  {
  records: [
    {event1},
    {event2}
  ]
  }
}

```

I've used the split filter to split the events into separate messages, but this strips the 'timestamp' and 'message' fields, i.e.

```auto
{
record
  {
  event1
  }
},
{
record
  {
  event2
  }
}

```

I'd like to preserve the message field of each message, i.e.

```auto
{
timestamp,
message
  {
  records: [
    {event1}
  ]
  }
},
{
timestamp,
message
  {
  records: [
    {event2}
  ]
  }
}

```

The filter I currently have is:

```auto
filter {
  json {
    source => "message"
  }
  split {
    field => ["records"]
    remove_field => ["message"]
  }
  mutate {
        add_field => {"@timestamp" => "%{@timestamp}"}
  }
}

```

But how can I split the records whilst retaining the message structure?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 23, 2023, 8:39pm UTC](https://discuss.elastic.co/t/splitting-logstash-message/345597/2 "2023-10-23T20:39:10Z")

</div>

> [@joecarter](#):
>
> I've used the split filter to split the events into separate messages, but this strips the 'timestamp' and 'message' fields

The `strip` filter will create a new event for every item in the array, but will keep every other field in the event, except the one being splitted.

It wont remove the `timestamp` field in the example you shared. Can you share an output where the timestamp field is removed after the split?

---

<div class="post-metadata">

**Author:** ![joecarter](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@joecarter](https://discuss.elastic.co/u/joecarter)\
**Post date:** [October 24, 2023, 9:32am UTC](https://discuss.elastic.co/t/splitting-logstash-message/345597/3 "2023-10-24T09:32:24Z")

</div>

I assume you meant 'split', not 'strip'.

Before using split, the 'records' array is contained _in_ the message field, e.g.

```auto
{"@timestamp":"2023-10-23T10:17:25.436Z","@version":"1","message":"{"records": [{ "attribute": "value1"},{"attribute": "value2"}]}

```

Adding the split filter, the records array is correctly split into two separate events, but each member the array is added to a new field called 'records', rather than replacing the contents of the message attribute (which is left as is), e.g.

```auto
{"@timestamp":"2023-10-23T10:42:48.294Z","@version":"1","records":{"attribute":"value1"},"message":"{"records": [{"attribute": "value1"},{"attribute": "value2"}]}

{"@timestamp":"2023-10-23T10:42:48.294Z","@version":"1","records":{"attribute":"value2"},"message":"{"records": [{"attribute": "value1"},{"attribute": "value2"}]}

```

I am hoping to preserve the event structure, whereby the single value record array is contained in the message field.

On closer inspection, the timestamp field is not removed - it was just moved to the end of the event when I deleted the message field.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 24, 2023, 11:59am UTC](https://discuss.elastic.co/t/splitting-logstash-message/345597/4 "2023-10-24T11:59:53Z")

</div>

> [@joecarter](#):
>
> I assume you meant 'split', not 'strip'.

Yeah, I meant `split`.

> [@joecarter](#):
>
> the records array is correctly split into two separate events, but each member the array is added to a new field called 'records', rather than replacing the contents of the message attribute (which is left as is)

This is the expected behavior, the split filter per default will use the same field name.

Another thing is that the `message` field is kind of a _special_ field name, this is the field that have the original message that logstash received, in version 8 with ecs compatibility enabled this field is renamed to `event.original`.

For example, your original message is something like this:

```auto
{"records": [{ "attribute": "value1"},{"attribute": "value2"}]}

```

So when this message enters your logstash pipeline, behind the scenes you will have something like this:

```auto
{ "message": {"records": [{ "attribute": "value1"},{"attribute": "value2"}]} }

```

To parse this message you would need to have a `json` filter with the `message` field as a source.

```auto
json {
    source => "message"
}

```

This will parse the content of the `message` field and put them on the root of the document as no _target_ was specified, the `message` field will not be changed or removed unless you explicitly remove it.

To arrive on the output example you give your pipeline filter block should look like this:

```auto
filter {
    json {
        source => "message"
    }
    split {
        field => ["[records]"]
    }
}

```

If you want to have the content of the `records` field inside the `message` field, like `message: {"attribute": "value1"}`, you need to remove the original message field and rename the records field before the split.

The following fitlers will give that:

```auto
filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
    mutate {
        rename => {
            "records" => "message"
        }
    }
    split {
        field => ["[message]"]
    }
}

```

The result of this would be something like this:

```auto
{"host":"lab","message":{"attribute":"value1"},"@version":"1","@timestamp":"2023-10-24T11:58:43.867652396Z"}
{"host":"lab","message":{"attribute":"value2"},"@version":"1","@timestamp":"2023-10-24T11:58:43.867652396Z"}

```

---

<div class="post-metadata">

**Author:** ![joecarter](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@joecarter](https://discuss.elastic.co/u/joecarter)\
**Post date:** [October 24, 2023, 3:24pm UTC](https://discuss.elastic.co/t/splitting-logstash-message/345597/5 "2023-10-24T15:24:37Z")

</div>

Thanks for the explanation. Is it not possible then to maintain the split 'records' array in the message field, thus retaining the original structure, i.e.

```auto
{"@timestamp":"2023-10-23T10:17:25.436Z","@version":"1","message":"{"records": [{ "attribute": "value1"}]}

{"@timestamp":"2023-10-23T10:19:22.726Z","@version":"1","message":"{"records": [{ "attribute": "value2"}]}

```

Ideally I'm trying to split the records 'array' without updating the Elastic mappings/indexes.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 24, 2023, 3:37pm UTC](https://discuss.elastic.co/t/splitting-logstash-message/345597/6 "2023-10-24T15:37:38Z")

</div>

> [@joecarter](#):
>
> Is it not possible then to maintain the split 'records' array in the message field

It is, just rename the field to `[message][records]` instead of just `message`.

```auto
    mutate {
        rename => {
            "records" => "[message][records]"
        }
    }

```

Then in your output you will have this:

```auto
{"host":"lab","@version":"1","@timestamp":"2023-10-24T15:33:11.511695899Z","message":{"records":{"attribute":"value1"}}}
{"host":"lab","@version":"1","@timestamp":"2023-10-24T15:33:11.511695899Z","message":{"records":{"attribute":"value2"}}}

```

The `records` won't be an array of a single item, but this doesn't matter as there is no dedicate array data type, so this makes no difference to the mapping.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2023, 3:37pm UTC](https://discuss.elastic.co/t/splitting-logstash-message/345597/7 "2023-11-21T15:37:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.