# Split filter and add\_field encoding object to a JSON string

**URL:** <https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590>\
**Category:** Logstash\
**Created:** [July 4, 2023, 8:46pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590 "2023-07-04T20:46:33Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![MrOg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrog/32/123011_2.png) [@MrOg](https://discuss.elastic.co/u/MrOg)\
**Post date:** [July 4, 2023, 8:46pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590/1 "2023-07-04T20:46:33Z")

</div>

Hi, I have such a set of filters

```auto
filter {
    json {
        source => "[sql_data][response]"
    }
    split {
        field => "docs"
        add_field => {
            "id" => "%{[docs][id]}"
            "names" => "%{[docs][names]}"
        }
        remove_field => [
            "sql_data",
            "total",
            "limit",
            "page",
            "pages"
        ]
    }
}

```

And it gives me a following result

```auto
...
{
	"docs" => {
		"id" => 5271175,
		"names" => [
			[0] {
				"name" => "Element A"
				"type" => "F"
			},
			[1] {
				"name" => "Element B"
				"type" => "F"
			}
		]
	},
	"@timestamp" => 2023-07-04T20:36:17.268796400Z,
	"names" => "{name=Element A, type=F},{name=Element B, type=F}",
	"@version" => "1",
	"id" => "5271175"
}
...

```

Can someone help me to understand, please! If there is any option to move `[docs][names]` to `[names]` without converting/encoding it to a JSON string, or should I move it like that and restore it back to JSON somehow?

Thanks!

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 4, 2023, 8:59pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590/2 "2023-07-04T20:59:52Z")

</div>

Hello,

It is not clear what is the issue, what is not working and what is the result you expect.

Also, you need to share a sample of your message before the `split` filters so it will be possible to replicate your pipeline.

---

<div class="post-metadata">

**Author:** ![MrOg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrog/32/123011_2.png) [@MrOg](https://discuss.elastic.co/u/MrOg)\
**Post date:** [July 4, 2023, 9:17pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590/3 "2023-07-04T21:17:41Z")

</div>

Ok, my current data structure is following

```auto
2023-07-05 00:11:56 [2023-07-04T21:11:56,855][INFO][logstash.inputs.jdbc][main][bbba3150f9c946cd7407c40277f99ddb5c77b481b669b359f5f6fd68c1b732b5] (0.001225s) SELECT id, endpoint, method, response, response_code, created_at, processed_at FROM request_queue WHERE id > 0 ORDER BY id ASC LIMIT 1
2023-07-05 00:11:56 {
2023-07-05 00:11:56 "total" => 944195,
2023-07-05 00:11:56 "limit" => 2,
2023-07-05 00:11:56 "@version" => "1",
2023-07-05 00:11:56 "page" => 1,
2023-07-05 00:11:56 "pages" => 472098,
2023-07-05 00:11:56 "docs" => [
2023-07-05 00:11:56 [0] {
2023-07-05 00:11:56 "id" => 5309714,
2023-07-05 00:11:56 "names" => [
2023-07-05 00:11:56 [0] {
2023-07-05 00:11:56 "name" => "Untitled the Pope's Exorcist sequel",
2023-07-05 00:11:56 "language" => nil
2023-07-05 00:11:56 }
2023-07-05 00:11:56 ]
2023-07-05 00:11:56 },
2023-07-05 00:11:56 [1] {
2023-07-05 00:11:56 "id" => 5307599,
2023-07-05 00:11:56 "names" => [
2023-07-05 00:11:56 [0] {
2023-07-05 00:11:56 "name" => "Masters of the Universe: Revolution",
2023-07-05 00:11:56 "language" => nil
2023-07-05 00:11:56 }
2023-07-05 00:11:56 ]
2023-07-05 00:11:56 }
2023-07-05 00:11:56 ],
2023-07-05 00:11:56 "@timestamp" => 2023-07-04T21:11:56.858622200Z,
2023-07-05 00:11:56 "sql_data" => {
2023-07-05 00:11:56 "endpoint" => "v1.3/movie",
2023-07-05 00:11:56 "response_code" => 200,
2023-07-05 00:11:56 "method" => "GET",
2023-07-05 00:11:56 "processed_at" => 2023-07-04T21:10:00.000Z,
2023-07-05 00:11:56 "created_at" => 2023-07-04T21:10:00.000Z,
2023-07-05 00:11:56 "id" => 1,
2023-07-05 00:11:56 "response" => #<Sequel::SQL::Blob:0x15f8 bytes=235 start="{\"docs\":[{" end="s\":472098}">
2023-07-05 00:11:56 }
2023-07-05 00:11:56 }

```

Content of `request_queue.response`

```auto
{
    "docs": [
        {
            "id": 5309714,
            "names": [
                {
                    "name": "Untitled the Pope's Exorcist sequel",
                    "language": null
                }
            ]
        },
        {
            "id": 5307599,
            "names": [
                {
                    "name": "Masters of the Universe: Revolution",
                    "language": null
                }
            ]
        }
    ],
    "total": 944195,
    "limit": 2,
    "page": 1,
    "pages": 472098
}

```

I want to save each object from `docs` as an index document.

My current conf

```auto
input {
    jdbc {
        jdbc_driver_library => "${LOGSTASH_JDBC_DRIVER_JAR_LOCATION}"
        jdbc_driver_class => "${LOGSTASH_JDBC_DRIVER}"
        jdbc_connection_string => "${LOGSTASH_JDBC_URL}"
        jdbc_user => "${LOGSTASH_JDBC_USERNAME}"
        jdbc_password => "${LOGSTASH_JDBC_PASSWORD}"
        schedule => "*/1 * * * * *"
        statement => "SELECT id, endpoint, method, response, response_code, created_at, processed_at FROM request_queue WHERE id > 0 ORDER BY id ASC LIMIT 1"
        target => "sql_data"
    }
}

filter {
    json {
        source => "[sql_data][response]"
    }
    split {
        field => "docs"
        add_field => {
            "id" => "%{[docs][id]}"
            "names" => "%{[docs][names]}"
        }
        remove_field => [
            "sql_data",
            "total",
            "limit",
            "page",
            "pages"
        ]
    }
}

```

After

```auto
        add_field => {
            "id" => "%{[docs][id]}"
            "names" => "%{[docs][names]}"
        }

```

the field `names` being converted to JSON string.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 4, 2023, 9:30pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590/4 "2023-07-04T21:30:30Z")

</div>

And what does your `docs` field looks like? You didn't share how it looks like before the `split` filter.

Remove the `split` filter from your pipeline and share a sample of your document, without it is pretty hard to replicate your issue.

EDIT: Nevermind, the content of the docs is on the `request_queue.response` right?

I will try to simulate your pipeline.

---

<div class="post-metadata">

**Author:** ![MrOg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrog/32/123011_2.png) [@MrOg](https://discuss.elastic.co/u/MrOg)\
**Post date:** [July 4, 2023, 9:33pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590/5 "2023-07-04T21:33:00Z")

</div>

> EDIT: Nevermind, the content of the docs is on the `request_queue.response` right?

Yes

---

<div class="post-metadata">

**Author:** ![MrOg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrog/32/123011_2.png) [@MrOg](https://discuss.elastic.co/u/MrOg)\
**Post date:** [July 18, 2023, 7:50pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590/6 "2023-07-18T19:50:07Z")

</div>

@leandrojmp Hi!  
Sorry to bother you, any news 🥺

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 20, 2023, 1:36pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590/7 "2023-07-20T13:36:59Z")

</div>

Hello,

Just replicated it here and there is no issue with the filters, but to get your desired output you will need to make a little change.

Your split filter looks like this:

```auto
    split {
        field => "docs"
        add_field => {
            "id" => "%{[docs][id]}"
            "names" => "%{[docs][names]}"
        }
    }

```

It will create documents like this:

```auto
{
          "docs" => {
           "id" => 5309714,
        "names" => [
            [0] {
                    "name" => "Untitled the Pope's Exorcist sequel",
                "language" => nil
            }
        ]
    },
         "total" => 944195,
         "limit" => 2,
         "pages" => 472098,
    "@timestamp" => 2023-07-20T13:22:01.219353969Z,
          "host" => "lab",
         "names" => "{name=Untitled the Pope's Exorcist sequel, language=null}",
      "@version" => "1",
          "page" => 1,
            "id" => "5309714"
}

```

There are two issues here, first is that your `docs.names` field is also an array, maybe you should also split on this field, but this depends on your use case.

Second, when you use `add_field` and the source field is an json object, the target field will not be a json object, it will receive the json as string, which is the output you are getting.

```auto
"names" => "{name=Untitled the Pope's Exorcist sequel, language=null}"

```

If you want the `names` field to be a json object you need to user another mutate with a rename or a copy after the split filter.

```auto
    mutate {
        copy => {
            "[docs][names]" => "names"
        }
    }

```

---

<div class="post-metadata">

**Author:** ![MrOg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mrog/32/123011_2.png) [@MrOg](https://discuss.elastic.co/u/MrOg)\
**Post date:** [July 27, 2023, 4:46pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590/8 "2023-07-27T16:46:34Z")

</div>

Thank you soo much Sir.  
I'm in a trip atm. Will check it at the weekend.  
Thank you soo much for your help and time!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 24, 2023, 4:46pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590/9 "2023-08-24T16:46:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.