# Extract certain fields from JSON

**URL:** <https://discuss.elastic.co/t/extract-certain-fields-from-json/335487>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 8, 2023, 1:10am UTC](https://discuss.elastic.co/t/extract-certain-fields-from-json/335487 "2023-06-08T01:10:35Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\
**Post date:** [June 8, 2023, 1:10am UTC](https://discuss.elastic.co/t/extract-certain-fields-from-json/335487/1 "2023-06-08T01:10:36Z")

</div>

Hi,

I am forwarding filebeat logs to an ES, and I would like to only extract certain fields in the JSON `message` and write them to ES.

For example, if I have the JSON message below:

```auto
{
   "field1": "info",
   "field2": "info2",
   "field3": {
      "field3a": {
         "field3b": {
            "field4": 123,
            "field5": "abc",
            "field6": {
               "field7": 456
            }
         }
      }
   }
}

```

On Kibana Discover, this whole JSON object is stored as `message` in a string. Is it possible to only extract and store `field4`, `field5`, and `field7`?

I've tried the `decode_json_fields` processor, but it seems to extract the entire JSON object and all its fields, and this "explosion" of data caused the data size to exceed some limit and the message was not sent to ES instead.

Thank you.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 8, 2023, 2:09am UTC](https://discuss.elastic.co/t/extract-certain-fields-from-json/335487/2 "2023-06-08T02:09:14Z")

</div>

Curious how many fields are in this object?

But no there is no direct "selector" there is `depth`.

Perhaps You could decode the whole JSON and then drop the unneeded field with a `drop_fields` with some conditions.

Just a thought.... But this could also be memory/ CPU intensive.

---

<div class="post-metadata">

**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\
**Post date:** [June 8, 2023, 9:00am UTC](https://discuss.elastic.co/t/extract-certain-fields-from-json/335487/3 "2023-06-08T09:00:05Z")

</div>

There are probably more than 30 fields int this object.

I can try the `drop_fields`, do you know if I drop, say, `field3b`, will it drop all its children (`field4` to `field7`), so I don't have to specify each field?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 8, 2023, 2:45pm UTC](https://discuss.elastic.co/t/extract-certain-fields-from-json/335487/4 "2023-06-08T14:45:48Z")

</div>

Ohhh I was thinking you meant 100s of fields.... so I am a bit confused by this statement (I guess you have some other mapping issues / many fields already)

> [@hjazz6](#):
>
> and this "explosion" of data caused the data size to exceed some limit and the message was not sent to ES instead.

Anyways yes when you drop the parent field it should drop all the children...

Give it a try, you can also do this with an ingest pipeline in Elasticsearch so the logic is centralized.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 8, 2023, 3:33pm UTC](https://discuss.elastic.co/t/extract-certain-fields-from-json/335487/5 "2023-06-08T15:33:48Z")

</div>

I think that in this case the best solution would be to use a Ingest Pipeline is suggested by @stephenb.

You would need to use the [json processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html) with a custom field in the `target_field` option.

Then you could use a couple of [rename processors](https://www.elastic.co/guide/en/elasticsearch/reference/current/rename-processor.html) to rename the desired fields and after that you would remove the top-level field.

For example, if you use `_tmp` as the target field you would have something like this after the json processor:

```auto
{
   "_tmp": {
      "field1": "info",
      "field2": "info2",
      "field3": {
         "field3a": {
            "field3b": {
               "field4": 123,
               "field5": "abc",
               "field6": {
                  "field7": 456
               }
            }
         }
      }
   }
}

```

Since you want just `field4`, `field5` and `field7`, you could use a rename field on them so you would rename `_tmp.field3.field3a.field3b.field4` to `field4` for example.

Then after the renames you would remove the entire `_tmp` field.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2023, 5:34pm UTC](https://discuss.elastic.co/t/extract-certain-fields-from-json/335487/6 "2023-07-06T17:34:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.