# How can I add Json-Data to historic index

**URL:** <https://discuss.elastic.co/t/how-can-i-add-json-data-to-historic-index/322457>\
**Category:** Elasticsearch\
**Created:** [January 4, 2023, 11:41am UTC](https://discuss.elastic.co/t/how-can-i-add-json-data-to-historic-index/322457 "2023-01-04T11:41:55Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Steven\_Tschache](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steven_tschache/32/115540_2.png) [@Steven\_Tschache](https://discuss.elastic.co/u/Steven_Tschache)\
**Post date:** [January 4, 2023, 11:41am UTC](https://discuss.elastic.co/t/how-can-i-add-json-data-to-historic-index/322457/1 "2023-01-04T11:41:55Z")

</div>

We currently are importing data using Logstash. One of the fields ("request") is a JSON stored as a string. We now require a section of this stored JSON as fields in the searchable index. I have updated Logstash filter using

```auto
filter {
    json {
        source => "request"
        target => "[@metadata][request_json]"
    }
    if [@metadata][request_json][merchant] {
        # in the Request, pull out the Merchant-ID
        mutate {
            add_field => {
                "merchant_id" => "%{[@metadata][request_json][merchant][id]}"
                "merchant_name" => "%{[@metadata][request_json][merchant][name]}"
            }
        }
    }
}

```

Which works great for new data.

How can I update the indices for the historic data? I'm using Elasticsearch, Logstash and Kibana 8.5.3

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 4, 2023, 12:57pm UTC](https://discuss.elastic.co/t/how-can-i-add-json-data-to-historic-index/322457/2 "2023-01-04T12:57:56Z")

</div>

You may be able to run an [update by query with an ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/docs-update-by-query.html#docs-update-by-query-api-query-params) containing a [JSON processor](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/json-processor.html).

---

<div class="post-metadata">

**Author:** ![Steven\_Tschache](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steven_tschache/32/115540_2.png) [@Steven\_Tschache](https://discuss.elastic.co/u/Steven_Tschache)\
**Post date:** [January 5, 2023, 1:47pm UTC](https://discuss.elastic.co/t/how-can-i-add-json-data-to-historic-index/322457/3 "2023-01-05T13:47:32Z")

</div>

Thx for the advice. So, I'm looking at this, but struggling how to implement the "painless" script for this, since painless has limited support for JSON. I've started off OK with

```auto
[
  {
    "json": {
      "field": "request",
      "target_field": "request_json"
    }
  },
  "script": {
    "inline": "def now_what;",
    "lang": "painless"
  },
  {
    "remove": {
      "field": "request_json"
    }
  }
]

```

But the bit in the middle is where I'm struggling. Never did "painless" and not much help out there in Google.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 5, 2023, 1:52pm UTC](https://discuss.elastic.co/t/how-can-i-add-json-data-to-historic-index/322457/4 "2023-01-05T13:52:03Z")

</div>

Why not use a number of [set processors](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/set-processor.html) instead of painless?

---

<div class="post-metadata">

**Author:** ![Steven\_Tschache](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steven_tschache/32/115540_2.png) [@Steven\_Tschache](https://discuss.elastic.co/u/Steven_Tschache)\
**Post date:** [January 5, 2023, 2:39pm UTC](https://discuss.elastic.co/t/how-can-i-add-json-data-to-historic-index/322457/5 "2023-01-05T14:39:23Z")

</div>

Because YOU are magic! LOL, thank you so so much!

```auto
[
  {
    "json": {
      "field": "request",
      "target_field": "request_json"
    }
  },
  {
    "set": {
      "field": "merchant_name",
      "copy_from": "request_json.merchant.name",
      "ignore_empty_value": true
    }
  },
  {
    "set": {
      "field": "merchant_name",
      "copy_from": "request_json.data.application.merchant.name",
      "override": false,
      "ignore_empty_value": true
    }
  },
  {
    "set": {
      "field": "merchant_id",
      "copy_from": "request_json.merchant.id",
      "ignore_empty_value": true
    }
  },
  {
    "set": {
      "field": "merchant_id",
      "copy_from": "request_json.data.application.merchant.id",
      "override": false,
      "ignore_empty_value": true
    }
  },
  {
    "remove": {
      "field": "request_json"
    }
  }
]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 2, 2023, 2:39pm UTC](https://discuss.elastic.co/t/how-can-i-add-json-data-to-historic-index/322457/6 "2023-02-02T14:39:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.