# Watcher with Index Action - Not writing data properly

**URL:** <https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714>\
**Category:** Kibana\
**Created:** [May 14, 2020, 9:22pm UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714 "2020-05-14T21:22:55Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 14, 2020, 9:22pm UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/1 "2020-05-14T21:22:55Z")

</div>

Hello, I am trying to create a watcher that scans a log index and once it finds a search term, writes the JSON payload to a new index with an Index action.

I am seeing that the data that is reaching the Index this way is different in format than when we hit the elastic endpoint directly through Postman with a PUT request .

The data is properly parsed and inserted when done from postman as shown in snapshot below.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/7/57d1d19c296c6c68b9eb13d072fc76a4893f4a2a.png)

The JSON body is under the \_source at parent level and hence the parsing is happening .

When same thing happens through Index action of a watcher , this is how it shows up in Kibana.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/9/f9be1d971b7e641c058d38b0bdff151c36709db7.png)

As you can see , the actual json payload is not at parent \_source but inside hits-\>hits-\>\_source.

How can I update my watcher script to write the json payload at the parent \_source and not inside the hits-\>hits-\>\_source

Any help is greatly appreciated. Thank you so much.

Here is my watcher script :

```auto
{
  "trigger": {
    "schedule": {
        "hourly" : { "minute" : [0, 5, 10, 15 ,20, 25, 30, 35 ,40, 45, 50, 55] }  
      }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "iks-dev*"
        ],
        "types": [],
        "body": {
          "size": 1000,
          "query": {
            "bool": {
              "must": [
                {
                  "match_all": {}
                },
                {
                  "match_phrase": {
                    "app_className": {
                      "query": "RequestAndResponseLogger"
                    }
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m/m"
                    }
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          },
          "_source": [
            "app_message"
          ],
          "sort": [
            {
              "@timestamp": {
                "order": "desc"
              }
            }
          ]
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "to": [
          "example@gmail.com"
        ],
        "subject": "Encountered {{ctx.payload.hits.total}} stats(Environment)!",
        "body": {
          "text": " Report \n\n {{#ctx.payload.hits.hits}}{{_source.app_recordLoc}}\n\n{{_source.app_message}}\n\n{{/ctx.payload.hits.hits}} "
        }
      }
    },
	"index_payload" : { 
	  "index" : {
      "index" : "tnr-doc-store-dev", 
      "doc_type" : "_doc" 
    }
  }
  },
  "throttle_period_in_millis": 900000
}

```

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [May 14, 2020, 10:48pm UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/2 "2020-05-14T22:48:49Z")

</div>

Hello @Bhavani_Prasad

It is normal as the `index` action will take the whole content of the response and index it.

If you want to index the hits you matched on the input search, you have to perform some preprocessing using a transform.

Prior to run it again with the code below, please delete the index `tnr-doc-store-dev` as you might end up in a mapping conflict.

```auto
"actions": {
      "index_payload": {
        "transform": {
          "script": {
            "source": "['_doc': ctx.payload.hits.hits.stream().map(h -> h._source).collect(Collectors.toList())];",
            "lang": "painless"
          }
        },
        "index": {
          "index": "tnr-doc-store-dev",
          "doc_type": "_doc"
        }
      }
    }

```

---

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 14, 2020, 11:45pm UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/3 "2020-05-14T23:45:03Z")

</div>

> [@Luca\_Belluccini](#):
>
> "index\_payload": { "transform": { "script": { "source": "['\_doc': ctx.payload.hits.hits.stream().map(h -\> h.\_source).collect(Collectors.toList())];", "lang": "painless" } }, "index": { "index": "tnr-doc-store-dev", "doc\_type": "\_doc" } }

Thank you so much. I tried with the script below and it gave me below error .Also I only need the contents of \_source.app\_message hence i added that in the transform script

```auto
"index_payload": {
        "transform": {
          "script": {
            "source": "['_doc': ctx.payload.hits.hits.stream().map(h -> h._source.app_message).collect(Collectors.toList())];",
            "lang": "painless"
          }
        },
        "index": {
          "index": "tnr-doc-store-qa",
          "doc_type": "_doc"
        }
   }

```

My search is returning multiple rows and all rows need to be added to index.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d9fd5f8d52ba18a6cf93b807eda67e02fbc6ff88.png)

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [May 14, 2020, 11:49pm UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/4 "2020-05-14T23:49:11Z")

</div>

The script I've provided takes each `hit._source` and indexes it in the destination index.  
I've tested just before sharing it to you.

If you only need one field in the final document, you have to use:

```auto
"index_payload": {
        "transform": {
          "script": {
            "source": "['_doc': ctx.payload.hits.hits.stream().map(h -> ['app_message' : h._source.app_message]).collect(Collectors.toList()) ];",
            "lang": "painless"
          }
        },
        "index": {
          "index": "tnr-doc-store-qa",
          "doc_type": "_doc"
        }
   }

```

---

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 15, 2020, 12:45am UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/5 "2020-05-15T00:45:09Z")

</div>

Thank you again..getting close.

Its writing the key "app\_message" as well in the index because of which the index is not mapping all fields of JSON. I just need the json body or value of the key "app\_message" .

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/a/8a7161b3e493dd53aa57bb38a271749afc1a3806.png)

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [May 15, 2020, 1:23am UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/6 "2020-05-15T01:23:51Z")

</div>

From your screenshot, the content of `app_message` is a \*\*stringified representation of a JSON \*\*, not a JSON object.

Painless has no JSON parsing library so it's not possible to convert the string into an actual object.

The only workaround I can think of is using an Ingest Pipeline.

**The following solution requires Ingest nodes.**

1. Delete the index `tnr-doc-store-qa`

```auto
DELETE tnr-doc-store-dev

```

1. Create a Ingest Pipeline to decode JSON strings

```auto
PUT _ingest/pipeline/jsondecode-app_message
{
  "processors": [
    {
      "json": {
        "field": "app_message",
        "add_to_root": true
      }
    },
    {
      "remove": {
        "field": "app_message"
      }
    }
  ]
}

```

1. Create the destination index

```auto
PUT tnr-doc-store-dev

```

1. Assign the index a default ingest pipeline

```auto
PUT tnr-doc-store-dev/_settings
{
  "index.default_pipeline": "jsondecode-app_message"
}

```

1. Run the Watcher

```auto
...
    "actions": {
      "index_payload": {
        "transform": {
          "script": {
            "source": "['_doc': ctx.payload.hits.hits.stream().map(h -> [ 'app_message ': h._source.app_message]).collect(Collectors.toList()) ];",
            "lang": "painless"
          }
        },
        "index": {
          "index": "tnr-doc-store-dev",
          "doc_type": "_doc"
        }
      }
    }
...

```

---

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 15, 2020, 7:54am UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/7 "2020-05-15T07:54:11Z")

</div>

Thank you, I tried the steps but getting error : cannot add non-map fields to root of document]; nested: IllegalArgumentException[cannot add non-map fields to root of document . Here is the execution output from index action .

``````````````````````````````````````````````````````````````````````````auto

 "actions": [
      {
        "id": "index_payload",
        "type": "index",
        "status": "failure",
        "transform": {
          "type": "script",
          "status": "success",
          "payload": {
            "_doc": [
              {
                "app_message ": "{\"request\":{\"header\":{\"clientId\":\"AACORN\",\"channel\":\"ARC\",\"transactionId\":\"aacd6333-7164-4ec6-a9c5-2a751f5da9a8\",\"host\":\"TSTS\",\"stationId\":null,\"nofepSessionToken\":null,\"swstoken\":null,\"pointOfSale\":\"XTM\",\"recordLocator\"
              }
            ]
          }
        },
        "index": {
          "response": [
            {
              "failed": true,
              "message": "ElasticsearchException[java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: cannot add non-map fields to root of document]; nested: IllegalArgumentException[java.lang.IllegalArgumentException: cannot add non-map fields to root of document]; nested: IllegalArgumentException[cannot add non-map fields to root of document];",
              "id": null,
              "type": "_doc",
              "index": "tnr-doc-store-qa"
            }
          ]
        }
      }

`````````````````````````````````````````````````````````````````````````
``````````````````````````````````````````````````````````````````````````

---

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 15, 2020, 7:57am UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/8 "2020-05-15T07:57:50Z")

</div>

Also, why does it behave differently when writing same JSON payload from watcher vs through POSTMAN hitting elastic endpoint with PUT operation directly ? As you can see below , the json body is right under \_source, when we run through postman . Thats exactly what I am looking for to achieve through this watcher as well.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/2/2241e64c769fd0545f31afb75df7059b7e335ce0.png)

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [May 15, 2020, 9:12am UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/9 "2020-05-15T09:12:52Z")

</div>

You didn't follow the process as you changed the destination index as you're writing to `tnr-doc-store-qa`.

---

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 15, 2020, 7:38pm UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/10 "2020-05-15T19:38:40Z")

</div>

Unfortunately I cant delete the `tnr-doc-store-dev` and hence I have been updating only the `tnr-doc-store-qa` . Below are the exact steps I did :

```````````````````````````````````````````````````````````````````````````````auto

DELETE tnr-doc-store-qa

PUT _ingest/pipeline/jsondecode-app_message
{
  "processors": [
    {
      "json": {
        "field": "app_message",
        "add_to_root": true
      }
    },
    {
      "remove": {
        "field": "app_message"
      }
    }
  ]
}

PUT tnr-doc-store-qa

PUT tnr-doc-store-qa/_settings
{
  "index.default_pipeline": "jsondecode-app_message"
}

"actions": {
      "index_payload": {
        "transform": {
          "script": {
            "source": "['_doc': ctx.payload.hits.hits.stream().map(h -> [ 'app_message ': h._source.app_message]).collect(Collectors.toList()) ];",
            "lang": "painless"
          }
        },
        "index": {
          "index": "tnr-doc-store-qa",
          "doc_type": "_doc"
        }
      }
    }

``````````````````````````````````````````````````````````````````````````````

After I ran the watcher, I get below error message : 

"index": {
          "response": [
            {
              "failed": true,
              "message": "ElasticsearchException[java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: cannot add non-map fields to root of document]; nested: IllegalArgumentException[java.lang.IllegalArgumentException: cannot add non-map fields to root of document]; nested: IllegalArgumentException[cannot add non-map fields to root of document];",
              "id": null,
              "type": "_doc",
              "index": "tnr-doc-store-qa"
            },
```````````````````````````````````````````````````````````````````````````````

---

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 15, 2020, 8:01pm UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/11 "2020-05-15T20:01:51Z")

</div>

My bad..it worked..The issue was I had an extra space after app\_message in the index action. After I removed it , it worked like a charm and I see all the data getting properly indexed.

Thank you so so much [Luca\_Belluccini] for your help . You are amazing !!!

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [May 15, 2020, 8:04pm UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/12 "2020-05-15T20:04:54Z")

</div>

Glad it helped! ☀

---

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 15, 2020, 11:13pm UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/13 "2020-05-15T23:13:03Z")

</div>

One last help .. My JSON payload doesnt contain any time field so that I can search my data on. Is there a way we can write the current timestamp to the index payload that I can use to run my time based searches ?? Thank you so much.

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [May 16, 2020, 7:02am UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/14 "2020-05-16T07:02:09Z")

</div>

In the painless transform, use:

```auto
...
['app_message ': h._source.app_message, '@timestamp': ctx.execution_time]
...

```

---

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 18, 2020, 4:09am UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/15 "2020-05-18T04:09:40Z")

</div>

> [@Luca\_Belluccini](#):
>
> '@timestamp': ctx.execution\_time

That worked. Thank you so much.

---

<div class="post-metadata">

**Author:** ![Bhavani\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavani_prasad/32/68370_2.png) [@Bhavani\_Prasad](https://discuss.elastic.co/u/Bhavani_Prasad)\
**Post date:** [May 21, 2020, 6:21am UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/16 "2020-05-21T06:21:07Z")

</div>

Hi Luca, I need to add a scripted field which is a sum of two values in my payload.

I created a scripted field as below under my index pattern but it throws an error .

```auto
Double.parseDouble(doc['request.orderGroups.orderItems.totalItemBaseAmount'].value) + Double.parseDouble(doc['request.orderGroups.orderItems.totalItemTaxAmount'].value)

"type": "illegal_argument_exception",
     "reason": "No field found for [request.orderGroups.orderItems.totalItemBaseAmount] in mapping with types []"

Do I need to access the values inside my json payload in a different way to get the fields I need for the scripted field
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 18, 2020, 6:21am UTC](https://discuss.elastic.co/t/watcher-with-index-action-not-writing-data-properly/232714/17 "2020-06-18T06:21:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.