Watcher / Deprecation: [types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})

[pierrejutard]

Hello,

When trying to insert a watcher in ElasticSearch:

PUT /watcher/watch/pricealert
{
"watch":
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"indices": [
"price-request-2020.01.28"
],
"body": {
"query": {
"match_all": {}
}
}
}
}
}
},
{
"second": {
"search": {
"request": {
"indices": [
"mktdata-2020.01.28"
],
"body": {
"query": {
"match_all": {}
}
}
}
}
}
}
]
}
},
"condition": {
"script": {
"source": "return ctx.payload.first.hits.hits.0._source.spot >= ctx.payload.second.hits.hits.0._source.lbound && ctx.payload.first.hits.hits.0._source.spot <= ctx.payload.second.hits.hits.0._source.ubound",
"lang": "painless"
}
},
"actions": {
"log": {
"logging": {
"level": "info",
"text": "spot out of bound!"
}
}
}
}
}

I met this problem:
#! Deprecation: [types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id}).
{
"_index" : "watcher",
"_type" : "watch",
"_id" : "pricealert",
"_version" : 2,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 14,
"_primary_term" : 1
}

Does anyone know from where does it come from?

I try to compare three fields ( one field "spot" from "price-request-2020.01.28" indice with the other two: "lbound" and "ubound" from "mktdata-2020.01.28" indice)
This is what look like the "mktdata-2020.01.28" indice :
"hits" : [
{
"_index" : "mktdata-2020.01.28",
"_type" : "_doc",
"_id" : "c8Ze628BDeRluIZMFHSD",
"_score" : 1.0,
"_source" : {
"twod" : 5.0,
"@timestamp" : "2020-01-28T08:56:48.165Z",
"spot" : 1.0,
"oned" : 3.0
}

Thanks
Best regards
Pierre Jutard

[pierrejutard]

I went through this problem by deleting the:
{
"watch":
{
at the begining of my watch configuration.

Best,
Pierre Jutard

[spinscale (Alexander Reelsen)]

This looks however as if you are missing an underscore the beginning of the URL, i.e. like _watcher

The request you send simply creates an watcher index, but is not creating a watch using the put watch API.

[pierrejutard]

Yes you' re right Alex.

But even with the "_" the watcher is not created.

With this watcher configuration:

PUT _watcher/watch/pricealert
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"indices": [
"price-request-2020.01.28"
],
"body": {
"size": 10,
"query": {
"match_all": {}
}
}
}
}
}
},
{
"second": {
"search": {
"request": {
"indices": [
"mktdata-2020.01.28"
],
"body": {
"size": 10,
"query": {
"match_all": {}
}
}
}
}
}
}
]
}
},
"condition": {
"script": {
"source": "return ctx.payload.first.hits.hits.0._source.spot >= 0.8",
"lang": "painless"
}
},
"actions": {
"log": {
"logging": {
"level": "info",
"text": "spot out of bound!"
}
}
}
}

I get this result:

{
"_id" : "pricealert",
"_version" : 13,
"_seq_no" : 12,
"_primary_term" : 1,
"created" : false
}

Do you know why?

When running with POST method this watch i have also this result:
{
"_id" : "pricealert_77888106-c314-4075-a55e-6a9066ca9d71-2020-01-28T13:13:57.142Z",
"watch_record" : {
"watch_id" : "pricealert",
"node" : "p-BVjNRJT_WIpKDIrre42Q",
"state" : "failed",
"status" : {
"state" : {
"active" : true,
"timestamp" : "2020-01-28T13:10:54.115Z"
},
"actions" : {
"log" : {
"ack" : {
"timestamp" : "2020-01-28T13:10:54.115Z",
"state" : "awaits_successful_execution"
}
}
},
"execution_state" : "failed",
"version" : 13
},
"trigger_event" : {
"type" : "manual",
"triggered_time" : "2020-01-28T13:13:57.142Z",
"manual" : {
"schedule" : {
"scheduled_time" : "2020-01-28T13:13:57.142Z"
}
}
},
"input" : {
"chain" : {
"inputs" : [
{
"first" : {
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"price-request-2020.01.28"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 10,
"query" : {
"match_all" : { }
}
}
}
}
}
},
{
"second" : {
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"mktdata-2020.01.28"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 10,
"query" : {
"match_all" : { }
}
}
}
}
}
}
]
}
},
"condition" : {
"script" : {
"source" : "return ctx.payload.first.hits.hits.0._source.spot >= 0.8",
"lang" : "painless"
}
},
"result" : {
"execution_time" : "2020-01-28T13:13:57.142Z",
"execution_duration" : 3,
"input" : {
"type" : "chain",
"status" : "success",
"payload" : {
"first" : {
"_shards" : {
"total" : 1,
"failed" : 0,
"successful" : 1,
"skipped" : 0
},
"hits" : {
"hits" : [
{
"_index" : "price-request-2020.01.28",
"_type" : "_doc",
"_source" : {
"@timestamp" : "2020-01-28T08:57:34.000Z",
"subject" : "price request",
"lbound" : 0.8,
"ubound" : 1.2
},
"_id" : "dMZe628BDeRluIZM-HR9",
"_score" : 1.0
}
],
"total" : 1,
"max_score" : 1.0
},
"took" : 1,
"timed_out" : false
},
"second" : {
"_shards" : {
"total" : 1,
"failed" : 0,
"successful" : 1,
"skipped" : 0
},
"hits" : {
"hits" : [
{
"_index" : "mktdata-2020.01.28",
"_type" : "_doc",
"_source" : {
"@timestamp" : "2020-01-28T08:56:48.165Z",
"spot" : 1.0,
"oned" : 3.0,
"twod" : 5.0
},
"_id" : "c8Ze628BDeRluIZMFHSD",
"_score" : 1.0
}
],
"total" : 1,
"max_score" : 1.0
},
"took" : 0,
"timed_out" : false
}
},
"chain" : {
"first" : {
"type" : "search",
"status" : "success",
"payload" : {
"_shards" : {
"total" : 1,
"failed" : 0,
"successful" : 1,
"skipped" : 0
},
"hits" : {
"hits" : [
{
"_index" : "price-request-2020.01.28",
"_type" : "_doc",
"_source" : {
"@timestamp" : "2020-01-28T08:57:34.000Z",
"subject" : "price request",
"lbound" : 0.8,
"ubound" : 1.2
},
"_id" : "dMZe628BDeRluIZM-HR9",
"_score" : 1.0
}
],
"total" : 1,
"max_score" : 1.0
},
"took" : 1,
"timed_out" : false
},
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"price-request-2020.01.28"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 10,
"query" : {
"match_all" : { }
}
}
}
}
},
"second" : {
"type" : "search",
"status" : "success",
"payload" : {
"_shards" : {
"total" : 1,
"failed" : 0,
"successful" : 1,
"skipped" : 0
},
"hits" : {
"hits" : [
{
"_index" : "mktdata-2020.01.28",
"_type" : "_doc",
"_source" : {
"@timestamp" : "2020-01-28T08:56:48.165Z",
"spot" : 1.0,
"oned" : 3.0,
"twod" : 5.0
},
"_id" : "c8Ze628BDeRluIZMFHSD",
"_score" : 1.0
}
],
"total" : 1,
"max_score" : 1.0
},
"took" : 0,
"timed_out" : false
},
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"mktdata-2020.01.28"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 10,
"query" : {
"match_all" : { }
}
}
}
}
}
}
},
"actions" :
},
"exception" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"return ctx.payload.first.hits.hits.0._source.spot >= 0.8",
" ^---- HERE"
],
"script" : "return ctx.payload.first.hits.hits.0._source.spot >= 0.8",
"lang" : "painless",
"caused_by" : {
"type" : "null_pointer_exception",
"reason" : null,
"stack_trace" : """java.lang.NullPointerException
at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:385)
at org.elasticsearch.painless.PainlessScript$Script.execute(return ctx.payload.first.hits.hits.0._source.spot >= 0.8:45)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:60)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:55)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:513)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:320)
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:159)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:627)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"""
},
"stack_trace" : """ScriptException[runtime error]; nested: NullPointerException;
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94)
at org.elasticsearch.painless.PainlessScript$Script.execute(return ctx.payload.first.hits.hits.0._source.spot >= 0.8:45)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:60)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:55)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:513)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:320)
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:159)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:627)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:385)
... 12 more
"""
}
}
}

So my question is : Is a watch not created when there is an error in my configuration?

Best,
Pierre Jutard

[spinscale (Alexander Reelsen)]

What Elasticsearch version is this?

[pierrejutard]

7.5.2 basic license with 30 day trial extension for the watcher use.

[spinscale (Alexander Reelsen)]

Three things.

First, please format your snippets. You can use markdown in here, and this is super hard to read, especially JSON or exceptions. Thanks.

Second, the PUT Watch API response from above with the pricealert executed as expected and was stored as a watch.

Third, the third was the output of the exeucte watch API I suppose and has nothing to do with storing the watch. The exception you can read there shows that you tried to access a field, that did not exist. The script is wrong, as you used mustache syntax. Try ctx.payload.first.hits.hits[0]._source.spot - also check the result as I do not see a spot field in the first payload, but I may be missing it due to the formatting.

hope this helps

[pierrejutard]

Sorry for the first....

For the second, i understand that this is just updated and not created.

For the third, you're right i have inverted the second with the first.
But even by following your syntax and make the inversion, it is not working.

This is the result of my two indices that are in the chain inputs:
First:

Second:

My configuration watch:

The result:

Is it a size problem of my search?

Thank you,
Pierre Jutard

[pierrejutard]

Yes, it' s a size problem, when setting Size:10 it worked and i go to a cast error after but when you change the equality in inequality it worked definitly.
Final solution:

Thanks
Pierre Jutard