Issues with CTX behavior while using chains, http input and tranform

Elastic Stack Elasticsearch
1

Hi,

I would like to get some precision and guidance on how the ctx content change/behave while working with a watch that combine transform, input (http) and action.

FYI - I'm trying to achieved a watch that will run API calls on my ES cluster using http input [note: the path/body of the calls are set up in a transform before my input] and then I want to use the results of my http chain to trigger a slack notification [for the purpose of this example - I have replace my slack action by a plain log text as it's helping me troubleshot].

I've listed below the issues that I've come to realized while working on this watch and I hope that you could bring some light on those to help me move forward :

1. The watch will stop execute a top level transform if you replace your input by a chain input

2. If you use a top level transform in combination with a http input only the content of the transform will end up in the CTX.payload while accessing it from the action section (end of watch sequence). While as soon as you remove the transform from the watch then suddenly your CTX.payload contains the results of the input(s) (as you would expect !!).

Here is the watch that I'm using - if you want to review/play with the two use cases expose above:

 POST _watcher/watch/_execute
    {
      "watch":{
        "metadata": {
          "keep_history_days": 7
        },
         "transform": {
          "script" : "return [ ParserIndexToDelete : '/.parser_metric_' + ctx.execution_time.minusDays(ctx.metadata.keep_history_days + 1).toString('yyyy.MM.dd'), WatcherIndexToDelete : '/.watch-history_' + ctx.execution_time.minusDays(ctx.metadata.keep_history_days + 1).toString('yyyy.MM.dd') ]"
        },
        "trigger": {
          "schedule": { "daily": { "at" : "00:01" }}
        },
        "input": {
          "chain" : {
            "inputs" : [ {
              "first" : {
                "http" : {
                  "request": {
                    "scheme" : "https",
                    "host": "localhost",
                    "port": 9243,
                    "path": "{{ctx.payload.ParserIndexToDelete}}",
                    "method": "delete",
                    "auth" : {
                      "basic" : {
                        "username" : "username_here", 
                        "password" : "password_here" 
                      }
                    }
                  }
                }
              },
              "second": {
                "http" : {
                  "request": {
                    "scheme" : "https",
                    "host": "localhost",
                    "port": 9243,
                    "path": "{{ctx.payload.WatcherndexToDelete}}",
                    "method": "delete",
                    "auth" : {
                      "basic" : {
                        "username" : "username_here", 
                        "password" : "password_here" 
                      }
                    }
                  }
                }
              }
            } ]
          }  
        },
        "condition": {
          "always": {}
        },
        "actions": {
          "log" : { 
            "transform" : {}, 
            "logging" : {
              "text" : "{{ctx}}" 
            }
          }
        }
      }
    }

Note: I'm running this on an ElasticCloud 2.4 cluster (Gold).

As you might have guess my goals are to be able to 1. be able to use the ctx.payload on my transform in a input chain and 2. to have the http input(s) to record their results in ctx.payload so I can then use that in the action section of my watch.

Thanks in advance for your help!
Gregory

2

I believe the transform phase is always run between the inputs and actions, which is why you do not have the results of it available in the input phase. Why are you simply not deleting the data in two webhook actions, which will always run after the transform, instead of as an input (which to me sounds counterintuitive)?

3

I have been using webhooks extensively so far but the idea behind this approach was to run the webhook (e.i. http) and then have the possibility to act upon it's results... for example if my API call fail I could have trigger a slack notification, etc.

Note: I know I could probably acheive this by doing two watches... 1 to perform the action(s) then a 2nd to look at the result of the first in watch_history-* but I really wanted to avoid that overhead by combining both in a more elegant/easier to maintain solution.

Regarding the transform phase looks like you are right... my bad - I jump too quickly to conclusion when I was able to deploy the watch with this located above my input. That's really an area for improvement though because it would have been very useful to be able to run the transform before or even better within the input blocks (similar to the actions ones) of course all that while keeping the CTX content consistent across all sections/blocks of the watch :slight_smile:

4

Hey,

first, I am not sure if the input of your operations should be the deletion of an index, but that is of course entirely up to you.

Second, have you thought about using date math instead of trying to calculate everything yourself? This would prevent you from needed any pre calculations.

--Alex

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.