Watcher multiple queries/ query within range field

Elastic Stack Elasticsearch
1

Hi,

I am trying to find a way to use two different queries (on different indices) in the same watcher.

My case:

I have two indices:

  • index1 - contains the documents I need to count

  • index2 - here I will store the countings

The watcher should:

  • get the number of documents from index1 within a date range (would be nice if I could use the date from the last entry of index2)
  • sum the counted documents from index 1 and the value of last entry in index2
  • save the sum to index 2

Here's my code so far:

PUT _watcher/watch/postsurveyrequest_count/
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          "index1"
        ],
        "body": {
          "query": { //how to add second query?
            "bool": {
              "must": {
                "match": {
                  "fields.RequestType": "PostSurveyRequest"
                }
              },
              "filter": {
                "range": {
                  "@timestamp": {
                    "lt": "now-3m/m" //here would be nice to return the date from last entry in index2
                  }
                }
              }
            }
          }
        }
      },
      "extract": [
        "hits.total"
      ]//extract hits.total from first query and latest entry count value from the to-be second query
    }
  },
  "actions": {
    "index": {
      "transform": {
        "script": {
          "inline": "return [ 'sum': ctx.payload.hits.total  , 'time' :ctx.trigger.scheduled_time  ]", //the sum should be something like: ctx.payload.hits.total + secondquery_result

        }
      },
      "index": {
        "index": "index2",
        "doc_type": "PostSurveyRequestSum"
      }
    }
  }
}

I've added comments in the code with the functionality I would like. Can something like this be achieved?

Other questions:

  • Could I insert a specific date instead of ctx.trigger.scheduled_time? If yes, how?
  • Would it be possible to have some "global variables"? To store query results and use them in other parts of the code.

Any tips or links to documentation would be appreciated.

2

Hey,

you can use the a chained input and just issue two different queries? Would that help already?

I dont understand your specific date question, can you clarify?

Also, there are no global variables, but inside of a single watch execution you can access those results, you can also store them using the index action, and use another watch to query them. Apart from that every watch gets executed by itself and does not contain any state of other watches.

--Alex

3

Hi Alex,

Thanks for the reply.

I will try to use the chained input.

About the date, I want to use something like this
"return ['time': 'now-2d/d']"
But I couldn't find the right syntax to actually parse it to a date and not text.

4

Hey,

that date math only works in index names and in queries, but nowhere else.

Maybe you can explain your whole use-case in more detail, to see if we find other solutions.

--Alex

5

Hi Alex,

I want to save the counts from index 1 in "chunks", for example:
Chunk1: begining to date1
Chunk2: date1 to date2
Chunk3: date2 to date3
and so on
But I want the date in index 2 to be the end date of the chunk, for example Chunk1 would be saved in index 2 with the "time" field set to date1.

The big picture: I have a lot of logs that I keep just to have a cumulative sum of all PostSurveyRequests. I want to "move" the data I need to a new index and remove the old ones (automatically). But to start with I need to set some older dates on the entries to the new index (index 2), otherwise my chart would show the data as if all happened in the last few minutes instead of the few months.

Hope it helps.

6

Hey,

why does your requirement not work with date math based queries? I might miss something. See the docs

--Alex

7

Hi,

I have no issue with the queries, they work fine.
The problem is when trying to return a custom date I set:
'time' : \"5/10/2016\" - it will see this as a string and not a date value
Found this syntax for groovy:
"script":"use( groovy.time.TimeCategory ) { new Date( doc[\"aDate\"].value ) }"
Is there an equivalent for painless?

Thanks.

8

Hi again,

I managed to set a custom date by using new Date()with a timestamp like so:
'time' : new Date(1475625600000L)

Thanks again for the help.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.