Ability to search ES based on a result of previous search in a watcher

[Jasonespo (JasonE)]

Hi

I was wondering if it was possible (in a watcher) to do the following:

• Search an index for a keyword e.g document must contain "tag": "laptop"

• In that document it contains another field called "identifier": "091029"

• Search again for all documents with the same "identifier": "091029" within the past X time period

• Display the result in json

Any ideas?

Thanks

[spinscale (Alexander Reelsen)]

Hey Jason,

can you explain the connection between those two queries? Right now for me these two look like independent queries, so I guess I am missing something.

For running two queries as part of a watch input, check out the chain input

Hope this helps!

--Alex

[Jasonespo (JasonE)]

@spinscale - Looks like a potential so far, thank you. Not 100% sure it operates how i want it to though..

Let me try and explain a bit further...

The first search is to alert/execute if any logs come in with the tag "laptop" (note this will only execute if the tag is "laptop". The document will have another field called identifier which I will need to set as a variable for the next search.

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "chain": {
      "indices": [
        "tid*"
      ],
      "inputs": [
        {
          "first": {
            "simple": {
              "query": {
                "bool": {
                  "filter": [
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-{{ctx.metadata.window_period}}"
                        }
                      }
                    },
                    {
                      "term": {
                        "tag": "laptop"
                      }
                    }
                  ]
                }
              }
            }
          }
        },

After the first search returns the document I then need to do the second search on the identifier variable

 ... "second": {
            "simple": {
              "query": {
                "bool": {
                  "filter": [
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-{{ctx.metadata.window_period}}"
                        }
                      }
                    },
                    {
                      "term": {
                        "identifier": $value
                      }
                    }
                  ]
                }
              }
            }
          }
        }...

Then the json body result would be all the documents with that identifier $value in the past 2 minutes..

Do you think this is possible? And any guidance on accessing the initial document to set the variable for the second search?

Thanks for the help so far!

[spinscale (Alexander Reelsen)]

Yes, you can do this. You can convert the data using a script transform in your chain inputs.

See https://www.elastic.co/guide/en/elasticsearch/reference/7.6/input-chain.html#_transforming_chained_input_data

[Jasonespo (JasonE)]

Thanks again @spinscale

I've been looking at this a while now and can't get it to work even with transforms and chain inputs... (It probably is the solution)..

Let me give you the exact use case and if you could confirm one last time that it would work for this, that would be great (then i'll get out of your hair!)

We want to alert on certain audit logs. These logs have a "key" field. The watcher would look for the occurence of this "key" field. (Relatively simple start, I know how to do this).

After it has matched a certain key value e.g

"key": "alert_key"

However, the document that contains this field does not have all the relevant information that we want to produce to our users.

It contains another field called "alert_id". This unique alert_id field exists in 5 or more documents, however the previous key we searched for does not exist in these documents (otherwise this would be to easy.. eh..) I need to take a different field from the initial document to be used in the second search... This would be alert_id.

"alert_id": "01723213-0183291013"

We now after finding this "alert_id" need to find ALL the documents with this unique key in the past X time period.

I'm probably out of my depth attempting this and need to seek help from one of other members of the team..

Maybe this is more my solution - what do you think

Tried it a bit here:

{
  "trigger": {
    "schedule": {
      "interval": "2m"
    }
  },
  "transform": {
    "chain": [
      {
        "search": {
          "request": {
            "indices": [
              "tid*"
            ],
            "body": {
              "size": 0,
              "query": {
                "bool": {
                  "filter": [
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-{{ctx.metadata.window_period}}"
                        }
                      }
                    },
                    {
                      "term": {
                        "key": "alert_key"
                      }
                    },
                    {
                      "aggs": {
                        "term": {
                          "field": "alert_id"
                        }
                      }
                    }
                  ]
                },
                "query": {
                  "bool": {
                    "filter": [
                      {
                        "range": {
                          "@timestamp": {
                            "gte": "now-{{ctx.metadata.window_period}}"
                          }
                        }
                      },
                      {
                      //not sure how to query Kibana for the alert_id value from the previous query 
                      }
                    ]
                  }
                }
              }
            }
          }
        }
      }
    ]
  }
}

Thanks again.

[spinscale (Alexander Reelsen)]

Check out this snippet, hope it helps

DELETE first_index,second_index

PUT first_index/_doc/1?refresh
{
  "field_to_check" : "foo"
}

PUT second_index/_doc/1
{
  "key" : "foo"
}

PUT second_index/_doc/2?refresh
{
  "key" : "bar"
}

POST _xpack/watcher/watch/_execute
{
  "watch": {
    "trigger": {
      "schedule": {
        "interval": "10h"
      }
    },
    "input": {
      "chain": {
        "inputs": [
          {
            "first": {
              "search": {
                "request": {
                  "indices": [
                    "first_index"
                  ],
                  "body": {
                    "query": {
                      "match_all": {}
                    }
                  }
                }
              }
            }
          },
          {
            "second": {
              "search": {
                "request": {
                  "indices": [
                    "second_index"
                  ],
                  "body": {
                    "query": {
                      "term": {
                        "key": {
                          "value": "{{ctx.payload.first.hits.hits.0._source.field_to_check}}"
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        ]
      }
    },
    "actions": {
      "logme": {
        "logging": {
          "text": "{{ctx}}"
        }
      }
    }
  }
}

[Jasonespo (JasonE)]

GREAT, thank you! Looks like we are getting there...

This would match every document with the field named "field_to_check".

So for us this would be "alert_id". But we only care about matching the values of "field_to_check" if it matches what was in the initial query...

So of the 10 documents that contain that field name... we might only need 3 of them. Would the next step be a condition that compares the alert_ids in query 1 and query 2?

Then output the offenders that match the condition?