Ctx.payload.hits.hits.0._source.field to watcher-history index

Elastic Stack Elasticsearch
1

Hi all,

I am trying to aggregate ctx source fields (triggered source event fields) and force watcher to write them to watcher-history index. For eg i wanna pass the ctx.payload.hits.hits.0._source.computer_name of a winlogbeat as new field to the watcher-history index. Is there a way?
I tried tranform script , webhook and update_query but none of the seem to work as expected. I really could use an example .

Thanks in advanced for any help

2

the watcher history index mapping should not be fiddled with. However, all the results of a search are already written into a watch record (the name for a single entry in the history), albeit not searchable.

If you need to have this searchable you should probably reindex the data in your own index. When doing that you might want to take a look at the result.input field which contains your search.

Hope this helps.

--Alex

3

I need to write ctx.payload.hits.hits.0._source.field or result.input.payload.hits.hits fields (if accessible) to watcher-history index or another index if it is possible and write watches that correlates the correlated events.
I understand why i shouldn't write to watcher-history index but i need to have the fields accessed and searched. Is it possible? Can a watcher action write payload fields to an index to map them?

I would appreciate an example .
Thanks again for your help

4

How about using the index action to write to an index you maintain yourself?

5

I will try something like that:

"index_payload": {
  "index": {
    "index": "correlated",
    "doc_type": "data"
  }
}

This will populate the fields from the watch ?

6

the current payload will be used, but you may likely want to use a script transform before the action is executed. There is even support to index into multiple documents, see https://www.elastic.co/guide/en/x-pack/6.2/actions-index.html

7

I tried transforming a field.

"index_payload": {
"transform": {
"script": {
"source": "[ 'sourceHostname' : ctx.payload.hits.hits.0._source.computer_name ]",
"lang": "painless"
}
},
"index": {
"index": "correlated",
"doc_type": "data"
}
}

Watch got a hit the result was the following:

{

"id": "index_payload",
"type": "index",
"status": "success",
"transform": {
"type": "script",
"status": "success",
"payload": {
"sourceHostname": "xxxxx"
}
},
"index": {
"response": {
"created": true,
"result": "created",
"id": "UGrQuGIBIQgb1Is59367",
"version": 1,
"type": "data",
"index": "correlated"
}
}
}

And although the response is successive in the correlated index i see nothing. Note that without transform hit is indexed in the correlated.
Any clues?

Thanks again

8

Update:

Only sourceHostname is populated in the index . Is there a way to loop with transform to extract all fields and populated the recursively?

Thanks for any help

9

Do you want to have all source hostnames?

def hostnames = ctx.payload.hits.hits.stream().map(hit -> hit._source.computer_name).collect(Collectors.toList());
1 Like
10

Nope i want all the field from the hit to populated to the new index through a transform loop. And not add them manually. Or maybe pass someway the hits json as a whole?
I am not sure which way is better. An example could help.

Thanks again

11

well, then dont filter on the field name but just use the whole _source in the above example.

We have a couple of examples, you might want to take a look at.

As you are testing with out own data, I think the most important part would be to reduce your feedback loop while testing. I wrote a longish blog post on that some time ago, which I can just advice to read. See https://www.elastic.co/blog/watching-the-watches-writing-debugging-and-testing-watches

12 
"index_payload": {
  "transform": {
    "script": {
      "source": "[ ctx.payload.hits.hits.0._source ]",
      "lang": "painless"
    }
  },

OR

"index_payload": {
  "transform": {
    "script": {
      "source": "['message': ctx.payload.hits.hits.0._source ]",
      "lang": "painless"
    }
  },

Something like that you mean? Thanks for the examples i really wanted 2 lines of an example for my case

1 Like
13

sorry for not being, I was referring to my example above, so you include all the hits.

14

No worries thanks. One last question can i add in the same script multiple field values. For eg:

"source": "[ 'sourceHostname' : ctx.payload.hits.hits.0._source.computer_name, 'destinationHostname' : ctx.payload.hits.hits.0._source.test_field ]"

Whould it work? Thanks again

15

Hey,

yes that would work. my example above simply copies all fields. Also you could of course create black/whitelist to only include a certain set of fields, without having to specify them manually.

--Alex

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.