Separate index for every aggs bucket

Elastic Stack Elasticsearch
1

Hi, I want to create a separate index for every aggs buckets. here is my watcher.
I know I have to use "foreach" for my problem but can't figure out what value should I give it.
thanks.

1 Like
2

Your tranform will produce this payload, so when you refer ctx.payload.aggregations.groupbyhost.keys it will produce following exception specified foreach object was null: [ctx.payload.aggregations.device_name.keys]
there no aggregations.device_name.keys under payload ....
Please review your tranform script.

What is your target ? is to get for each device :

  • Number of documents

  • Array of users_name

  • Array of source_ip
    Or something specific ?

      {
    "payload": {
      "hits": [
        {
          "number": 17
        },
        {
          "number": 1
        }
      ],
      "@timestamp": "2020-09-06T09:20:14.3169139Z",
      "attacker IPs": [
        "10.10.1.10",
        "10.10.2.10"
      ],
      "host": [
        {
          "name": "Cisco Firewall"
        },
        {
          "name": "FortiGate Firewall"
        }
      ],
      "usernames": [
        "bornatalebi 1",
        "bornatalebi 1"
      ]
    }
  }
1 Like
3

Thanks for your reply,
Yes I want to create an index for each device containing the Number of documents, Array of users_name and Array of source_ip.
What should i replace ctx.payload.aggregations.groupbyhost.buckets.keys with? If i use something like ctx.payload.host it will only add host.name to index.

4

I would suggest you try composite aggregation to get a simplified output index
I'm not sure if that will help you, otherwise you need to review your tranform to produce the correct array you need

PUT _watcher/watch/ciscoioswatcher
{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          "filebeat-*"
        ],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": {
                "match": {
                  "event.code": {
                    "query": "LOGIN_FAILED"
                  }
                }
              },
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "now-5m"
                  }
                }
              }
            }
          },
          "aggs": {
            "composite_buckets": {
              "composite": {
                "sources": [
                  {
                    "device_name": {
                      "terms": {
                        "field": "DevName"
                      }
                    }
                  },
                  {
                    "source_ip": {
                      "terms": {
                        "field": "source.ip"
                      }
                    }
                  },
                  {
                    "user_name": {
                      "terms": {
                        "field": "user.name"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      }
    }
  },
  "actions": {
    "index_payload": {
      "foreach": "ctx.payload.aggregations.composite_buckets.buckets",
      "max_iterations": 100,
      "index": {
        "index": "outputindex"
      }
    }
  }
}
1 Like
5

I would suggest you try composite aggregation to get a simplified output index
I'm not sure if that will help you, otherwise you need to review your tranform to produce the correct array you need

Thanks. I think I have to edit my transform script.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.