Watcher to index all data from /cat/indices/*,-.* each indexname should be inserted as one doc

Elastic Stack Kibana
1

Hi team,
Can you please help me on below requirement:

  1. Get index name and size (in bytes) from GET /cat/indices/,-.
  2. Insert this data into new index
  3. IDs should be elastic generated. Add timestamp field in every doc

Problem:
I have created below watcher but it is inserting all index info as one document . I need each index name and size as one document.

{
  "trigger": {
    "schedule": {
      "daily": {
        "at": [
          "noon"
        ]
      }
    }
  },
  "input": {
    "http": {
      "request": {
        "scheme": "http",
        "host": "localhost",
        "port": 9200,
        "method": "get",
        "path": "/_cat/indices/*,-.*",
        "params": {
          "format": "json",
          "h": "index,store.size",
          "bytes": "b"
        },
        "headers": {},
        "auth": {
          "basic": {
            "username": "abc",
            "password": "abc"
          }
        }
      }
    }
  },
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "to": [
          "abc@abc.com"
        ],
        "subject": "404 recently encountered",
        "body": {
          "text": "indices \n {{ctx.payload}}"
        }
      }
    },
    "index_payload": {
      "index": {
        "index": "my-index",
        "doc_type": "_doc"
      }
    }
  }
}
2

Following this example in the docs, you could try

{
    "trigger": {
      "schedule": {
        "interval": "1d"
      }
    },
    "input": {
      "http": {
        "request": {
          "host": "localhost",
          "port": 9200,
          "scheme": "https",
        "path": "/_cat/indices/*,-.*",
        "params": {
          "format": "json",
          "h": "index,store.size",
          "bytes": "b"
        },
          "headers": {},
          "auth": {
            "basic": {
              "username": "elastic",
              "password": "xxx"
            }
          }
        }
      }
    },
  "condition": {
    "always": {}
  }, 
  "actions" : {
    "index_payload": {
    "transform": {
      "script": """
       def documents = ctx.payload.data.stream()
        .map(hit -> [
          "_index": "index_sizes", 
          "index": hit.index, 
          "store.size": hit['store.size']
        ])
        .collect(Collectors.toList());
      return [ "_doc" : documents]; 
      """
    },
    "index": {
        "execution_time_field": "@timestamp"
        }
    }
  }
}
3

GET index_sizes/_search

yields something like:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "index_sizes",
        "_id": "y1jD-IwBRZWNq0rMQ4W9",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-01-11T13:43:15.438Z",
          "index": "test-data",
          "store.size": "907197"
        }
      },
      {
        "_index": "index_sizes",
        "_id": "zFjD-IwBRZWNq0rMQ4W9",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-01-11T13:43:15.438Z",
          "index": "my-index",
          "store.size": "284852"
        }
      },
       {
        "_index": "index_sizes",
        "_id": "0VjD-IwBRZWNq0rMQ4W9",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-01-11T13:43:15.438Z",
          "index": "kibana_sample_data_ecommerce",
          "store.size": "8783478"
        }
      }
    ]
4

Hi @richcollier ,
thanks for the help,
i am using 6.8.23 version and getting below 1 error . can you please help me on this.

NOTE:
it is working in 8.6 version

image

5

Seems like it doesn't like the defining of a script using the triple double-quote notation (i.e """) . This syntax means that you're able to use free-formatting until the closing """, including new lines. If you look back at the v6.8 documentation, it only shows defining a script on a single line deliniated with only a single double-quote (") on either side of the code snippet.

You could try to take what I wrote, remove the new lines, put it all on one contiguous line, and surround it with a single double-quote. Might work!

But you should know 6.x is end-of-life and has been for years. I'm sorry that it doesn't work for you but if my above suggestion doesn't help I'm not sure I can help you further.

6

its working now thank you so much

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.