Roll Up Index does not get populated with documents

Elastic Stack Elasticsearch
1

Hi folks,

my goal is to archive the document _size of incoming logs in specific archive index, because the original document gets deleted using ILM.

Example Document:

{
   "_index":"filebeat-live-index-01",
   "_type":"_doc",
   "_id":"okl-3H0BqY5xTzuloFW0",
   "_size":1435,
   "_source":{
      "kubernetes":{
         "namespace":"some_namespace",
         "container":{
            "name":"some_container"
         },
 
      "message":" some log message"
   }
}

From this post I learned that roll up mechanism can be used for it. I have configured the rollup-job but my rollup index has always 0 documents.

This is my rollup job:

{
  "config": {
    "id": "rollup_test",
    "index_pattern": "filebeat-live-index-*",
    "rollup_index": "size_archive",
    "cron": "0 * * * * ?",
    "groups": {
      "date_histogram": {
        "fixed_interval": "30s",
        "field": "@timestamp",
        "time_zone": "UTC"
      },
      "terms": {
        "fields": [
          "_size"
        ]
      }
    },
    "metrics": [],
    "timeout": "20s",
    "page_size": 1000
  },
  "status": {
    "job_state": "stopped",
    "current_position": {
      "@timestamp.date_histogram": 1640081820000,
      "_size.terms": 1461
    },
    "upgraded_doc_id": true
  },
  "stats": {
    "pages_processed": 554,
    "documents_processed": 2156167,
    "rollups_indexed": 397469,
    "trigger_count": 156,
    "index_time_in_ms": 48054,
    "index_total": 398,
    "index_failures": 0,
    "search_time_in_ms": 5844,
    "search_total": 554,
    "search_failures": 0,
    "processing_time_in_ms": 1524,
    "processing_total": 554
  }
}

Do you have any Idea why size_archive index is empty?

Thanks

2

When I do rollup on some other field such as kubernetes.namespace (which is a keyword) it works.

I can see documents in the rollup index. Can anyone help me to add _size field to rollup index? Thanks

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.