Don't get doc count after setting ilm

Elastic Stack Elasticsearch
1

I am using two indices each index is pointed to ilm.
below is the index setting for two indices

 "index": {
    "lifecycle": {
      "name": "packetbeat_flow",
      "rollover_alias": "log-pb-flow"
    },
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }

 "index": {
    "lifecycle": {
      "name": "packetbeat_dns",
      "rollover_alias": "log-pb-dns"
    },
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }

ILM Policy for packetbeat_flow

PUT _ilm/policy/packetbeat_flow
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "2d",
        "actions": {
          "set_priority": {
            "priority": 50
          },
          "shrink": {
            "number_of_shards": 1
          },
          "allocate": {
            "require": {
              "data": "warm"
            },
            "number_of_replicas": 1
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

ILM Policy for packetbeat_dns

PUT _ilm/policy/packetbeat_dns
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "2d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

In logstash pipeline file

output {
 if[agent][type]=="packetbeat"
 {
if[type]=="flow"
   {
    elasticsearch {
    hosts => ["http://em1:9200","http://em2:9200"]
    manage_template => false
    ilm_rollover_alias => "log-pb-flow"
    ilm_pattern => "000001"
    ilm_policy => "packetbeat_flow"
  }
  }

if[type]=="dns"
   {
    elasticsearch {
    hosts => ["http://em1:9200","http://em2:9200"]
    manage_template => false
    ilm_rollover_alias => "log-pb-dns"
    ilm_pattern => "000001"
    ilm_policy => "packetbeat_dns"
  }
  }

}
}

In elasticsearch log file

[2021-02-12T16:03:04,567][INFO ][o.e.x.i.IndexLifecycleTransition] [em2] moving index [log-pb-flow-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-index-color"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [packetbeat_flow]
[2021-02-12T16:03:04,594][INFO ][o.e.x.i.IndexLifecycleTransition] [em2] moving index [log-pb-dns-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-index-color"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [packetbeat_dns]

i am not getting any data the doc count is 0

2

What's the output from _cat/indices?v.

3 
health status index                                uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   log-pb-flow-000001                   maExfsMhS_aDv94iLJZcIA   1   2          0            0       624b           208b
green  open   .apm-agent-configuration             zlDBqb8oQ7ae0Gw-vDVOPQ   1   1          0            0       416b           208b
green  open   .kibana_2                            lXIoyQ2qS0iNugyjnIXNMw   1   1        114           16      4.3mb          2.1mb
green  open   .kibana_1                            c1MPSH3hQMmpPCM_Oa_wGg   1   1         19            0      4.2mb          2.1mb
green  open   .monitoring-kibana-7-2021.02.11      bJFTt3-4SI6iHohNhlqPZQ   1   1         28            0    211.1kb           90kb
green  open   .monitoring-kibana-7-mb-2021.02.11   2JCoYiwjQu2wwv9yLm-ZLg   1   1       6736            0      2.8mb          1.4mb
green  open   .kibana-event-log-7.11.0-000001      GCcuHOSFSfaiDfRK_vQH4Q   1   1          1            0     11.2kb          5.6kb
green  open   .monitoring-logstash-7-mb-2021.02.12 Q2blipw0TduR_Skp7LIT7g   1   1       4830            0      1.3mb        613.4kb
green  open   .monitoring-logstash-7-mb-2021.02.14 Aw7TzjcqRom9kSGS_QcEBg   1   1      25742            0      4.6mb          2.3mb
green  open   .monitoring-es-7-mb-2021.02.12       1ej1h66GQPucFab-uu60Rg   1   1     249819            0    388.9mb        193.3mb
green  open   .apm-custom-link                     shLfP5DIR-eOJmEvovTDtw   1   1          0            0       416b           208b
green  open   metricbeat-7.11.0-2021.02.11-000001  23neCPLmTgOC1edGdMzUbg   1   1         63            0    255.9kb          128kb
green  open   .kibana_task_manager_1               el_Z2YB_QGiScTUtbr7fxQ   1   1          8       353129    220.4mb        110.3mb
green  open   .monitoring-es-7-mb-2021.02.13       7zqFQrgxRLqLoduaiCiUEw   1   1     276558            0    439.9mb        220.5mb
green  open   .monitoring-es-7-mb-2021.02.11       RZ4QiZgTRqi-PdGe5lGd-A   1   1      84659            0    127.8mb           64mb
green  open   .monitoring-es-7-mb-2021.02.14       8Rs6P4lFQ0eASQlGE3aRPQ   1   1     302645            0    468.9mb        234.9mb
green  open   .monitoring-es-7-mb-2021.02.15       xG5VNGpKQt2MZ6dRaAlsNw   1   1      43344            0     70.6mb         35.2mb
green  open   .monitoring-kibana-7-mb-2021.02.15   RfFvQdgGRe6ObQBJRmmUqw   1   1       2346            0    954.5kb        517.2kb
green  open   .monitoring-kibana-7-mb-2021.02.14   k9u5_8WYS8KLL38fW4oBwg   1   1      17280            0      6.9mb          3.4mb
green  open   .kibana-event-log-7.10.2-000001      TpCUELDLQIqC5qh8GG-dCw   1   1          1            0     11.2kb          5.6kb
green  open   .monitoring-es-7-2021.02.11          2O8dJMB8Sq-9xErk-8HrhA   1   1         29            0    322.3kb          206kb
green  open   .monitoring-kibana-7-mb-2021.02.13   sJ82zqbaRDSoekR9E6Nwuw   1   1      17280            0      6.4mb          3.2mb
green  open   .monitoring-kibana-7-mb-2021.02.12   dzBqtjsGSCG2Jw9y-KlHiA   1   1      17280            0      6.6mb          3.3mb
4

Hey @warkolm i am using one index for make it simple.Index is created but the doc count is 0

5

The problem is not solved index is created no data is coming.

6

The problem is solved it was packetbeat issue

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.