Metricbeat ILM rollover issue

Elastic Stack Beats
,
1

Hi, My metric dashboard in kibana crashed every now and then in 2 days with error

Error

Internal Server Error (500)

URL

https://kibana.integ.tvld.tech/api/metrics/snapshot

i suspect this might an issue with metric beat ILM configuration, i hjave configured ILM with life cycle jsonon metric.yml

metricbeat.config.modules:
  path: "${path.config}/modules.d/*.yml"
  reload.enabled: 'false'
setup.template.settings:
  index.number_of_shards: '1'
  index.codec: best_compression
logging.level: info
logging.to_files: 'true'
logging.files:
  path: "/var/log/metricbeat"
  name: metricbeat
  keepfiles: '5'
  permissions: '0644'
setup.kibana:
  host: http://parkbni01.infra.tvld.tech:8080
output.elasticsearch:
  hosts:
  - https://parelki03.infra.tvld.tech:9200
  - https://parelki02.infra.tvld.tech:9200
  - https://parelki01.infra.tvld.tech:9200
  protocol: https
  username: tvld_metricbeat
  password: wacAIrAurw/FcK1juPfKR5dtzfXDway2DnrUc+DbbNE=
setup.ilm.policy_name: metricbeat-rollover-10d
setup.ilm.policy_file: "/etc/metricbeat/lifecycle.json"

lifecycle.json

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "10gb",
            "max_age": "10d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "5d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "10d",
        "actions": {
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "15d",
        "actions": {
          "delete": {

          }
        }
      }
    }
  }
}

While index and index template is created it works fine. but after 2 days the iLM policy defined dissappears from the metrics.
anything i miss for config.

2

Please share the Kibana log.

4

Hi this is error i get on kibana,

Aug 24 08:46:52 parkbni01 kibana: {"type":"error","@timestamp":"2020-08-24T06:46:52Z","tags":[],"pid":73650,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:160:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/metrics/snapshot","path":"/api/metrics/snapshot","href":"/api/metrics/snapshot"},"message":"Internal Server Error"}

To resolve this error i have to stop the metricbeat and remove the index and templates and new created index will have ILM config with lifecycle.json.

5

@warkolm
WIll you be able to help :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.