Issue in metricbeat 7.12

Elastic Stack Beats
1

Hi,

I've debugging an issue for some time now where metricbeat-7.12 doesn't seem to be able to create a new alias for an index template.

About the set up:

  • Elasticsearch 7.12
  • Kibana 7.12
  • Metricbeat 7.12
  • Security enabled (SSL + authentication)

ES and Kibana are behind a reverse proxy.

This is what I see in the logs (apologies if the following doesn't show as code, something in the content seems to make the rendering fail to convert that to code):

    2021-03-26T11:56:24.155+0100    ERROR   [publisher_pipeline_output]     pipeline/output.go:154  
    Failed to connect to backoff(elasticsearch(https://<host>/elasticsearch)): Connection marked as failed because the onConnect callback failed: failed to create alias: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
404
</body></html>
    2021-03-26T11:56:24.156+0100    INFO    [publisher_pipeline_output]     pipeline/output.go:145  
    Attempting to reconnect to backoff(elasticsearch(https://<host>/elasticsearch)) with 1 reconnect attempt(s)

I initially thought that this issue was due to the proxy (since I had some 405 before) but this doesn't seem to be the case. I went to kibana and run the same request and got the following response:

    PUT <metricbeat-7.12.0-{now/d}-000001>
    {"aliases":{"metricbeat-7.12.0":{"is_write_index":true}}}

    {
      "error" : "Incorrect HTTP method for uri [/%3Cmetricbeat-7.12.0-%7Bnow/d%7D-000001%3E?pretty=true] and method [PUT], allowed: [POST]",
      "status" : 405
    }

Is metricbeat using a endpoint from Elasticsearch that has been removed (or modified)?

Any guidance on how to that issue further would be very appreciated.
Thanks

2

Hi!

Can you share your configuration and I can give it a try to see if this is a regression. Also what version of ES you are running?

C.

3

Hi!
Thanks for the offer. Here is the metricbeat config:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
fields:
  env: dev
output.elasticsearch:
  hosts: ["https://<es-host>:443/elasticsearch"]
  protocol: "https"
  username: "<metricbeat_user>"
  password: "${output.elasticsearch.password}"
  
  ssl.enabled: true
  ssl.verification_mode: full
  
  proxy_url: http://<proxy-host>:80

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

And here is my ES config:

cluster.name: monitoring-cluster
node.name: node-1
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
cluster.initial_master_nodes: ["node-1"]

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: <path_to_ssl_key>
xpack.security.http.ssl.certificate: <path_to_ssl_crt>
xpack.security.http.ssl.certificate_authorities: <path_to_ssl_crt>

Let me know if any more information is required.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.