Configuration Prometheus data in ElasticSearch via Metricbeats

Elastic Stack Elasticsearch
1

Sub: We are looking for a solution to push the Prometheus data to Elasticsearch for long term storage.

We have deployed Elasticsearch, Metric Beat, prometheus and Grafana on a kubernetes cluster. We need to push prometheus data to ElasticSearch and the grafana needs to pull prometheus data from Elasticsearch to render the data. Please help us with steps to integrate these components.

ElasticSearch Version: 8.3.2
Metric Beat: metricbeat-7.17.3
Prometheus: 2.27.0

Regards,
Ganesh

2

Hi @tgkumarmca, have you looked at this blog post: How to implement Prometheus long-term storage using Elasticsearch | Elastic Blog ?

I believe it can help you.

3

@TiagoQueiroz,

Thanks for your reply. I have installed all components as per the documentation provided. Prometheus is not scraping the metrics and not pushing to the metricbeat. I have updated the remote write url to metricbeat service. I am getting following error:

Remote Write URL: http://metricbeat-ext.elastic-system:9201/metrics

the error logs:

ts=2022-08-01T15:04:27.769Z caller=dedupe.go:112 component=remote level=warn remote_name=ba53bf url=http://metricbeat-ext.elastic-system:9201/metrics msg="Failed to send batch, retrying" err="Post "http://metricbeat-ext.elastic-system:9201/metrics\": dial tcp 100.67.96.145:9201: connect: connection refused"

4

The log error you posted says connect: connection refused, it seems your Prometheus instance cannot connect to your Metricbeat instance.

Check the connectivity between the machines as well as the IP the domain is resolving to.

Also double check your Metricbeat configuration and Prometheus match (things like port, endpoint, etc).

If it does not work, please post here the configuration of Metricbeat and Prometheus. Also any error logs you see on Metricbeat or Prometheus.

5

I have installed metricbeat-7.17.3. I have initially scraped metrics to 9201 port of the metricbeat. Below are my configurations of prometheus and metricbeat.

I just noticed that the port 9201 is being connected when I exec into the pod and curl to the localhost:9201. but, this is not working from outside pod.

Please suggest if you come across any issue like this.

MetricBeat configuration:

  metricbeat.modules:
  - module: prometheus
    metricsets: ["remote_write"]
    host: "localhost"
    port: "9201"

Prometheus configuration as follows:

remote_write:
http://metricbeat-ext.elastic-system.svc.cluster.local:9201/metrics

6

That seem to be the problem, you need to edit your pod/deployment configuration on Kubernetes to enable access to this port.

7

Hi,
I have already updated the port in the yaml file. although, it is not working.

8

Unfortunately I can't help more because I don't have access to your environment.

But if you cannot connect from outside the pod, and it works when testing from inside the pod, it's clearly a issue outside Metricbeat and related to you Kubernetes environment. You'll have to find out why Kubernetes is not routing the traffic to this port on the container.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.