# Metricbeat not sending data or Elasticsearch not recieving (?)

**URL:** <https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [March 2, 2023, 2:32pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863 "2023-03-02T14:32:51Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![enigmatic](https://avatars.discourse-cdn.com/v4/letter/e/898d66/32.png) [@enigmatic](https://discuss.elastic.co/u/enigmatic)\
**Post date:** [March 2, 2023, 2:32pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/1 "2023-03-02T14:32:51Z")

</div>

Hi Guys,

hope you can help me out in the following.

i'm using:

- Elasticseearch 8.4.3
- metricbeat 8.4.3
- kubernetes / AWS EKS

I've deployed metricbeat in a kubernetes cluster. it gathering all sort of data and i also want it scrape a prometheus endpoint, the endpoint is available via:

```auto
testservice.default:9090

```

my prometheus module config looks like this:

```auto
    - module: prometheus
      metricsets: ["collector"]
      enabled: true
      period: 10s
      hosts: ["testservice.default:9090"]
      metrics_path: /metrics

```

so this seems to be working, as i don't see any error in the logs of metricbeat. However i don't see these logs coming in to my elasticstack. (other logs that metricbeat is collecting are being received).

i turned on debug logging in metricbeat to see if this would get me any further as to identify what the issue could be, however also here there is no error.  
i see this log line:

```auto
{"log.level":"debug","@timestamp":"2023-03-02T13:56:10.972Z","log.logger":"module","log.origin":{"file.name":"module/wrapper.go","file.line":191},"message":"Starting metricSetWrapper[module=prometheus, name=collector, host=testservice.default:9090]","service.name":"metricbeat","ecs.version":"1.6.0"}

```

witch suggest to me the module is starting.  
I also start seeing the log lines i want in the debug logs. How ever i don't see them appearing in Elasticsearch. I searched all indexes via:

```auto
GET /_search
{
  "query": {
    "query_string": {
      "query": "myapp"
    }
  }
}

```

and the log lines are not present in any index..  
the elasticsearch logs them self don't show any error's

What am i doing wrong?  
Any help or suggestion will be appreciated!

---

<div class="post-metadata">

**Author:** ![Tetiana\_Kravchenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tetiana_kravchenko/32/102683_2.png) [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)\
**Post date:** [March 6, 2023, 12:50pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/2 "2023-03-06T12:50:51Z")

</div>

Hi @enigmatic

did you try to access metrics endpoint from the metricbeat container, using the `curl`?

```auto
kubectl exec -it metricbeat-pod bash
$ curl testservice.default:9090/metrics

```

are you getting a valid response?

---

<div class="post-metadata">

**Author:** ![enigmatic](https://avatars.discourse-cdn.com/v4/letter/e/898d66/32.png) [@enigmatic](https://discuss.elastic.co/u/enigmatic)\
**Post date:** [March 6, 2023, 1:18pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/3 "2023-03-06T13:18:52Z")

</div>

Hi Tetiana,

Thanks for your reply, yes, i'm getting a valid response.

also in the debug log of metricbeat i can see the log lines I'm looking for.

---

<div class="post-metadata">

**Author:** ![Tetiana\_Kravchenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tetiana_kravchenko/32/102683_2.png) [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)\
**Post date:** [March 6, 2023, 1:42pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/4 "2023-03-06T13:42:22Z")

</div>

> However i don't see these logs coming in to my elasticstack. (other logs that metricbeat is collecting are being received).

could you clarify it?  
do you have one metricbeat container, that collect prometheus metrics and metrics from other application(s)? both modules are using the same `output` configuration?

do you install metricbeat using DaemonSet, similar to this [example](https://github.com/elastic/beats/blob/main/deploy/kubernetes/metricbeat-kubernetes.yaml)?

---

<div class="post-metadata">

**Author:** ![enigmatic](https://avatars.discourse-cdn.com/v4/letter/e/898d66/32.png) [@enigmatic](https://discuss.elastic.co/u/enigmatic)\
**Post date:** [March 6, 2023, 2:03pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/5 "2023-03-06T14:03:46Z")

</div>

Hi Tetiana,

yes, i deploy metricbeat via a daemonset (like in the link)

i have 2 nodes running at the moment. on each node a container with metricbeat is running.  
They are using indeed the same output configuration.  
I have the kubernetes, system and prometheus modules running.

not sure if relevant, but the containers also elect a leadership, in the logs that becomes clear.

Kr,  
Nathan

---

<div class="post-metadata">

**Author:** ![enigmatic](https://avatars.discourse-cdn.com/v4/letter/e/898d66/32.png) [@enigmatic](https://discuss.elastic.co/u/enigmatic)\
**Post date:** [March 8, 2023, 1:06pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/6 "2023-03-08T13:06:31Z")

</div>

so i added Prometheus/Grafana within the same cluster and was able to get/see the data i want  
(using the autodiscover function).  
i 'm completely lost as to why this isn't working with metricbeat.

Kr,  
Nathan

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 8, 2023, 2:51pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/7 "2023-03-08T14:51:44Z")

</div>

Can you go to Kibana Discover and see if there are any documents with the data view `metricbeat-*` or `metrics-*`?

If so are there any fields that start with `prometheus`

---

<div class="post-metadata">

**Author:** ![enigmatic](https://avatars.discourse-cdn.com/v4/letter/e/898d66/32.png) [@enigmatic](https://discuss.elastic.co/u/enigmatic)\
**Post date:** [March 8, 2023, 3:27pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/8 "2023-03-08T15:27:26Z")

</div>

Hi @stephenb  
Thanks for your reply!  
There aren't any fields in the index that start with prometheus.

edit, this is not entirely true.  
i have some other applications on prem where i use the prometheus module.  
those are ingested successfully in elasticsearch.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 8, 2023, 3:39pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/9 "2023-03-08T15:39:38Z")

</div>

But you are getting the system metrics etc?

This is just a suggestion... What I would do is temporarily disable the kubernetes and system modules in your metricbeat config.

Then set  
`logging.level: debug`

And look at the metricbeat logs...

You can also set the `-d "*"` flag on the metribeat command in the config that will even produce more output...

> [@enigmatic](#):
>
> also in the debug log of metricbeat i can see the log lines I'm looking for.

What exactly does this mean? can you elaborate...

In the logs there should be some log entries that have `events.published` etc do you see those with only the prometheus modules configured?

---

<div class="post-metadata">

**Author:** ![enigmatic](https://avatars.discourse-cdn.com/v4/letter/e/898d66/32.png) [@enigmatic](https://discuss.elastic.co/u/enigmatic)\
**Post date:** [March 8, 2023, 3:52pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/10 "2023-03-08T15:52:14Z")

</div>

Hi @stephenb ,

sorry for the confusion, i just updated my post before i saw you comment.

- yes i'm getting system metrics.
- thanks for the suggestion! i will do that tomorrow morning.

in the debug logs i can see the event's that metricbeat get's from the prometheus endpoint that is configured. (based on the content of the event)

the line starts with:

```auto
{"log.level":"debug","@timestamp":"2023-03-06T09:26:22.760Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event:

```

i also see that elasticsearch is publishing events

```auto
{"log.level":"debug","@timestamp":"2023-03-06T09:26:23.158Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":247},"message":"PublishEvents: 50 events have been published to elasticsearch in 43.16418ms.","service.name":"metricbeat","ecs.version":"1.6.0"}

```

i will update further tomorrow when i disable kubernetes and system modules.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 8, 2023, 4:37pm UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/11 "2023-03-08T16:37:15Z")

</div>

> [@enigmatic](#):
>
> i also see that elasticsearch is publishing events
> 
> ```auto
> {"log.level":"debug","@timestamp":"2023-03-06T09:26:23.158Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":247},"message":"PublishEvents: 50 events have been published to elasticsearch in 43.16418ms.","service.name":"metricbeat","ecs.version":"1.6.0"}
> 
> ```

Apologies but elasticsearch does not publish event metricbeat does... so to be clear you are looking at the metricbeat logs?

If you exec into metricbeat pod you should be able to run a test of the modules this should hit the prometheus endpoint / config you have in the modules.

```auto
./metricbeat test modules
prometheus...
  collector...OK
    result: 
    {
     "@timestamp": "2023-03-08T16:32:59.461Z",
     "event": {
      "dataset": "prometheus.collector",
      "duration": 3328208,
      "module": "prometheus"
     },
     "metricset": {
      "name": "collector",
      "period": 10000
     },
     "prometheus": {
      "labels": {
       "code": "200",
       "handler": "found",
       "instance": "localhost:8080",
       "job": "prometheus",
       "le": "0.1",
       "method": "get"
      },
      "metrics": {
       "http_request_duration_seconds_bucket": 2
      }
     },
     "service": {
      "address": "http://localhost:8080/metrics",
      "type": "prometheus"
     }
    }

```

Also if you have `log.level:debug`

you should see outputs of the with `\"dataset\": \"prometheus.collector\",\n `

```auto
{"log.level":"debug","@timestamp":"2023-03-08T08:35:45.366-0800","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n \"@timestamp\": \"2023-03-08T16:35:45.362Z\",\n \"@metadata\": {\n \"beat\": \"metricbeat\",\n \"type\": \"_doc\",\n \"version\": \"8.4.3\"\n },\n \"event\": {\n \"dataset\": \"prometheus.collector\",\n \"module\": \"prometheus\",\n \"duration\": 3688206\n },\n \"metricset\": {\n \"name\": \"collector\",\n \"period\": 10000\n },\n \"prometheus\": {\n \"labels\": {\n \"handler\": \"found\",\n \"method\": \"get\",\n \"le\": \"2.5\",\n \"instance\": \"localhost:8080\",\n \"job\": \"prometheus\",\n \"code\": \"200\"\n },\n \"metrics\": {\n \"http_request_duration_seconds_bucket\": 2\n }\n },\n \"service\": {\n \"address\": \"http://localhost:8080/metrics\",\n \"type\": \"prometheus\"\n },\n \"host\": {\n \"os\": {\n \"name\": \"macOS\",\n \"kernel\": \"21.6.0\",\n \"build\": \"21G419\",\n \"type\": \"macos\",\n \"platform\": \"darwin\",\n \"version\": \"12.6.3\",\n \"family\": \"darwin\"\n },\n \"id\": \"9E46F076-B7F1-53AA-921B-C2F983746B79\",\n \"name\": \"hyperion\",\n \"ip\": [\n \"fe80::aede:48ff:fe00:1122\",\n \"fe80::183a:c2d4:bc27:f1ec\",\n \"192.168.1.159\",\n \"fe80::3468:7ff:fe33:fcee\",\n \"fe80::3468:7ff:fe33:fcee\",\n 

```

Also Look in the looks for a mapping exceptions

---

<div class="post-metadata">

**Author:** ![enigmatic](https://avatars.discourse-cdn.com/v4/letter/e/898d66/32.png) [@enigmatic](https://discuss.elastic.co/u/enigmatic)\
**Post date:** [March 9, 2023, 8:34am UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/12 "2023-03-09T08:34:16Z")

</div>

@stephenb ,

> [@stephenb](#):
>
> Apologies but elasticsearch does not publish event metricbeat does... so to be clear you are looking at the metricbeat logs?

sorry for the confusion I'm looking indeed in the metricbeat logs

This morning i disabled all modules, and looked at the debug logs again, below information is based on that.

> [@stephenb](#):
>
> If you exec into metricbeat pod you should be able to run a test of the modules this should hit the prometheus endpoint / config you have in the modules.

i did this, the module is working. The output of the command is:

```auto
prometheus...
  collector...OK
    result:
    {
     "@timestamp": "2023-03-09T08:21:48.304Z",
     "event": {
      "dataset": "prometheus.collector",
      "duration": 18134449,
      "module": "prometheus"
     },
     "metricset": {
      "name": "collector",
      "period": 10000
     },
     "prometheus": {
      "labels": {
       "instance": "testservice.default:9090",
       "job": "prometheus",
       "port": "6001",
       "space": "code"
      },
      "metrics": {
       "app_nodejs_heap_space_size_available_bytes": 26432,
       "app_nodejs_heap_space_size_total_bytes": 1155072,
       "app_nodejs_heap_space_size_used_bytes": 1046720
      }
     },
     "service": {
      "address": "http://testservice.default:9090/metrics",
      "type": "prometheus"
     }
    }

```

> [@stephenb](#):
>
> you should see outputs of the with `\"dataset\": \"prometheus.collector\",\n `

correct, i can see these events in the debug logs.

i dont see any exception / mapping error in the logs.

there are a few errors in the debug logs, but they are related to add\_cloud\_metadata, like:

```auto
{"log.level":"debug","@timestamp":"2023-03-09T08:14:28.469Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":167},"message":"add_cloud_metadata: received disposition for huawei after 1.491925ms. result=[provider:huawei, error=failed with http status code 401, metadata={}]","service.name":"metricbeat","ecs.version":"1.6.0"}

```

but i expect this, as the node / cluster is in AWS and not Huawei.

---

<div class="post-metadata">

**Author:** ![enigmatic](https://avatars.discourse-cdn.com/v4/letter/e/898d66/32.png) [@enigmatic](https://discuss.elastic.co/u/enigmatic)\
**Post date:** [March 9, 2023, 9:06am UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/13 "2023-03-09T09:06:34Z")

</div>

so i started looking furthere in elasticsearch, looking if i have overlooked something, and it turns out i was searching for the wrong data!  
i was searching for the appname (as i've given the logs a special prefix) , how ever this prefix was used in the fieldname in elasticsearch, so i never looked at it correctly 😑

@stephenb thanks for your help, the suggestion to disable all other modules really did the trick for me!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2023, 11:07am UTC](https://discuss.elastic.co/t/metricbeat-not-sending-data-or-elasticsearch-not-recieving/326863/14 "2023-04-06T11:07:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.