Metricbeat not sending data or Elasticsearch not recieving (?)

Elastic Stack Beats
11

Apologies but elasticsearch does not publish event metricbeat does... so to be clear you are looking at the metricbeat logs?

If you exec into metricbeat pod you should be able to run a test of the modules this should hit the prometheus endpoint / config you have in the modules.

./metricbeat test modules
prometheus...
  collector...OK
    result: 
    {
     "@timestamp": "2023-03-08T16:32:59.461Z",
     "event": {
      "dataset": "prometheus.collector",
      "duration": 3328208,
      "module": "prometheus"
     },
     "metricset": {
      "name": "collector",
      "period": 10000
     },
     "prometheus": {
      "labels": {
       "code": "200",
       "handler": "found",
       "instance": "localhost:8080",
       "job": "prometheus",
       "le": "0.1",
       "method": "get"
      },
      "metrics": {
       "http_request_duration_seconds_bucket": 2
      }
     },
     "service": {
      "address": "http://localhost:8080/metrics",
      "type": "prometheus"
     }
    }

Also if you have log.level:debug

you should see outputs of the with \"dataset\": \"prometheus.collector\",\n 

{"log.level":"debug","@timestamp":"2023-03-08T08:35:45.366-0800","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n  \"@timestamp\": \"2023-03-08T16:35:45.362Z\",\n  \"@metadata\": {\n    \"beat\": \"metricbeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.4.3\"\n  },\n  \"event\": {\n    \"dataset\": \"prometheus.collector\",\n    \"module\": \"prometheus\",\n    \"duration\": 3688206\n  },\n  \"metricset\": {\n    \"name\": \"collector\",\n    \"period\": 10000\n  },\n  \"prometheus\": {\n    \"labels\": {\n      \"handler\": \"found\",\n      \"method\": \"get\",\n      \"le\": \"2.5\",\n      \"instance\": \"localhost:8080\",\n      \"job\": \"prometheus\",\n      \"code\": \"200\"\n    },\n    \"metrics\": {\n      \"http_request_duration_seconds_bucket\": 2\n    }\n  },\n  \"service\": {\n    \"address\": \"http://localhost:8080/metrics\",\n    \"type\": \"prometheus\"\n  },\n  \"host\": {\n    \"os\": {\n      \"name\": \"macOS\",\n      \"kernel\": \"21.6.0\",\n      \"build\": \"21G419\",\n      \"type\": \"macos\",\n      \"platform\": \"darwin\",\n      \"version\": \"12.6.3\",\n      \"family\": \"darwin\"\n    },\n    \"id\": \"9E46F076-B7F1-53AA-921B-C2F983746B79\",\n    \"name\": \"hyperion\",\n    \"ip\": [\n      \"fe80::aede:48ff:fe00:1122\",\n      \"fe80::183a:c2d4:bc27:f1ec\",\n      \"192.168.1.159\",\n      \"fe80::3468:7ff:fe33:fcee\",\n      \"fe80::3468:7ff:fe33:fcee\",\n

Also Look in the looks for a mapping exceptions