Hi All,
I am working on sending data from prometheus to elasticsearch using remote_write in k8s and I am using Prometheus 2.28 image.
Below is the remote write configuration defined in configMap (k8s).
---
kind: ConfigMap
apiVersion: v1
metadata:
name: monitor-config
data:
prometheus.yml: |-
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: elasticsearch
scrape_interval: 10s
metrics_path: "/_prometheus/metrics"
scheme: https
static_configs:
- targets:
- elasticsearch-cluster-master:9200
- elasticsearch-cluster-data:9200
tls_config:
insecure_skip_verify: true
basic_auth:
username: 'admin'
password: 'admin'
remote_write:
- url: "https://elasticsearch-cluster-master:9200/write"
remote_timeout: 30s
queue_config:
capacity: 10000
max_shards: 10
min_shards: 2
max_samples_per_send: 5000
batch_send_deadline: 30m
min_backoff: 50ms
max_backoff: 1s
basic_auth:
username: 'admin'
password: 'admin'
tls_config:
insecure_skip_verify: true
datasource.yml: |-
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: http://127.0.0.1:9090
Once the prometheus pod starts running below is the error I am getting in pod logs,
ts=2022-04-18T09:35:06.123Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg=“non-recoverable error” count=495 exemplarCount=0 err=“server returned HTTP status 405 Method Not Allowed: {“error”:“Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]”,“status”:405}”**
[root@k8master1 prometheus-grafana]# kubectl logs monitoring-6c7546db6c-h9djx -n es -c prometheus -f
level=info ts=2022-04-18T09:34:16.086Z caller=main.go:148 msg="Experimental remote-write-receiver enabled"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:388 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:426 msg="Starting Prometheus" version="(version=2.28.0, branch=HEAD, revision=ff58416a0b0224bab1f38f949f7d7c2a0f658940)"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:431 build_context="(go=go1.16.5, user=root@32b9079a2740, date=20210621-15:45:36)"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:432 host_details="(Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 monitoring-6c7546db6c-h9djx (none))"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:433 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:434 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2022-04-18T09:34:16.094Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2022-04-18T09:34:16.095Z caller=main.go:807 msg="Starting TSDB ..."
ts=2022-04-18T09:34:16.098Z caller=log.go:124 component=web level=info msg="TLS is disabled." http2=false
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=6.709µs
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.859µs wal_replay_duration=322.23µs total_replay_duration=364.154µs
level=info ts=2022-04-18T09:34:16.104Z caller=main.go:834 fs_type=EXT4_SUPER_MAGIC
level=info ts=2022-04-18T09:34:16.104Z caller=main.go:837 msg="TSDB started"
level=info ts=2022-04-18T09:34:16.104Z caller=main.go:964 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2022-04-18T09:34:16.111Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Starting WAL watcher" queue=e3afd4
ts=2022-04-18T09:34:16.111Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Starting scraped metadata watcher"
ts=2022-04-18T09:34:16.111Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Replaying WAL" queue=e3afd4
level=info ts=2022-04-18T09:34:16.112Z caller=main.go:995 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=8.219377ms remote_storage=5.66082ms web_handler=637ns query_engine=877ns scrape=870.296µs scrape_sd=39.662µs notify=1.047µs notify_sd=4.678µs rules=4.199µs
level=info ts=2022-04-18T09:34:16.112Z caller=main.go:779 msg="Server is ready to receive web requests."
ts=2022-04-18T09:34:23.455Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Done replaying WAL" duration=7.344249957s
ts=2022-04-18T09:35:03.459Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="non-recoverable error" count=5000 exemplarCount=0 err="server returned HTTP status 405 Method Not Allowed: {\"error\":\"Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]\",\"status\":405}"
ts=2022-04-18T09:35:06.119Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Remote storage resharding" from=2 to=10
**ts=2022-04-18T09:35:06.123Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="non-recoverable error" count=494 exemplarCount=0 err="server returned HTTP status 405 Method Not Allowed: {\"error\":\"Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]\",\"status\":405}"
ts=2022-04-18T09:35:06.123Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="non-recoverable error" count=495 exemplarCount=0 err="server returned HTTP status 405 Method Not Allowed: {\"error\":\"Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]\",\"status\":405}"**
Please let me know how we can resolve this issue and correct me If I am doing anything wrong in the setup.
Thanks,
Ganeshbabu R
)