Prometheus to Elasticsearch using remote_write - server returned HTTP status 405 method not allowed

Elastic Stack Elasticsearch
1

Hi All,

I am working on sending data from prometheus to elasticsearch using remote_write in k8s and I am using Prometheus 2.28 image.

Below is the remote write configuration defined in configMap (k8s).

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: monitor-config
data:
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    scrape_configs:
      - job_name: elasticsearch
        scrape_interval: 10s
        metrics_path: "/_prometheus/metrics"
        scheme: https
        static_configs:
        - targets:
          - elasticsearch-cluster-master:9200
          - elasticsearch-cluster-data:9200
        tls_config:
          insecure_skip_verify: true
        basic_auth:
          username: 'admin'
          password: 'admin'
    remote_write:
      - url: "https://elasticsearch-cluster-master:9200/write"
        remote_timeout: 30s
        queue_config:
          capacity: 10000
          max_shards: 10
          min_shards: 2
          max_samples_per_send: 5000
          batch_send_deadline: 30m
          min_backoff: 50ms
          max_backoff: 1s
        basic_auth:
          username: 'admin'
          password: 'admin'
        tls_config:
          insecure_skip_verify: true
  datasource.yml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://127.0.0.1:9090

Once the prometheus pod starts running below is the error I am getting in pod logs,

ts=2022-04-18T09:35:06.123Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg=“non-recoverable error” count=495 exemplarCount=0 err=“server returned HTTP status 405 Method Not Allowed: {“error”:“Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]”,“status”:405}”**

[root@k8master1 prometheus-grafana]# kubectl logs monitoring-6c7546db6c-h9djx -n es -c prometheus -f
level=info ts=2022-04-18T09:34:16.086Z caller=main.go:148 msg="Experimental remote-write-receiver enabled"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:388 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:426 msg="Starting Prometheus" version="(version=2.28.0, branch=HEAD, revision=ff58416a0b0224bab1f38f949f7d7c2a0f658940)"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:431 build_context="(go=go1.16.5, user=root@32b9079a2740, date=20210621-15:45:36)"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:432 host_details="(Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 monitoring-6c7546db6c-h9djx (none))"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:433 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:434 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2022-04-18T09:34:16.094Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2022-04-18T09:34:16.095Z caller=main.go:807 msg="Starting TSDB ..."
ts=2022-04-18T09:34:16.098Z caller=log.go:124 component=web level=info msg="TLS is disabled." http2=false
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=6.709µs
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.859µs wal_replay_duration=322.23µs total_replay_duration=364.154µs
level=info ts=2022-04-18T09:34:16.104Z caller=main.go:834 fs_type=EXT4_SUPER_MAGIC
level=info ts=2022-04-18T09:34:16.104Z caller=main.go:837 msg="TSDB started"
level=info ts=2022-04-18T09:34:16.104Z caller=main.go:964 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2022-04-18T09:34:16.111Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Starting WAL watcher" queue=e3afd4
ts=2022-04-18T09:34:16.111Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Starting scraped metadata watcher"
ts=2022-04-18T09:34:16.111Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Replaying WAL" queue=e3afd4
level=info ts=2022-04-18T09:34:16.112Z caller=main.go:995 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=8.219377ms remote_storage=5.66082ms web_handler=637ns query_engine=877ns scrape=870.296µs scrape_sd=39.662µs notify=1.047µs notify_sd=4.678µs rules=4.199µs
level=info ts=2022-04-18T09:34:16.112Z caller=main.go:779 msg="Server is ready to receive web requests."
ts=2022-04-18T09:34:23.455Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Done replaying WAL" duration=7.344249957s
ts=2022-04-18T09:35:03.459Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="non-recoverable error" count=5000 exemplarCount=0 err="server returned HTTP status 405 Method Not Allowed: {\"error\":\"Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]\",\"status\":405}"
ts=2022-04-18T09:35:06.119Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="Remote storage resharding" from=2 to=10
**ts=2022-04-18T09:35:06.123Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="non-recoverable error" count=494 exemplarCount=0 err="server returned HTTP status 405 Method Not Allowed: {\"error\":\"Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]\",\"status\":405}"
ts=2022-04-18T09:35:06.123Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://elasticsearch-cluster-master:9200/write msg="non-recoverable error" count=495 exemplarCount=0 err="server returned HTTP status 405 Method Not Allowed: {\"error\":\"Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]\",\"status\":405}"**

Please let me know how we can resolve this issue and correct me If I am doing anything wrong in the setup.

Thanks,
Ganeshbabu R

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.