Unable to point metricbeat to monitor elasticsearch instance running on custom port

Elastic Stack Beats
1

Hi,

I am trying to configure metricbeat-oss to monitor ES-OSS instance hosted on custom port i.e. 7077 (which is further forwarded to 9200 in docker instance configured on localhost). However I am unable to monitor as expected & it's throwing below error. I guess it is not referring to custom port 7077. Instead it is trying default port i.e. 9200

Any possible hints, please ?

metricbeat-oss logs:

2020-08-05T17:22:46.969Z	INFO	[esclientleg]	eslegclient/connection.go:306	Attempting to connect to Elasticsearch version 7.8.0
2020-08-05T17:22:46.970Z	INFO	[esclientleg]	eslegclient/connection.go:306	Attempting to connect to Elasticsearch version 7.8.0
2020-08-05T17:22:46.971Z	INFO	template/load.go:89	Template metricbeat-7.8.0 already exists and will not be overwritten.
2020-08-05T17:22:46.971Z	INFO	[index-management]	idxmgmt/std.go:296	Loaded index template.
2020-08-05T17:22:46.972Z	INFO	[publisher_pipeline_output]	pipeline/output.go:152	Connection to backoff(elasticsearch(http://172.17.0.1:7077)) established
2020-08-05T17:22:55.966Z	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.node_stats: error making http request: Get http://localhost:9200/_nodes/_local/stats: dial tcp 127.0.0.1:9200: connect: connection refused
2020-08-05T17:22:55.967Z	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.node: error making http request: Get http://localhost:9200/_nodes/_local: dial tcp 127.0.0.1:9200: connect: connection refused
2020-08-05T17:22:55.968Z	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.pending_tasks: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: dial tcp 127.0.0.1:9200: connect: connection refused
2020-08-05T17:22:55.968Z	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: dial tcp 127.0.0.1:9200: connect: connection refused
2020-08-05T17:23:05.966Z	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.node_stats: error making http request: Get http://localhost:9200/_nodes/_local/stats: dial tcp 127.0.0.1:9200: connect: connection refused
2020-08-05T17:23:05.967Z	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.node: error making http request: Get http://localhost:9200/_nodes/_local: dial tcp 127.0.0.1:9200: connect: connection refused
2020-08-05T17:23:05.967Z	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: dial tcp 127.0.0.1:9200: connect: connection refused
2020-08-05T17:23:05.968Z	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.pending_tasks: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: dial tcp 127.0.0.1:9200: connect: connection refused

Output of metricbeat export config

cloud_user@vijaygharge1c:~/Downloads/metricbeat-7.8.0-linux-x86_64$ sudo -s ./metricbeat export config
[sudo] password for cloud_user: 
metricbeat:
  config:
    modules:
      path: /home/cloud_user/Downloads/metricbeat-7.8.0-linux-x86_64/modules.d/*.yml
      reload:
        enabled: false
  modules:
  - core:
      metrics:
      - percentages
    cpu:
      metrics:
      - percentages
      - normalized_percentages
    enabled: true
    metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - uptime
    - socket_summary
    - core
    - diskio
    - filesystem
    - fsstat
    - socket
    - service
    module: system
    period: 10s
    processes:
     - .*
- hosts: 
- http://172.17.0.1:7077 
metricsets: 
- node 
- node_stats 
- index 
- index_recovery 
- index_summary 
- shard 
module: elasticsearch
period: 10s
  - enabled: true
    hosts:
    - unix:///var/run/docker.sock
    metricsets:
    - container
    - cpu
    - diskio
    - event
    - healthcheck
    - info
    - image
    - memory
    - network
    module: docker
    period: 10s
  - enabled: true
    metricsets:
    - pageinfo
    - ksm
    module: linux
    period: 10s
output:
  elasticsearch:
    hosts:
    - 172.17.0.1:7077
path:
  config: /home/cloud_user/Downloads/metricbeat-7.8.0-linux-x86_64
  data: /home/cloud_user/Downloads/metricbeat-7.8.0-linux-x86_64/data
  home: /home/cloud_user/Downloads/metricbeat-7.8.0-linux-x86_64
  logs: /home/cloud_user/Downloads/metricbeat-7.8.0-linux-x86_64/logs
processors:
- add_host_metadata: null
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup:
  kibana: null
  template:
    settings:
      index:
        codec: best_compression
        number_of_shards: 1
cloud_user@vijaygharge1c:~/Downloads/metricbeat-7.8.0-linux-x86_64$

Output of ES query on custom port (7077)

cloud_user@vijaygharge1c:~/Downloads/metricbeat-7.8.0-linux-x86_64$ curl http://localhost:7077
{
  "name" : "elk",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "dugWGEYQRUCs4s5BIyOqpQ",
  "version" : {
    "number" : "7.8.0",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "757314695644ea9a1dc2fecd26d1a43856725e65",
    "build_date" : "2020-06-14T19:35:50.234439Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
cloud_user@vijaygharge1c:~/Downloads/metricbeat-7.8.0-linux-x86_64$
2

Hi,

I discovered that there is one more configuration for ES inside metricbeat/modules.d/elasticsearch.yaml.

9200 port was configured in that file.

Changing it resolved issue.

BTW, what is the purpose of having 2 different configuration for ES instance configuration i.e. one in metricbeat.yml & other one in modules.d/elasticsearch.yml ?

Thanks once again!
Vijay

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.