Elasticsearch monitoring with metricbeat not working as expected

[ramdas]

Hi,

i am using metricbeat for elasticsearch monitoring and the monitoring indexes are created with pattern like .ds-.monitoring-es-8-mb-* and i can see data is also coming in. however when i go to stack monitoring it says no monitoring data found.

i can confirm there is enough monitoring data like almost a week and still it says no monitoring data found. i do not know what else to check?? any help?

[ingri.mahecha (Ingri)]

Hello,

How do you have the Metricbeat configuration?

[ramdas]

Hi,

Thanks for your response. below is my config file for metricbeat

metricbeat.yml:
setup:
  template:
    enabled: true
    name: "${INDEX_NAME:cluster-metrics}"
    pattern: "${INDEX_NAME:cluster-metrics}-*"
    fields: "fields.yml"
    settings:
      index.number_of_shards: 1
      index.number_of_replicas: 1
      index.lifecycle.name: "bmap_obsv_ilm_policy"
  ilm:
    enabled: false
  kibana:
    host: "${KIBANA_HOST:kibana.example.com:5601}"
    protocol: "${KIBANA_PROTOCOL:https}"
    ssl.verification_mode: "${KIBANA_VERIFYSSL:none}"
    username: "${KIBANA_USERNAME:kibana}"
    password: "${KIBANA_PASSWORD:welcome1}"

logging:
  level: "${LOG_LEVEL:warning}"
  to_stderr: true
  json: true

metricbeat.modules:
- module: prometheus
  metricsets: ["remote_write"]
  host: "localhost"
  port: "8080"

- module: kubernetes
  enabled: true
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_pod
    - state_container
  period: 10s
  hosts: ["${KUBE_STATE_METRICS_HOSTS:kube-state-metrics:8080}"]
  ssl.verification_mode: "none"

- module: kubernetes
  metricsets:
    - container
    - node
    - pod
    - system
    - volume
  period: 1m
  host: "${NODE_NAME}"
  hosts: ["https://${NODE_NAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.verification_mode: "none"
  processors:
  - add_kubernetes_metadata: ~

- module: kubernetes
  enabled: true
  metricsets:
    - event

- module: elasticsearch
  xpack.enabled: true
  scope: node
  metricsets:
    - node
    - node_stats
    - index
    - index_recovery
    - index_summary
    - shard
    - cluster_stats
    - enrich
    - pending_tasks
  period: 10s
  hosts: ["https://localhost:${ES_NODEPORT}"]
  username: "${ELASTICSEARCH_USERNAME}"
  password: "${ELASTICSEARCH_PASSWORD}"
  ssl.verification_mode: "certificate"

- module: system
  period: 1m
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
  processes: ['.*']
  process.include_top_n:
    by_cpu: 5
    by_memory: 5

- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat

  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

  - add_cloud_metadata: ~

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true

output.elasticsearch:
  hosts: "[${ELASTICSEARCH_HOST}:9200]"
  protocol: "${ELASTICSEARCH_PROTOCOL:https}"
  username: "${ELASTICSEARCH_USERNAME:elastic}"
  ssl.verification_mode: "${ELASTICSEARCH_VERIFYSSL:none}"
  password: "${ELASTICSEARCH_PASSWORD:welcome1}"
  index: "${INDEX_NAME}-%{[agent.version]}-%{+yyyy.MM.dd}"
  allow_older_versions: true

i am using Elasticsearch and metricbeat with version 8.3.3. hope this helps for you to identify issue

[ingri.mahecha (Ingri)]

To configure the INDEX PATTERN settings you must configure the following:

metricbeat.yml

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.template.name: "logs-rds.nameofindex-default"
setup.template.pattern: "logs-rds.nameofindex-default"
setup.ilm.enabled: false
output.elasticsearch.index: "logs-rds.nameofindex-default"

for example if you are using a module, simply leave the following configuration in the input

# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

to activate the modules I leave you the guide
[Configure modules | Metricbeat Reference [master] | Elastic](Configure modules)

Then go to your elastisearch
Menu -> Management -> Stack Management-> Data ->Index Management

In the Data Stream option you verify that data is arriving and then go to

Menu -> Management -> Stack Management-> Kibana ->Data views

Create the index pattern

To create the template for the fields of this configuration in the path
Menu -> Management -> Stack Management-> Data ->Index Management

In the Index template option look for the name of your Index and select EDIT, there in the mapping part you can add the Fields in Json format an example:

{
  "properties": {

    "@timestamp": {
      "type": "date"
    },
	
	"data_stream.​dataset":{
	"type":"constant_keyword"
	},
	
	"data_stream.​namespace":{
	"type":"constant_keyword"
	}
}
}

I hope this explanation helps you a little