Elasticsearch Stack monitoring feature does not work when using metricbeat to monitor elasticsearch cluster

Elastic Stack Kibana
1

Hi,

I am using elasticsearch/kibana 8.7 and also using metricbeat 8.7 to monitor elasticsearch cluster in kubernetes environment.

following is the elasticsearch module config in metricbeat:

- module: elasticsearch
  xpack.enabled: true
  scope: node
  metricsets:
    - node
    - node_stats
    - index
    - index_recovery
    - index_summary
    - shard
    - cluster_stats
    - enrich
    - pending_tasks
  period: 10s
  hosts: ["Elasticsearch URL"]
  username: "${ELASTICSEARCH_USERNAME}"
  password: "${ELASTICSEARCH_PASSWORD}"
  ssl.verification_mode: "certificate"

now, metricbeat is able to get all metrics and push to elasticsearch successfully. i can see dats-stream ".monitoring-es-8-mb" got created and all relevant metrics are coming in.

i also have a dashboard setup based on this data in grafana instance and it works perfectly fine. however stack monitoring feature in kibana can not detect this monitoring data received from metricbeat.

i am not able to understand why it wont work when there is data coming in and it is correct data considering my grafana dashboard work.

any help to fix this is appreciated as stack monitoring feature is very useful to cross check grafana dashboard and also monitoring index recovery in case of issues.

2

Hi @matschaffer , pinging you as i saw you answered some other question on stack monitoring. can you please help me here?

3

Hi,

I also tried removing metricsets from module configuration as xpack feature do not require metricsets defined. i receive data in elasticsearch however stack monitoring does not work.

- hosts:
      - https://elastic.<redacted>.com:9200
      module: elasticsearch
      password: ${ELASTICSEARCH_PASSWORD}
      period: 10s
      scope: cluster
      ssl.verification_mode: certificate
      username: ${ELASTICSEARCH_USERNAME}
      xpack.enabled: true
4

Hi @chrisronline can you help me here?

5

What is the monitoring page showing you currently? Can you provide a screenshot?
Can you check your cluster settings? We'll want to check anything related to xpack.monitoring.*.
What do the Kibana/Elasticsearch logs show while you are attempting to access the monitoring page?

6

Thanks @eMitch for your reply. i have mentioned below my cluster settings especially xpack.monitoring ones

		"xpack": {
			"monitoring": {
				"elasticsearch": {
					"collection": {
						"enabled": "false"
					}
				},
				"collection": {
					"enabled": "true"
				}
			}
		}

I have also uploaded screenshot of stack monitoring page i get now.

2023-05-16_22-17-13
2023-05-16_22-17-13893×700 83.4 KB

below is my kibana.yml

#elasticsearch.serviceAccountToken: "${KIBANA_SVC_TOKEN}"
    elasticsearch.ssl.verificationMode: certificate
    monitoring.ui.enabled: true
    monitoring.ui.ccs.enabled: false
    monitoring.ui.container.elasticsearch.enabled: true
    elasticsearch.hosts: "https://elastic.<redacted>.com:9200"
    elasticsearch.username: admin
    elasticsearch.password: "password"
    xpack.encryptedSavedObjects.encryptionKey: "${KB_TOKEN}"
    xpack.security.encryptionKey: "${KB_TOKEN}"

    server.ssl.enabled: true
    server.ssl.certificate: "/usr/share/kibana/config/certs/kibana/tls.crt"
    server.ssl.key: "/usr/share/kibana/config/certs/kibana/tls.key"
    server.publicBaseUrl: "https://kibana.platdev2-obsv-eastus.bentleyhosted.com"

    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: apm
        version: latest
      - name: fleet_server
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server + APM policy
        id: fleet-server-apm-policy
        description: Fleet server policy with APM and System logs and metrics enabled
        namespace: default
        is_default_fleet_server: true
        is_managed: false
        monitoring_enabled:
          - logs
          - metrics
        package_policies:
          - name: system-1
            package:
              name: system
          - name: apm-1
            package:
              name: apm
            inputs:
              - type: apm
                keep_enabled: true
                vars:
                  - name: host
                    value: apm.<redacted>.com:8200
                    frozen: true
                  - name: url
                    value: "https://apm.<redacted>.com:8200"
                    frozen: true
                  - name: enable_rum
                    value: true
                    frozen: true
                  - name: read_timeout
                    value: 1m
                    frozen: true
                  - name: shutdown_timeout
                    value: 2m
                    frozen: true
                  - name: write_timeout
                    value: 1m
                    frozen: true
                  - name: rum_allow_headers
                    value:
                      - x-custom-header
                    frozen: true

below is my kibana logs when i try refresh on stack monitoring page..nothing seems to show up here.

541ms ramdas ❯ kube logs kibana-kibana-85948c685c-7w8pt -f
[2023-05-16T21:33:50.283+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2023-05-16T21:34:01.638+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
[2023-05-16T21:34:01.639+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2023-05-16T21:34:01.639+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
[2023-05-16T21:34:01.640+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
[2023-05-16T21:34:01.649+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2023-05-16T21:34:01.744+00:00][INFO ][http.server.Preboot] http server running at https://0.0.0.0:5601
[2023-05-16T21:34:01.788+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-05-16T21:34:01.831+00:00][WARN ][config.deprecation] Config key [xpack.fleet.agentPolicies.is_default_fleet_server] is deprecated.
[2023-05-16T21:34:01.832+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2023-05-16T21:34:02.126+00:00][INFO ][plugins-system.standard] Setting up [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2023-05-16T21:34:02.139+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
[2023-05-16T21:34:02.143+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 8479a044-d1c6-485e-967d-e3d7886727c1
[2023-05-16T21:34:02.257+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: Pku6n16VhzXDJznHw5RBRKjoGDjW8JQPRFipqKI1+s8=
[2023-05-16T21:34:02.274+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
[2023-05-16T21:34:02.400+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-05-16T21:34:02.401+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2023-05-16T21:34:02.428+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2023-05-16T21:34:02.709+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2023-05-16T21:34:03.082+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2023-05-16T21:34:03.205+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2023-05-16T21:34:03.205+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2023-05-16T21:34:03.240+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 13ms.
[2023-05-16T21:34:03.245+00:00][INFO ][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 21ms.
[2023-05-16T21:34:03.247+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 7ms.
[2023-05-16T21:34:03.250+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 5ms.
[2023-05-16T21:34:03.255+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 8ms.
[2023-05-16T21:34:03.257+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 7ms.
[2023-05-16T21:34:03.259+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 4ms.
[2023-05-16T21:34:03.260+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 1ms.
[2023-05-16T21:34:03.261+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 1ms.
[2023-05-16T21:34:03.261+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 34ms
[2023-05-16T21:34:03.262+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 5ms.
[2023-05-16T21:34:03.263+00:00][INFO ][savedobjects-service] [.kibana] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 1ms.
[2023-05-16T21:34:03.263+00:00][INFO ][savedobjects-service] [.kibana] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 0ms.
[2023-05-16T21:34:03.263+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 39ms
[2023-05-16T21:34:03.268+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2023-05-16T21:34:03.269+00:00][INFO ][plugins-system.standard] Starting [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2023-05-16T21:34:04.583+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Sender-1.1.0 scheduled with interval 1h
[2023-05-16T21:34:04.608+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
[2023-05-16T21:34:06.485+00:00][INFO ][http.server.Kibana] http server running at https://0.0.0.0:5601
[2023-05-16T21:34:06.555+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Logger-Task scheduled with interval 15m
[2023-05-16T21:34:06.615+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
[2023-05-16T21:34:06.616+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2023-05-16T21:34:06.632+00:00][INFO ][plugins.ruleRegistry] Installed common resources shared between all indices
[2023-05-16T21:34:06.632+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.uptime.alerts
[2023-05-16T21:34:06.632+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-security.alerts
[2023-05-16T21:34:06.633+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .preview.alerts-security.alerts
[2023-05-16T21:34:06.633+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.logs.alerts
[2023-05-16T21:34:06.633+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.metrics.alerts
[2023-05-16T21:34:06.633+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.apm.alerts
[2023-05-16T21:34:06.659+00:00][INFO ][status] Kibana is now degraded
[2023-05-16T21:34:06.668+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.apm.alerts
[2023-05-16T21:34:06.669+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.metrics.alerts
[2023-05-16T21:34:06.670+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.logs.alerts
[2023-05-16T21:34:06.672+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-security.alerts
[2023-05-16T21:34:06.673+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.uptime.alerts
[2023-05-16T21:34:06.721+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts
[2023-05-16T21:34:07.026+00:00][INFO ][plugins.fleet] Fleet setup completed
[2023-05-16T21:34:07.047+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
[2023-05-16T21:34:07.431+00:00][INFO ][plugins.synthetics] Installed synthetics index templates
[2023-05-16T21:34:07.743+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2023-05-16T21:34:09.735+00:00][INFO ][status] Kibana is now available (was degraded)
[2023-05-16T21:35:24.797+00:00][INFO ][plugins.ruleRegistry] Installing namespace-level resources and creating concrete index for .alerts-observability.uptime.alerts-default
[2023-05-16T21:36:51.674+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}}
7

Thanks @ramdas. How do the metricbeat logs look?

In your Metricbeat configuration, do you have the module path specified?
for example, this is what my metricbeat config for a dev cluster on docker with docker-compose.
When standing the cluster up from scratch, monitoring is working correctly.

if you don't already have the metricbeat.config.modules:... section, can you add it and reload metricbeat?

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

metricbeat.modules:
- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://es01:9200"]
  ssl.certificate_authorities: "certs/ca/ca.crt"
  ssl.certificate: "certs/es01/es01.crt"
  ssl.key: "certs/es01/es01.key"
  username: 'elastic'
  password: 'changeme'
  ssl.enabled: true

- module: logstash
  xpack.enabled: true
  period: 10s
  hosts: ["http://logstash01:9600"]

- module: kibana
  metricsets: 
    - stats
  period: 10s
  hosts: ["http://kibana:5601"]
  username: 'elastic'
  password: 'changeme'
  xpack.enabled: true

- module: docker
  metricsets:
    - "container"
    - "cpu"
    - "diskio"
    - "healthcheck"
    - "info"
    #- "image"
    - "memory"
    - "network"
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

processors:
  - add_host_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: 'https://es01:9200'
  username: 'elastic'
  password: 'changeme'
  ssl:
    certificate: "certs/es01/es01.crt"
    certificate_authorities: "certs/ca/ca.crt"
    key: "certs/es01/es01.key"
8

Thanks @eMitch for your response. i am using both elasticsearch and kibana modules in my metricbeat.modules config. below is my config and also logs below that.

metricbeat.modules:
  - hosts:
      - https://elastic.<redacted>.com:9200
      module: elasticsearch
      password: iepbLQn$^3sgq6#X
      period: 10s
      scope: cluster
      ssl.verification_mode: certificate
      username: admin
      xpack.enabled: true

    - module: kibana
      metricsets:
        - stats
      period: 10s
      hosts: ["https://kibana.<redacted>:5601"]
      xpack.enabled: true
      username: admin
      password: iepbLQn$^3sgq6#X

metricbeat logs:

{"log.level":"info","@timestamp":"2023-05-18T09:34:00.302Z","log.origin":{"file.name":"instance/beat.go","file.line":724},"message":"Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.302Z","log.origin":{"file.name":"instance/beat.go","file.line":732},"message":"Beat ID: f380f42c-81e8-4005-9f84-8f1361f64e26","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.303Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":69},"message":"Starting stats endpoint","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.303Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":71},"message":"Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.303Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.303Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1096},"message":"Beat info","service.name":"metricbeat","system_info":{"beat":{"path":{"config":"/usr/share/metricbeat","data":"/usr/share/metricbeat/data","home":"/usr/share/metricbeat","logs":"/usr/share/metricbeat/logs"},"type":"metricbeat","uuid":"f380f42c-81e8-4005-9f84-8f1361f64e26"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.303Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1105},"message":"Build info","service.name":"metricbeat","system_info":{"build":{"commit":"a8dbc6c06381f4fe33a5dc23906d63c04c9e2444","libbeat":"8.7.0","time":"2023-03-23T00:44:47.000Z","version":"8.7.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.303Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Go runtime info","service.name":"metricbeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":8,"version":"go1.19.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.305Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1114},"message":"Host info","service.name":"metricbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-05-15T21:17:11Z","containerized":true,"name":"aks-nodepool1-30653362-vmss000009","ip":["127.0.0.1","::1","172.19.54.154","fe80::222:48ff:fe2a:9248","fe80::a8aa:aaff:feaa:aaaa","fe80::a8aa:aaff:feaa:aaaa","fe80::a8aa:aaff:feaa:aaaa","fe80::a8aa:aaff:feaa:aaaa","fe80::a8aa:aaff:feaa:aaaa","fe80::a8aa:aaff:feaa:aaaa"],"kernel_version":"5.4.0-1103-azure","mac":["00:22:48:2a:92:48","00:22:48:2a:92:48","aa:aa:aa:aa:aa:aa","aa:aa:aa:aa:aa:aa","aa:aa:aa:aa:aa:aa","aa:aa:aa:aa:aa:aa","aa:aa:aa:aa:aa:aa","aa:aa:aa:aa:aa:aa"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.305Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1143},"message":"Process info","service.name":"metricbeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null},"cwd":"/usr/share/metricbeat","exe":"/usr/share/metricbeat/metricbeat","name":"metricbeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2023-05-18T09:33:59.690Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.305Z","log.origin":{"file.name":"instance/beat.go","file.line":297},"message":"Setup Beat: metricbeat; Version: 8.7.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.308Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.308Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://elastic.<>:9200","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.308Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: aks-nodepool1-30653362-vmss000009","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.329Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.366Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.384Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.405Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.420Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.428Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.434Z","log.logger":"kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node aks-nodepool1-30653362-vmss000009 discovered by NODE_NAME environment variable","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.441Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.454Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.462Z","log.logger":"kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node aks-nodepool1-30653362-vmss000009 discovered by NODE_NAME environment variable","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.475Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.478Z","log.origin":{"file.name":"util/kubernetes.go","file.line":649},"message":"could not retrieve cluster metadata: fail to get kubernetes cluster metadata: unable to retrieve cluster identifiers","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.478Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.481Z","log.origin":{"file.name":"util/kubernetes.go","file.line":649},"message":"could not retrieve cluster metadata: fail to get kubernetes cluster metadata: unable to retrieve cluster identifiers","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.484Z","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":73},"message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.24.9","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.484Z","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":73},"message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.24.9","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.484Z","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":73},"message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.24.9","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.484Z","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":73},"message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.24.9","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.484Z","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":73},"message":"add_kubernetes_metadata: kubernetes env detected, with version: v1.24.9","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.485Z","log.logger":"cfgwarn","log.origin":{"file.name":"sysinit/init.go","file.line":79},"message":"DEPRECATED: The --system.hostfs flag will be removed in the future and replaced by a config value. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.485Z","log.origin":{"file.name":"filesystem/filesystem.go","file.line":70},"message":"Ignoring filesystem types: sysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, efivarfs, mqueue, pstore, autofs, rpc_pipefs, binfmt_misc, overlay, aufs","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.485Z","log.logger":"system.fsstat","log.origin":{"file.name":"fsstat/fsstat.go","file.line":60},"message":"Ignoring filesystem types: %ssysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, efivarfs, mqueue, pstore, autofs, rpc_pipefs, binfmt_misc, overlay, aufs","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:00.485Z","log.logger":"cfgwarn","log.origin":{"file.name":"sysinit/init.go","file.line":79},"message":"DEPRECATED: The --system.hostfs flag will be removed in the future and replaced by a config value. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.493Z","log.logger":"kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node aks-nodepool1-30653362-vmss000009 discovered by NODE_NAME environment variable","service.name":"metricbeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.494Z","log.logger":"kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node aks-nodepool1-30653362-vmss000009 discovered by NODE_NAME environment variable","service.name":"metricbeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.495Z","log.logger":"kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node aks-nodepool1-30653362-vmss000009 discovered by NODE_NAME environment variable","service.name":"metricbeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.495Z","log.logger":"kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node aks-nodepool1-30653362-vmss000009 discovered by NODE_NAME environment variable","service.name":"metricbeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.495Z","log.logger":"kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node aks-nodepool1-30653362-vmss000009 discovered by NODE_NAME environment variable","service.name":"metricbeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.601Z","log.logger":"autodiscover.pod","log.origin":{"file.name":"kubernetes/util.go","file.line":146},"message":"kubernetes: Node aks-nodepool1-30653362-vmss000009 discovered by NODE_NAME environment variable","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.608Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.608Z","log.origin":{"file.name":"instance/beat.go","file.line":486},"message":"metricbeat start running.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:00.608Z","log.logger":"autodiscover","log.origin":{"file.name":"autodiscover/autodiscover.go","file.line":118},"message":"Starting autodiscover manager","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:01.023Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:01.648Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(https://elastic.<>:9200))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:01.670Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.7.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:01.681Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.7.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:01.682Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":230},"message":"Auto ILM enable success.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:01.687Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy metricbeat exists already.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:01.689Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":115},"message":"Template \"platdev2-metrics\" already exists and will not be overwritten.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:01.689Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":266},"message":"Loaded index template.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:01.690Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(https://elastic.<>.com:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:02.601Z","log.origin":{"file.name":"http/http.go","file.line":115},"message":"Starting HTTP server on localhost:8080","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:34:04.133Z","log.logger":"elasticsearch.ccr","log.origin":{"file.name":"ccr/ccr.go","file.line":78},"message":"the CCR feature is available with a platinum or enterprise Elasticsearch license. You currently have a basic license. Either upgrade your license or remove the ccr metricset from your Elasticsearch module configuration.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:34:30.612Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"/"},"cpuacct":{"id":"/","total":{"ns":3019264724}},"memory":{"id":"/","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":130228224}}}},"cpu":{"system":{"ticks":600,"time":{"ms":600}},"total":{"ticks":2480,"time":{"ms":2480},"value":2480},"user":{"ticks":1880,"time":{"ms":1880}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":30},"info":{"ephemeral_id":"fb630bac-2abe-4aad-868f-a7868c29ae14","name":"metricbeat","uptime":{"ms":30364},"version":"8.7.0"},"memstats":{"gc_next":89457856,"memory_alloc":52997392,"memory_sys":105169176,"memory_total":332543632,"rss":217116672},"runtime":{"goroutines":635}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2066,"active":0,"batches":67,"duplicates":741,"total":2807},"read":{"bytes":891190},"type":"elasticsearch","write":{"bytes":4879268}},"pipeline":{"clients":34,"events":{"active":95,"filtered":1,"published":2902,"retry":6,"total":2903},"queue":{"acked":2807,"max_events":4096}}},"metricbeat":{"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":15,"success":15},"index":{"events":369,"success":369},"index_recovery":{"events":558,"success":558},"index_summary":{"events":3,"success":3},"node_stats":{"events":10,"success":10},"shard":{"events":741,"success":741}},"kibana":{"cluster_actions":{"events":3,"success":3},"cluster_rules":{"events":3,"success":3},"node_actions":{"events":3,"success":3},"node_rules":{"events":3,"success":3},"stats":{"events":3,"success":3}},"kubernetes":{"container":{"events":19,"success":19},"event":{"events":7,"success":7},"node":{"events":1,"success":1},"pod":{"events":14,"success":14},"state_container":{"events":399,"success":399},"state_deployment":{"events":87,"success":87},"state_node":{"events":15,"success":15},"state_pod":{"events":306,"success":306},"state_replicaset":{"events":285,"success":285},"system":{"events":3,"success":3},"volume":{"events":28,"success":28}},"system":{"cpu":{"events":1,"success":1},"filesystem":{"events":1,"success":1},"fsstat":{"events":1,"success":1},"load":{"events":1,"success":1},"memory":{"events":1,"success":1},"network":{"events":9,"success":9},"process":{"events":10,"success":10},"process_summary":{"events":1,"success":1}}},"system":{"cpu":{"cores":8},"load":{"1":0.28,"15":0.84,"5":0.42,"norm":{"1":0.035,"15":0.105,"5":0.0525}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-05-18T09:35:00.612Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":1984007104}},"memory":{"mem":{"usage":{"bytes":147742720}}}},"cpu":{"system":{"ticks":740,"time":{"ms":140}},"total":{"ticks":3680,"time":{"ms":1200},"value":3680},"user":{"ticks":2940,"time":{"ms":1060}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":33},"info":{"ephemeral_id":"fb630bac-2abe-4aad-868f-a7868c29ae14","uptime":{"ms":60366},"version":"8.7.0"},"memstats":{"gc_next":80481744,"memory_alloc":48339776,"memory_sys":8912896,"memory_total":555525600,"rss":220504064},"runtime":{"goroutines":641}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2070,"active":0,"batches":69,"duplicates":741,"total":2811},"read":{"bytes":888960},"write":{"bytes":5121128}},"pipeline":{"clients":34,"events":{"active":95,"published":2811,"total":2811},"queue":{"acked":2811}}},"metricbeat":{"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":15,"success":15},"index":{"events":369,"success":369},"index_recovery":{"events":558,"success":558},"index_summary":{"events":3,"success":3},"node_stats":{"events":15,"success":15},"shard":{"events":741,"success":741}},"kibana":{"cluster_actions":{"events":3,"success":3},"cluster_rules":{"events":3,"success":3},"node_actions":{"events":3,"success":3},"node_rules":{"events":3,"success":3},"stats":{"events":3,"success":3}},"kubernetes":{"state_container":{"events":399,"success":399},"state_deployment":{"events":87,"success":87},"state_node":{"events":15,"success":15},"state_pod":{"events":306,"success":306},"state_replicaset":{"events":285,"success":285}}},"system":{"load":{"1":0.24,"15":0.81,"5":0.39,"norm":{"1":0.03,"15":0.1013,"5":0.0488}}}},"ecs.version":"1.6.0"}}
{"log.level":"warn","@timestamp":"2023-05-18T09:35:04.137Z","log.logger":"elasticsearch.ccr","log.origin":{"file.name":"ccr/ccr.go","file.line":78},"message":"the CCR feature is available with a platinum or enterprise Elasticsearch license. You currently have a basic license. Either upgrade your license or remove the ccr metricset from your Elasticsearch module configuration.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:35:30.612Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":2224406155}},"memory":{"mem":{"usage":{"bytes":157712384}}}},"cpu":{"system":{"ticks":1110,"time":{"ms":370}},"total":{"ticks":5110,"time":{"ms":1430},"value":5110},"user":{"ticks":4000,"time":{"ms":1060}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":36},"info":{"ephemeral_id":"fb630bac-2abe-4aad-868f-a7868c29ae14","uptime":{"ms":90366},"version":"8.7.0"},"memstats":{"gc_next":84310904,"memory_alloc":78317856,"memory_sys":4227072,"memory_total":802887992,"rss":229158912},"runtime":{"goroutines":647}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2256,"active":0,"batches":71,"duplicates":741,"total":2997},"read":{"bytes":935752},"write":{"bytes":5422000}},"pipeline":{"clients":34,"events":{"active":2,"filtered":1,"published":2904,"total":2905},"queue":{"acked":2997}}},"metricbeat":{"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":15,"success":15},"index":{"events":369,"success":369},"index_recovery":{"events":558,"success":558},"index_summary":{"events":3,"success":3},"node_stats":{"events":15,"success":15},"shard":{"events":741,"success":741}},"kibana":{"cluster_actions":{"events":3,"success":3},"cluster_rules":{"events":3,"success":3},"node_actions":{"events":3,"success":3},"node_rules":{"events":3,"success":3},"stats":{"events":3,"success":3}},"kubernetes":{"container":{"events":19,"success":19},"event":{"events":6,"success":6},"node":{"events":1,"success":1},"pod":{"events":14,"success":14},"state_container":{"events":399,"success":399},"state_deployment":{"events":87,"success":87},"state_node":{"events":15,"success":15},"state_pod":{"events":306,"success":306},"state_replicaset":{"events":285,"success":285},"system":{"events":3,"success":3},"volume":{"events":30,"success":30}},"system":{"cpu":{"events":1,"success":1},"filesystem":{"events":1,"success":1},"fsstat":{"events":1,"success":1},"load":{"events":1,"success":1},"memory":{"events":1,"success":1},"network":{"events":10,"success":10},"process":{"events":5,"success":5},"process_summary":{"events":1,"success":1}}},"system":{"load":{"1":0.42,"15":0.81,"5":0.43,"norm":{"1":0.0525,"15":0.1013,"5":0.0538}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-05-18T09:36:00.611Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":1908027330}},"memory":{"mem":{"usage":{"bytes":159199232}}}},"cpu":{"system":{"ticks":1260,"time":{"ms":150}},"total":{"ticks":6250,"time":{"ms":1140},"value":6250},"user":{"ticks":4990,"time":{"ms":990}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":36},"info":{"ephemeral_id":"fb630bac-2abe-4aad-868f-a7868c29ae14","uptime":{"ms":120365},"version":"8.7.0"},"memstats":{"gc_next":87348360,"memory_alloc":58845024,"memory_sys":262144,"memory_total":1005607992,"rss":228585472},"runtime":{"goroutines":647}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2081,"active":0,"batches":70,"duplicates":741,"total":2822},"read":{"bytes":891859},"write":{"bytes":5135315}},"pipeline":{"clients":34,"events":{"active":1,"published":2821,"total":2821},"queue":{"acked":2822}}},"metricbeat":{"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":15,"success":15},"index":{"events":369,"success":369},"index_recovery":{"events":558,"success":558},"index_summary":{"events":3,"success":3},"node_stats":{"events":15,"success":15},"shard":{"events":741,"success":741}},"kibana":{"cluster_actions":{"events":3,"success":3},"cluster_rules":{"events":3,"success":3},"node_actions":{"events":3,"success":3},"node_rules":{"events":3,"success":3},"stats":{"events":3,"success":3}},"kubernetes":{"event":{"events":10,"success":10},"state_container":{"events":399,"success":399},"state_deployment":{"events":87,"success":87},"state_node":{"events":15,"success":15},"state_pod":{"events":306,"success":306},"state_replicaset":{"events":285,"success":285}}},"system":{"load":{"1":0.37,"15":0.8,"5":0.42,"norm":{"1":0.0463,"15":0.1,"5":0.0525}}}},"ecs.version":"1.6.0"}}
{"log.level":"warn","@timestamp":"2023-05-18T09:36:06.348Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:36:14.135Z","log.logger":"elasticsearch.ccr","log.origin":{"file.name":"ccr/ccr.go","file.line":78},"message":"the CCR feature is available with a platinum or enterprise Elasticsearch license. You currently have a basic license. Either upgrade your license or remove the ccr metricset from your Elasticsearch module configuration.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-18T09:36:30.612Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":2357965064}},"memory":{"mem":{"usage":{"bytes":157483008}}}},"cpu":{"system":{"ticks":1670,"time":{"ms":410}},"total":{"ticks":7820,"time":{"ms":1570},"value":7820},"user":{"ticks":6150,"time":{"ms":1160}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":37},"info":{"ephemeral_id":"fb630bac-2abe-4aad-868f-a7868c29ae14","uptime":{"ms":150365},"version":"8.7.0"},"memstats":{"gc_next":83684984,"memory_alloc":43897304,"memory_sys":4521984,"memory_total":1256961784,"rss":227368960},"runtime":{"goroutines":649}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2160,"active":0,"batches":70,"duplicates":741,"total":2901},"read":{"bytes":911606},"write":{"bytes":5310354}},"pipeline":{"clients":34,"events":{"active":1,"filtered":1,"published":2901,"total":2902},"queue":{"acked":2901}}},"metricbeat":{"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":15,"success":15},"index":{"events":369,"success":369},"index_recovery":{"events":558,"success":558},"index_summary":{"events":3,"success":3},"node_stats":{"events":15,"success":15},"shard":{"events":741,"success":741}},"kibana":{"cluster_actions":{"events":3,"success":3},"cluster_rules":{"events":3,"success":3},"node_actions":{"events":3,"success":3},"node_rules":{"events":3,"success":3},"stats":{"events":3,"success":3}},"kubernetes":{"container":{"events":18,"success":18},"event":{"events":5,"success":5},"node":{"events":1,"success":1},"pod":{"events":13,"success":13},"state_container":{"events":399,"success":399},"state_deployment":{"events":87,"success":87},"state_node":{"events":15,"success":15},"state_pod":{"events":306,"success":306},"state_replicaset":{"events":285,"success":285},"system":{"events":3,"success":3},"volume":{"events":29,"success":29}},"system":{"cpu":{"events":1,"success":1},"filesystem":{"events":1,"success":1},"fsstat":{"events":1,"success":1},"load":{"events":1,"success":1},"memory":{"events":1,"success":1},"network":{"events":10,"success":10},"process":{"events":6,"success":6},"process_summary":{"events":1,"success":1}}},"system":{"load":{"1":0.51,"15":0.79,"5":0.44,"norm":{"1":0.0638,"15":0.0988,"5":0.055}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-05-18T09:37:00.611Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":1833849424}},"memory":{"mem":{"usage":{"bytes":164655104}}}},"cpu":{"system":{"ticks":1800,"time":{"ms":130}},"total":{"ticks":8890,"time":{"ms":1070},"value":8890},"user":{"ticks":7090,"time":{"ms":940}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":35},"info":{"ephemeral_id":"fb630bac-2abe-4aad-868f-a7868c29ae14","uptime":{"ms":180364},"version":"8.7.0"},"memstats":{"gc_next":87600176,"memory_alloc":55319304,"memory_sys":4194304,"memory_total":1458035744,"rss":233480192},"runtime":{"goroutines":645}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1977,"active":0,"batches":66,"duplicates":741,"total":2718},"read":{"bytes":865292},"write":{"bytes":5013267}},"pipeline":{"clients":34,"events":{"active":95,"published":2812,"total":2812},"queue":{"acked":2718}}},"metricbeat":{"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":15,"success":15},"index":{"events":369,"success":369},"index_recovery":{"events":558,"success":558},"index_summary":{"events":3,"success":3},"node_stats":{"events":15,"success":15},"shard":{"events":741,"success":741}},"kibana":{"cluster_actions":{"events":3,"success":3},"cluster_rules":{"events":3,"success":3},"node_actions":{"events":3,"success":3},"node_rules":{"events":3,"success":3},"stats":{"events":3,"success":3}},"kubernetes":{"event":{"events":1,"success":1},"state_container":{"events":399,"success":399},"state_deployment":{"events":87,"success":87},"state_node":{"events":15,"success":15},"state_pod":{"events":306,"success":306},"state_replicaset":{"events":285,"success":285}}},"system":{"load":{"1":0.71,"15":0.8,"5":0.5,"norm":{"1":0.0888,"15":0.1,"5":0.0625}}}},"ecs.version":"1.6.0"}}
{"log.level":"warn","@timestamp":"2023-05-18T09:37:07.057Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-05-18T09:37:14.135Z","log.logger":"elasticsearch.ccr","log.origin":
9

Can you add the following to your Metricbeat config and restart metricbeat?

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

It also looks like Metricbeat is picking up the template:

Can you verify that the template isn't having an impact on where the metrics are flowing?

10

Sorry for delayed response @eMitch

I added metricbeat.config configuration you provided and restarted metricbeat but that did not help.

metricbeat is using template for other metrics such as kubernetes and other deployments on it...however metricbeat sends elasticsearch metrics to separate datastream caled ".monitoring-es-8-mb"

and my grafana dashboard built on top of this datastream works all good with the data i received in above mentioned datastream.

i can not understand why kibana is not able to make sense of this data which is already there in elastic.

11

Hi @eMitch -- I would add I'm seeing the same issue after upgrading 7.17 to 8.8. I detailed my problems here Stack Monitoring with Metricbeat 8 -- No Cluster Found but just stumbled upon this post now.

From my debugging it appears that the format of the monitoring documents Kibana is expecting, does not match with what metricbeat is sending (See attached post)
I had a fleeting thought that perhaps the intention was to have an ingest pipeline to convert to the correct format (as this has been done in the past on previous versions, e.g. xpack_monitoring_6), but I can't find anything to suggest this is the case.

12

@ramdas - can you verify that at least one of the nodes in the cluster has the Ingest role assigned to it?

GET _cat/nodes?v

look for node.role and make sure you see an i

(fyi, @Evesy - you both have a similar ticket open, but we'll do our best to keep them separate, yet all work together :slight_smile: )

13

Thanks for your response Eddie, i can confirm that all nodes in cluster have ingest role. please find attached screenshot.

Thanks,
Ramdas

2023-07-13_09-39-37
2023-07-13_09-39-371490×239 79.3 KB

14

to add more context....below is the query run by kibana when i hit stack monitoring page and it returns empty result. hope this helps to get an idea what's happening.

  1. [2023-07-13T08:43:07.195+00:00][INFO ][plugins.monitoring.monitoring]
{
  "api_path": "/api/monitoring/v1/clusters",
  "referer_url": "https://kibana.<REDACTED>.bentleyhosted.com/app/monitoring",
  "query": {
    "params": {
      "method": "POST",
      "path": "/_security/user/_has_privileges",
      "body": {
        "index": [
          {
            "names": [
              ".monitoring-*"
            ],
            "privileges": [
              "read"
            ]
          }
        ]
      },
      "ignore_unavailable": true
    },
    "result": {
      "username": "elastic",
      "has_all_requested": true,
      "cluster": {},
      "index": {
        ".monitoring-*": {
          "read": true
        }
      },
      "application": {}
    }
  }
}
  1. [2023-07-13T08:43:07.777+00:00][INFO ][plugins.monitoring.monitoring]
{
  "api_path": "/api/monitoring/v1/clusters",
  "referer_url": "https://kibana.<REDACTED>.com/app/monitoring",
  "query": {
    "params": {
      "index": ".monitoring-es-*,metrics-elasticsearch.stack_monitoring.cluster_stats-*",
      "size": 10000,
      "ignore_unavailable": true,
      "filter_path": [
        "hits.hits._index",
        "hits.hits._source.cluster_uuid",
        "hits.hits._source.elasticsearch.cluster.id",
        "hits.hits._source.cluster_name",
        "hits.hits._source.elasticsearch.cluster.name",
        "hits.hits._source.version",
        "hits.hits._source.elasticsearch.version",
        "hits.hits._source.elasticsearch.cluster.node.version",
        "hits.hits._source.license.status",
        "hits.hits._source.elasticsearch.cluster.stats.license.status",
        "hits.hits._source.license.type",
        "hits.hits._source.elasticsearch.cluster.stats.license.type",
        "hits.hits._source.license.issue_date",
        "hits.hits._source.elasticsearch.cluster.stats.license.issue_date",
        "hits.hits._source.license.expiry_date",
        "hits.hits._source.elasticsearch.cluster.stats.license.expiry_date",
        "hits.hits._source.license.expiry_date_in_millis",
        "hits.hits._source.elasticsearch.cluster.stats.license.expiry_date_in_millis",
        "hits.hits._source.cluster_stats",
        "hits.hits._source.elasticsearch.cluster.stats",
        "hits.hits._source.cluster_state",
        "hits.hits._source.elasticsearch.cluster.stats.state",
        "hits.hits._source.cluster_settings.cluster.metadata.display_name"
      ],
      "body": {
        "query": {
          "bool": {
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "term": {
                        "data_stream.dataset": "elasticsearch.stack_monitoring.cluster_stats"
                      }
                    },
                    {
                      "term": {
                        "metricset.name": "cluster_stats"
                      }
                    },
                    {
                      "term": {
                        "type": "cluster_stats"
                      }
                    }
                  ]
                }
              },
              {
                "range": {
                  "timestamp": {
                    "format": "epoch_millis",
                    "gte": 1689236887115,
                    "lte": 1689237787115
                  }
                }
              }
            ]
          }
        },
        "collapse": {
          "field": "cluster_uuid"
        },
        "sort": {
          "timestamp": {
            "order": "desc",
            "unmapped_type": "long"
          }
        }
      }
    },
    "result": {}
  }
}

slight different to what @Evesy saw, i can see below field from the above query is present in indexes generated by metricbeat..however result is empty. i am not able to make more sense out of it further.

 "term": {
                        "metricset.name": "cluster_stats"
                      }
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.