Metricbeat jolokia autodiscover question

Elastic Stack Beats
,
1

this is my config , no jvm data sent to es ,so what goes wrong with this config
version : 7.6.0
metricbeat.yml

metricbeat.config.modules:
  # Mounted `metricbeat-daemonset-modules` configmap:
  path: ${path.config}/modules.d/*.yml
  # Reload module configs as they change:
  reload.enabled: false

# To enable hints based autodiscover uncomment this:
metricbeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true

processors:
  - add_cloud_metadata:

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

# jolokia.yml
metricbeat.autodiscover:
  providers:
    - type: jolokia
      templates:
        config:
        - module: jolokia
          metricsets: ["jmx"]
          hosts: "${data.host}:8080/jolokia"
          namespace: sxc
          jmx.mappings:
          - mbean: "java.lang:type=Runtime"
            attributes:
            - attr: Uptime
              field: uptime
# kubernetes.yml
- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  host: ${NODE_IP}
  hosts: ["https://${NODE_IP}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.verification_mode: "none"
      # If there is a CA bundle that contains the issuer of the certificate used in the Kubelet API,
      # remove ssl.verification_mode entry and use the CA, for instance:
      #ssl.certificate_authorities:
        #- /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    # Currently `proxy` metricset is not supported on Openshift, comment out section

Thanks in advance!

2

FYI,I expose ports through spring boot actuator,Could this be the cause

3

i try jolokia module remove autodiscover and it is worked

- module: jolokia
  metricsets: ["jmx"]
  hosts: ["http://ip:8080/jolokia"]
  period: 10s
  namespace: "sxc"
  jmx.mappings:
  - mbean: "java.lang:type=Memory"
    attributes:
      - attr: HeapMemoryUsage
        field: memory.heap_usage
      - attr: NonHeapMemoryUsage
        field: memory.non_heap_usage

I don't know why nothing works over here

4

@warkolm
Hello, sorry to disturb you, can you help me look at this problem

5

Hey @ZMMWMY,

Could you tell us more about your scenario?

I see in the initial post you were using Kubernetes autodiscover provider, are you trying to monitor Java services running in Kubernetes? Then it would be better to use the Kubernetes provider.

The Jolokia autodiscover provider is a lower level provider intended to discover services in networks where no supported orchestrator is being used.

6

@jsoriano Thanks for your reply, yes, I try to use metricbeat to monitor the java program
I have tried both kubenetes autodiscover and jolokia autodiscover, but unfortunately they failed to get results.
My Java program exposes the port through spring boot actuator jolokia module
Here is the official document
URL : http://localhost:8080/jolokia/read/java.lang:type=Memory/
This is the return value

{
    "request":{
        "mbean":"java.lang:type=Memory",
        "type":"read"
    },
    "value":{
        "ObjectPendingFinalizationCount":0,
        "Verbose":false,
        "HeapMemoryUsage":{
            "init":134217728,
            "committed":529006592,
            "max":1908932608,
            "used":171310648
        },
        "NonHeapMemoryUsage":{
            "init":2555904,
            "committed":121110528,
            "max":-1,
            "used":113887232
        },
        "ObjectName":{
            "objectName":"java.lang:type=Memory"
        }
    },
    "timestamp":1597159570,
    "status":200
}

metricbeat config

logging.level: debug

#metricbeat.config.modules:
  # Mounted `metricbeat-daemonset-modules` configmap:
#  path: ${path.config}/modules.d/*.yml
  # Reload module configs as they change:
#  reload.enabled: false

# To enable hints based autodiscover uncomment this:
metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
      - condition:
          contains:
            kubernetes.container.name: "commander"
        config:
        - module: jolokia
          metricsets: ["jmx"]
          hosts: ["http://${data.host}:8080/jolokia"]
          namespace: "sxc"
          jmx.mappings:
          - mbean: 'java.lang:type=Memory'
            attributes:
             - attr: HeapMemoryUsage
               field: memory.heap_usage
             - attr: NonHeapMemoryUsage
               field: memory.non_heap_usage
          - mbean: 'Catalina:name=*,type=ThreadPool'
            attributes:
             - attr: port
               field: catalina.port
             - attr: maxConnections
               field: catalina.maxConnections
          - mbean: 'java.lang:type=Runtime'
            attributes:
             - attr: Uptime
               field: uptime
          

processors:
  - add_cloud_metadata:
  - add_host_metadata:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
7

Configuration looks good, do you see something related on Metricbeat logs?

8

This is the log


2020-08-12T01:29:06.254Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:162	add_cloud_metadata: received disposition for gcp after 3.000183936s. result=[provider:gcp, error=failed requesting gcp metadata: Get http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), metadata={}]
2020-08-12T01:29:06.254Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:162	add_cloud_metadata: received disposition for digitalocean after 3.000366014s. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get http://169.254.169.254/metadata/v1.json: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2020-08-12T01:29:06.254Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:162	add_cloud_metadata: received disposition for openstack after 3.00045766s. result=[provider:openstack, error=failed requesting openstack metadata: Get http://169.254.169.254/2009-04-04/meta-data/instance-id: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2020-08-12T01:29:06.254Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:162	add_cloud_metadata: received disposition for aws after 3.000492515s. result=[provider:aws, error=failed requesting aws metadata: Get http://169.254.169.254/2014-02-25/dynamic/instance-identity/document: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2020-08-12T01:29:06.254Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:162	add_cloud_metadata: received disposition for az after 3.000519463s. result=[provider:az, error=failed requesting az metadata: Get http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2020-08-12T01:29:06.254Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:129	add_cloud_metadata: fetchMetadata ran for 3.000534335s
2020-08-12T01:29:06.254Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-08-12T01:29:06.254Z	DEBUG	[processors]	processors/processor.go:101	Generated new processors: add_cloud_metadata={}, add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], add_kubernetes_metadata
2020-08-12T01:29:06.254Z	DEBUG	[seccomp]	seccomp/seccomp.go:117	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-08-12T01:29:06.254Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-08-12T01:29:06.254Z	INFO	[beat]	instance/beat.go:983	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "085ca507-fd6d-484c-9ecd-e10e2414db7e"}}}
2020-08-12T01:29:06.255Z	INFO	[beat]	instance/beat.go:992	Build info	{"system_info": {"build": {"commit": "f79387d32717d79f689d94fda1ec80b2cf285d30", "libbeat": "7.8.0", "time": "2020-06-14T18:20:03.000Z", "version": "7.8.0"}}}
2020-08-12T01:29:06.255Z	INFO	[beat]	instance/beat.go:995	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.10"}}}
2020-08-12T01:29:06.256Z	INFO	[beat]	instance/beat.go:999	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-05-07T06:50:50Z","containerized":true,"name":"iZbp1e22um2oyzjtvpb3f9Z","ip":["127.0.0.1/8","172.16.51.148/20","169.254.123.1/24"],"kernel_version":"3.10.0-957.5.1.el7.x86_64","mac":["00:16:3f:00:b7:4a","02:42:57:90:97:fd","72:31:46:36:2b:df","ea:b2:84:00:c1:b1","16:f9:48:dd:fd:2d","5e:c5:1f:74:78:97","66:6e:e3:12:8c:43","26:d3:ae:a6:f8:b7","32:ba:f8:d0:5e:d0","1e:80:ae:cd:18:f6","ea:66:ae:19:85:92","fe:80:0a:a8:a2:88","3e:44:16:a8:27:c2","86:00:3a:f7:8f:94"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"83a8f1f835d84a9a9bf5417cecaf0c8e"}}}
2020-08-12T01:29:06.257Z	INFO	[beat]	instance/beat.go:1028	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2020-08-12T01:29:02.370Z"}}}
2020-08-12T01:29:06.257Z	INFO	instance/beat.go:310	Setup Beat: metricbeat; Version: 7.8.0
2020-08-12T01:29:06.257Z	DEBUG	[beat]	instance/beat.go:336	Initializing output plugins
2020-08-12T01:29:06.257Z	INFO	[index-management]	idxmgmt/std.go:183	Set output.elasticsearch.index to 'metricbeat-7.8.0' as ILM is enabled.
2020-08-12T01:29:06.257Z	INFO	eslegclient/connection.go:97	elasticsearch url: http://172.16.65.25:9200
2020-08-12T01:29:06.257Z	DEBUG	[publisher]	pipeline/consumer.go:137	start pipeline event consumer
2020-08-12T01:29:06.258Z	INFO	[publisher]	pipeline/module.go:113	Beat name: iZbp1e22um2oyzjtvpb3f9Z
2020-08-12T01:29:06.279Z	DEBUG	[modules]	beater/metricbeat.go:151	Available modules and metricsets: Register [ModuleFactory:[aws, azure, beat, cloudfoundry, docker, elasticsearch, kibana, linux, logstash, mongodb, mssql, mysql, oracle, postgresql, system, uwsgi], MetricSetFactory:[aerospike/namespace, apache/status, appsearch/stats, aws/cloudwatch, aws/ec2, aws/rds, aws/s3_daily_storage, aws/s3_request, aws/sqs, azure/compute_vm, azure/compute_vm_scaleset, azure/monitor, azure/storage, beat/state, beat/stats, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/mgr_cluster_disk, ceph/mgr_cluster_health, ceph/mgr_osd_perf, ceph/mgr_osd_pool_stats, ceph/mgr_osd_tree, ceph/mgr_pool_disk, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, cloudfoundry/container, cloudfoundry/counter, cloudfoundry/value, consul/agent, coredns/stats, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/enrich, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, golang/expvar, golang/heap, googlecloud/stackdriver, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, istio/citadel, istio/galley, istio/mesh, istio/mixer, istio/pilot, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/controllermanager, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/proxy, kubernetes/scheduler, kubernetes/state_container, kubernetes/state_cronjob, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_persistentvolume, kubernetes/state_persistentvolumeclaim, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_resourcequota, kubernetes/state_service, kubernetes/state_statefulset, kubernetes/state_storageclass, kubernetes/system, kubernetes/volume, kvm/dommemstat, kvm/status, linux/conntrack, linux/ksm, linux/pageinfo, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mssql/performance, mssql/transaction_log, munin/node, mysql/galera_status, mysql/status, nats/connections, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, openmetrics/collector, oracle/performance, oracle/tablespace, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, prometheus/query, prometheus/remote_write, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, sql/query, stan/channels, stan/stats, stan/subscriptions, statsd/server, system/core, system/cpu, system/diskio, system/entropy, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/network_summary, system/process, system/process_summary, system/raid, system/service, system/socket, system/socket_summary, system/uptime, system/users, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/connection, zookeeper/mntr, zookeeper/server], LightModules:[LightModules:[kafka/broker, kafka/producer, kafka/consumer, aws/billing, aws/sns, aws/lambda, aws/dynamodb, aws/vpn, aws/elb, aws/ebs, aws/usage, aws/transitgateway, aws/natgateway, iis/webserver, iis/website, redisenterprise/node, redisenterprise/proxy, tomcat/threading, tomcat/memory, tomcat/requests, tomcat/cache, googlecloud/compute, googlecloud/pubsub, googlecloud/loadbalancing, googlecloud/storage, ibmmq/qmgr, activemq/topic, activemq/broker, activemq/queue, azure/database_account, azure/container_registry, azure/container_instance, azure/container_service, cockroachdb/status]]]
2020-08-12T01:29:06.281Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-08-12T01:29:06.281Z	INFO	[autodiscover.pod]	kubernetes/util.go:94	kubernetes: Using pod name iZbp1e22um2oyzjtvpb3f9Z and namespace sxc to discover kubernetes node
2020-08-12T01:29:06.288Z	ERROR	[autodiscover.pod]	kubernetes/util.go:97	kubernetes: Querying for pod failed with error: pods "iZbp1e22um2oyzjtvpb3f9Z" not found
2020-08-12T01:29:06.288Z	DEBUG	[autodiscover.pod]	kubernetes/pod.go:67	Initializing a new Kubernetes watcher using node: localhost
2020-08-12T01:29:06.288Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:90	Configured autodiscover provider: kubernetes
2020-08-12T01:29:06.288Z	INFO	instance/beat.go:463	metricbeat start running.
2020-08-12T01:29:06.288Z	INFO	[autodiscover]	autodiscover/autodiscover.go:113	Starting autodiscover manager
2020-08-12T01:29:06.288Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-08-12T01:29:06.388Z	DEBUG	[kubernetes]	kubernetes/watcher.go:152	cache sync done

Could it be the cause of these

2020-08-12T01:29:06.281Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-08-12T01:29:06.288Z	ERROR	[autodiscover.pod]	kubernetes/util.go:97	kubernetes: Querying for pod failed with error: pods "iZbp1e22um2oyzjtvpb3f9Z" not found
9

i did it !!
This is the key

2020-08-12T01:29:06.288Z	ERROR	[autodiscover.pod]	kubernetes/util.go:97	kubernetes: Querying for pod failed with error: pods "iZbp1e22um2oyzjtvpb3f9Z" not found

add host: ${NODE_NAME} it work

logging.level: debug

#metricbeat.config.modules:
  # Mounted `metricbeat-daemonset-modules` configmap:
#  path: ${path.config}/modules.d/*.yml
  # Reload module configs as they change:
#  reload.enabled: false

# To enable hints based autodiscover uncomment this:
metricbeat.autodiscover:
  providers:
    - type: kubernetes
      host: ${NODE_NAME}
      templates:
      - condition:
          contains:
            kubernetes.container.name: "commander"
        config:
        - module: jolokia
          metricsets: ["jmx"]
          hosts: ["http://${data.host}:8080/jolokia"]
          namespace: "sxc"
          jmx.mappings:
          - mbean: 'java.lang:type=Memory'
            attributes:
             - attr: HeapMemoryUsage
               field: memory.heap_usage
             - attr: NonHeapMemoryUsage
               field: memory.non_heap_usage
          - mbean: 'Catalina:name=*,type=ThreadPool'
            attributes:
             - attr: port
               field: catalina.port
             - attr: maxConnections
               field: catalina.maxConnections
          - mbean: 'java.lang:type=Runtime'
            attributes:
             - attr: Uptime
               field: uptime
          

processors:
  - add_cloud_metadata:
  - add_host_metadata:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.