Hello,
could you please help me to setup Elasticsearch output. I managed to enroll my metricbeat agent but I don't get any information from agent.
As you can see I have created tag razvoj_metricbeat:
And this is my output of tag razvoj_metricbeat:
Am i doing something wrong?
Version of elasticsearch and metricbeat agent 5.6.0.
BR,
Mladen
Hello @pierhugues,
thank you for your quick answer. If you mean to enable system module, yes I have it.
BR,
Mladen
Is there any error in the Metricbeat log?
No, there is. This is message in log after I restarted agent:
2018-11-22T14:49:42.857+0100 INFO instance/beat.go:302 Setup Beat: metricbeat; Version: 6.5.0
2018-11-22T14:49:42.857+0100 INFO instance/beat.go:328 Output is configured through Central Management
2018-11-22T14:49:42.858+0100 INFO [publisher] pipeline/module.go:110 Beat name: kaplaytest.it.telekom.yu
2018-11-22T14:49:42.858+0100 INFO instance/beat.go:424 metricbeat start running.
2018-11-22T14:49:42.858+0100 WARN [cfgwarn] management/manager.go:100 BETA: Central management is enabled
2018-11-22T14:49:42.858+0100 INFO [centralmgmt] management/manager.go:101 Starting central management service
2018-11-22T14:49:42.858+0100 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2018-11-22T14:49:43.710+0100 INFO [centralmgmt] management/manager.go:198 Applying settings for output
2018-11-22T14:49:43.711+0100 INFO elasticsearch/client.go:163 Elasticsearch url: http://kaeskibanademo:9200
I will try to upgrade to 6.5.1 and add one or more metricbeat agent to Centralized management. I will provide you with an update upon completion this tasks.
BR,
Mladen
Thanks I do not see anything weird in the current trace,
Can you start one of the metricbeat with the following flags: metricbeat -v -e -d "*", this should give us the complete trace in debug mode.
This is the output:
2018-11-22T15:22:50.574+0100 INFO instance/beat.go:616 Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2018-11-22T15:22:50.575+0100 DEBUG [beat] instance/beat.go:653 Beat metadata path: /var/lib/metricbeat/meta.json
2018-11-22T15:22:50.575+0100 INFO instance/beat.go:623 Beat UUID: 774ee895-7adf-4d70-b556-09333e23ab80
2018-11-22T15:22:50.575+0100 DEBUG [keystore] keystore/keystore.go:119 accessing key 'management.accesstoken' from the keystore
2018-11-22T15:22:50.575+0100 INFO kibana/client.go:118 Kibana url: http://kaeskibanademo:5601
2018-11-22T15:22:50.575+0100 INFO [seccomp] seccomp/seccomp.go:93 Syscall filter could not be installed because the kernel does not support secc omp
2018-11-22T15:22:50.575+0100 INFO [beat] instance/beat.go:849 Beat info {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "774ee895-7adf-4d70-b556-09333e23ab80"}}}
2018-11-22T15:22:50.575+0100 INFO [beat] instance/beat.go:858 Build info {"system_info": {"build": {"commit": "ff5b9b3db49856a25b5eda133b6997f2157a4910", "libbeat": "6.5.0", "time": "2018-11-09T18:03:04.000Z", "version": "6.5.0"}}}
2018-11-22T15:22:50.575+0100 INFO [beat] instance/beat.go:861 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.10.3"}}}
2018-11-22T15:22:50.576+0100 INFO [beat] instance/beat.go:865 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-07 -03T16:16:01+02:00","containerized":true,"name":"kaplaytest","ip":["127.0.0.1/8","::1/128","10.123.33.166/27","fe80::250:56ff:fe8a:965/64"],"ker nel_version":"3.10.0-693.el7.x86_64","mac":["00:50:56:8a:09:65"],"os":{"family":"","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.4 ( Maipo)","major":7,"minor":4,"patch":0,"codename":"Maipo"},"timezone":"CET","timezone_offset_sec":3600,"id":"ae4fe779130e4623b7de590ceb7e3a5e"}}}
2018-11-22T15:22:50.576+0100 INFO [beat] instance/beat.go:894 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"perm itted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","ne t_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys _time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective": ["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin ","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time", "sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown" ,"dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_r aw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty _config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/var/log/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 20046, "ppid": 19988, "seccomp": {"mode":"disabled"}, "start_time": "2018-11-22T15:22:49.930+0100"}}}2018-11-22T15:22:50.576+0100 INFO instance/beat.go:302 Setup Beat: metricbeat; Version: 6.5.0
2018-11-22T15:22:50.576+0100 DEBUG [beat] instance/beat.go:323 Initializing output plugins
2018-11-22T15:22:50.576+0100 INFO instance/beat.go:328 Output is configured through Central Management
2018-11-22T15:22:50.576+0100 DEBUG [processors] processors/processor.go:66 Processors:
2018-11-22T15:22:50.577+0100 DEBUG [publish] pipeline/consumer.go:137 start pipeline event consumer
2018-11-22T15:22:50.577+0100 INFO [publisher] pipeline/module.go:110 Beat name: kaplaytest
2018-11-22T15:22:50.577+0100 DEBUG [modules] beater/metricbeat.go:103 Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system, u wsgi], MetricSetFactory:[aerospike/namespace, apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_df, ce ph/osd_tree, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/imag e, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/index, elasticsearch/index_ recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, e nvoyproxy/server, etcd/leader, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia /jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/event, kubernetes/node, kubern etes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_ statefulset, kubernetes/system, kubernetes/volume, kvm/dommemstat, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mo ngodb/metrics, mongodb/replstatus, mongodb/status, munin/node, mysql/galera_status, mysql/status, nginx/stubstatus, php_fpm/pool, php_fpm/process, postgresql/ activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, prometheus/stats, rabbitmq/connection, rabbitmq/exchange, rabb itmq/node, rabbitmq/queue, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, s ystem/network, system/process, system/process_summary, system/raid, system/socket, system/socket_summary, system/uptime, traefik/health, uwsgi/status, vsphere /datastore, vsphere/host, vsphere/virtualmachine, zookeeper/mntr]]
2018-11-22T15:22:50.577+0100 INFO instance/beat.go:424 metricbeat start running.
2018-11-22T15:22:50.577+0100 WARN [cfgwarn] management/manager.go:100 BETA: Central management is enabled
2018-11-22T15:22:50.577+0100 INFO [centralmgmt] management/manager.go:101 Starting central management service
2018-11-22T15:22:50.577+0100 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2018-11-22T15:22:50.577+0100 DEBUG [centralmgmt] management/manager.go:164 Retrieving new configurations from Kibana
2018-11-22T15:22:51.567+0100 DEBUG [centralmgmt] management/manager.go:172 configuration didn't change, sleeping
2018-11-22T15:22:51.567+0100 INFO [centralmgmt] management/manager.go:198 Applying settings for output
2018-11-22T15:22:51.567+0100 INFO elasticsearch/client.go:163 Elasticsearch url: http://kaeskibanademo:9200
I try also network connectivity and everything is OK.
BR,
Mladen
Can you add the content of the data/management.yml to this post?
From the log I only see the output to be applied to metricbeat, with the content of the data/management.yml I should be able see what is the actual config of the beat.
Also can you go to the management ui and show me all the config block for the specific tag?
From your screenshot it should be razvoj-metricbeat
This is the config of management.yml file:
configok: true
configs:
- type: output
blocks:
- raw:
elasticsearch:
hosts:
- kaeskibanademo:9200
password: ********
username: elastic
output: elasticsearch
And I have just one config block for this tag:
BR,
Mladen
When you setup metricbeat to use config management, The management API becomes the source of truth and will ignore any local configs.
So from what I see in the screenshots, you only have an elasticsearch output configured in the management UI and looking at the content of the management.yml metricbeat only have the output configured. So both of theses view are sync.
If you want to have metricbeat send data to Elasticsearch you have to click Add configuration block and add a metricbeat module configuration block. Doing so will make Metricbeat send event to Elasticsearch.
Sorry, I didn't understand part with system module in centralized management. I add this block:
Now I have two blocks:
Should I add some more blocks?
BR,
Mladen
@mladen With the two config block, you see events send to elasticsearch now from a metricbeat instance, and we should also see the change pick up in the metricbeat log and data/management.yml.
@pierhugues thank you very much for your help and patience. Please note that beside configuration that was applied I still had to load the index template in Elasticsearch manually. It seams to me that if you use centralized management, agent don't load index template in Elasticsearch automatically.
Just one more question, is it possible to use metricbeat key-store and put variable in management.yml file instead of plain text?
BR,
Mladen
@mladen Yes the template and the dashboard still requires you to run the setup command to make them available. Its a known issue and we are working on it.
Just one more question, is it possible to use metricbeat key-store and put variable in management.yml file instead of plain text?
Yes you can do that, you can add secret to the local keystore and reference them in the config management. We are considering allowing config management to send a keystore to the running instances.
FYI: The token that config management is using on the beats to connect to the API is already stored in the keystore.
@pierhugues thanks a lot for your explanations. You help me to understand how bets management works.
BR,
Mladen
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
