Setup output to Elasticsearch through Centralized Management


(Mladen Stankovic) #1

Hello,

could you please help me to setup Elasticsearch output. I managed to enroll my metricbeat agent but I don't get any information from agent.

As you can see I have created tag razvoj_metricbeat:

And this is my output of tag razvoj_metricbeat:

Am i doing something wrong?

Version of elasticsearch and metricbeat agent 5.6.0.

BR,
Mladen


(Pier-Hugues Pellerin) #2

Hello @mladen
Did you also add metricbeat module to the agent?


(Mladen Stankovic) #3

Hello @pierhugues,

thank you for your quick answer. If you mean to enable system module, yes I have it.

BR,
Mladen


(Pier-Hugues Pellerin) #4

Is there any error in the Metricbeat log?


(Mladen Stankovic) #5

No, there is. This is message in log after I restarted agent:

2018-11-22T14:49:42.857+0100    INFO    instance/beat.go:302    Setup Beat: metricbeat; Version: 6.5.0
2018-11-22T14:49:42.857+0100    INFO    instance/beat.go:328    Output is configured through Central Management
2018-11-22T14:49:42.858+0100    INFO    [publisher]     pipeline/module.go:110  Beat name: kaplaytest.it.telekom.yu
2018-11-22T14:49:42.858+0100    INFO    instance/beat.go:424    metricbeat start running.
2018-11-22T14:49:42.858+0100    WARN    [cfgwarn]       management/manager.go:100       BETA: Central management is enabled
2018-11-22T14:49:42.858+0100    INFO    [centralmgmt]   management/manager.go:101       Starting central management service
2018-11-22T14:49:42.858+0100    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2018-11-22T14:49:43.710+0100    INFO    [centralmgmt]   management/manager.go:198       Applying settings for output
2018-11-22T14:49:43.711+0100    INFO    elasticsearch/client.go:163     Elasticsearch url: http://kaeskibanademo:9200

I will try to upgrade to 6.5.1 and add one or more metricbeat agent to Centralized management. I will provide you with an update upon completion this tasks.

BR,
Mladen


(Pier-Hugues Pellerin) #6

Thanks I do not see anything weird in the current trace,

Can you start one of the metricbeat with the following flags: metricbeat -v -e -d "*", this should give us the complete trace in debug mode.


(Mladen Stankovic) #7

This is the output:

2018-11-22T15:22:50.574+0100    INFO    instance/beat.go:616    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2018-11-22T15:22:50.575+0100    DEBUG   [beat]  instance/beat.go:653    Beat metadata path: /var/lib/metricbeat/meta.json
2018-11-22T15:22:50.575+0100    INFO    instance/beat.go:623    Beat UUID: 774ee895-7adf-4d70-b556-09333e23ab80
2018-11-22T15:22:50.575+0100    DEBUG   [keystore] keystore/keystore.go:119 accessing key 'management.accesstoken' from the keystore
2018-11-22T15:22:50.575+0100    INFO    kibana/client.go:118    Kibana url: http://kaeskibanademo:5601
2018-11-22T15:22:50.575+0100    INFO    [seccomp]       seccomp/seccomp.go:93   Syscall filter could not be installed because the kernel does not support secc                                omp
2018-11-22T15:22:50.575+0100    INFO    [beat]  instance/beat.go:849    Beat info    {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "774ee895-7adf-4d70-b556-09333e23ab80"}}}
2018-11-22T15:22:50.575+0100    INFO    [beat]  instance/beat.go:858    Build info      {"system_info": {"build": {"commit": "ff5b9b3db49856a25b5eda133b6997f2157a4910", "libbeat": "6.5.0", "time": "2018-11-09T18:03:04.000Z", "version": "6.5.0"}}}
2018-11-22T15:22:50.575+0100    INFO    [beat]  instance/beat.go:861    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.10.3"}}}
2018-11-22T15:22:50.576+0100    INFO    [beat]  instance/beat.go:865    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-07                                -03T16:16:01+02:00","containerized":true,"name":"kaplaytest","ip":["127.0.0.1/8","::1/128","10.123.33.166/27","fe80::250:56ff:fe8a:965/64"],"ker                                nel_version":"3.10.0-693.el7.x86_64","mac":["00:50:56:8a:09:65"],"os":{"family":"","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.4 (                                Maipo)","major":7,"minor":4,"patch":0,"codename":"Maipo"},"timezone":"CET","timezone_offset_sec":3600,"id":"ae4fe779130e4623b7de590ceb7e3a5e"}}}
2018-11-22T15:22:50.576+0100    INFO    [beat]  instance/beat.go:894    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"perm                                itted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","ne                                t_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys                                _time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":                                ["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin                                ","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time",                                "sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown"                                ,"dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_r                                aw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty                                _config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/var/log/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 20046, "ppid": 19988, "seccomp": {"mode":"disabled"}, "start_time": "2018-11-22T15:22:49.930+0100"}}}

(Mladen Stankovic) #8
2018-11-22T15:22:50.576+0100    INFO    instance/beat.go:302    Setup Beat: metricbeat; Version: 6.5.0
    2018-11-22T15:22:50.576+0100    DEBUG   [beat]  instance/beat.go:323    Initializing output plugins
    2018-11-22T15:22:50.576+0100    INFO    instance/beat.go:328    Output is configured through Central Management
    2018-11-22T15:22:50.576+0100    DEBUG   [processors]    processors/processor.go:66      Processors:
    2018-11-22T15:22:50.577+0100    DEBUG   [publish]       pipeline/consumer.go:137        start pipeline event consumer
    2018-11-22T15:22:50.577+0100    INFO    [publisher]     pipeline/module.go:110  Beat name: kaplaytest
    2018-11-22T15:22:50.577+0100    DEBUG   [modules]       beater/metricbeat.go:103        Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system, u                                wsgi], MetricSetFactory:[aerospike/namespace, apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_df, ce                                ph/osd_tree, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/imag                                e, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/index, elasticsearch/index_                                recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, e                                nvoyproxy/server, etcd/leader, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia                                /jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/event, kubernetes/node, kubern                                etes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_                                statefulset, kubernetes/system, kubernetes/volume, kvm/dommemstat, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mo                                ngodb/metrics, mongodb/replstatus, mongodb/status, munin/node, mysql/galera_status, mysql/status, nginx/stubstatus, php_fpm/pool, php_fpm/process, postgresql/                                activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, prometheus/stats, rabbitmq/connection, rabbitmq/exchange, rabb                                itmq/node, rabbitmq/queue, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, s                                ystem/network, system/process, system/process_summary, system/raid, system/socket, system/socket_summary, system/uptime, traefik/health, uwsgi/status, vsphere                                /datastore, vsphere/host, vsphere/virtualmachine, zookeeper/mntr]]
    2018-11-22T15:22:50.577+0100    INFO    instance/beat.go:424    metricbeat start running.
    2018-11-22T15:22:50.577+0100    WARN    [cfgwarn]       management/manager.go:100       BETA: Central management is enabled
    2018-11-22T15:22:50.577+0100    INFO    [centralmgmt]   management/manager.go:101       Starting central management service
    2018-11-22T15:22:50.577+0100    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
    2018-11-22T15:22:50.577+0100    DEBUG   [centralmgmt]   management/manager.go:164       Retrieving new configurations from Kibana
    2018-11-22T15:22:51.567+0100    DEBUG   [centralmgmt]   management/manager.go:172       configuration didn't change, sleeping
    2018-11-22T15:22:51.567+0100    INFO    [centralmgmt]   management/manager.go:198       Applying settings for output
    2018-11-22T15:22:51.567+0100    INFO    elasticsearch/client.go:163     Elasticsearch url: http://kaeskibanademo:9200

I try also network connectivity and everything is OK.

BR,
Mladen


(Pier-Hugues Pellerin) #9

Can you add the content of the data/management.yml to this post?


(Pier-Hugues Pellerin) #10

From the log I only see the output to be applied to metricbeat, with the content of the data/management.yml I should be able see what is the actual config of the beat.

Also can you go to the management ui and show me all the config block for the specific tag?
From your screenshot it should be razvoj-metricbeat


(Mladen Stankovic) #11

This is the config of management.yml file:

configok: true
configs:
- type: output
  blocks:
  - raw:
      elasticsearch:
        hosts:
        - kaeskibanademo:9200
        password: ********
        username: elastic
      output: elasticsearch

And I have just one config block for this tag:

BR,
Mladen


(Pier-Hugues Pellerin) #12

When you setup metricbeat to use config management, The management API becomes the source of truth and will ignore any local configs.

So from what I see in the screenshots, you only have an elasticsearch output configured in the management UI and looking at the content of the management.yml metricbeat only have the output configured. So both of theses view are sync.

If you want to have metricbeat send data to Elasticsearch you have to click Add configuration block and add a metricbeat module configuration block. Doing so will make Metricbeat send event to Elasticsearch.


(Mladen Stankovic) #13

Sorry, I didn't understand part with system module in centralized management. I add this block:

Now I have two blocks:

image

Should I add some more blocks?

BR,
Mladen


(Pier-Hugues Pellerin) #14

@mladen With the two config block, you see events send to elasticsearch now from a metricbeat instance, and we should also see the change pick up in the metricbeat log and data/management.yml.


(Mladen Stankovic) #15

@pierhugues thank you very much for your help and patience. Please note that beside configuration that was applied I still had to load the index template in Elasticsearch manually. It seams to me that if you use centralized management, agent don't load index template in Elasticsearch automatically.

Just one more question, is it possible to use metricbeat key-store and put variable in management.yml file instead of plain text?

BR,
Mladen


(Pier-Hugues Pellerin) #16

@mladen Yes the template and the dashboard still requires you to run the setup command to make them available. Its a known issue and we are working on it.

Just one more question, is it possible to use metricbeat key-store and put variable in management.yml file instead of plain text?

Yes you can do that, you can add secret to the local keystore and reference them in the config management. We are considering allowing config management to send a keystore to the running instances.

FYI: The token that config management is using on the beats to connect to the API is already stored in the keystore.


Centralized beats configuration
(Mladen Stankovic) #17

@pierhugues thanks a lot for your explanations. You help me to understand how bets management works.

BR,
Mladen