There is no data to display - Infrastructure tab

Elastic Stack Beats
1

Hi,

I can't display Hosts in the Infrastructure tab in Kibana. I checked the other threads on the forum but it still doesn't work. Can you please help me?

Metricbeat is installed on multiple servers and they send data through logstash.

My logstash configuration is :

input {
  beats {
    port => 5046
  }
}

output {
    elasticsearch {
        hosts => ["http://SELC01.groups.local:9200","http://SELC02.groups.local:9200","http://SELC03.groups.local:9200"]
        index => "%{[@metadata][beat]}-qa-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        user => #########
        password => #########
        ssl => false
    }
}

I tested the following things :

  • Deleting all metricbeat index and recreate a template with the following command :

.\metricbeat.exe setup --index-management -E output.logstash.enabled=false -E
'output.elasticsearch.hosts=["http://SELC01.groups.local:9200", "http://SELC02.groups.local:9200", "http://SELC03.groups.local:9200"]'

  • adding field in the kibana.yml file :

xpack.infra:
sources:
default:
fields:
host: 'beat.hostname.keyword'

  • changing the configuration source as :

hostname : host.name or beat.hostname or beat.name

Can you advise me?

Thanks in advance.

2

Hi!

You should try in an order the following things:

  1. Did the setup of the Metricbeat templates, indexes etc work. The command you provided works? Can you see the template to be loaded after that?

  2. If the templates/mappings are already setup then the problem lies in the data shipiing from Filebeat to Logastsh (need to verify that) or from Logstash to ES. So can you check the logs and see if sth goes wrong there?

C.

3
  1. Did the setup of the Metricbeat templates, indexes etc work. The command you provided works? Can you see the template to be loaded after that?

I guess so, i get the following results :

In the command prompt :

Index setup finished

In the metricbeat.log :

2020-02-03T11:14:42.857+0100	INFO	instance/beat.go:606	Home path: [C:\metricbeat-7.2.0] Config path: [C:\metricbeat-7.2.0] Data path: [C:\metricbeat-7.2.0\data] Logs path: [C:\metricbeat-7.2.0\logs]
2020-02-03T11:14:42.894+0100	INFO	instance/beat.go:614	Beat ID: df327b1b-5d9d-4a10-9dd6-c5996e24d01a
2020-02-03T11:14:42.901+0100	INFO	[beat]	instance/beat.go:902	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\metricbeat-7.2.0", "data": "C:\\metricbeat-7.2.0\\data", "home": "C:\\metricbeat-7.2.0", "logs": "C:\\metricbeat-7.2.0\\logs"}, "type": "metricbeat", "uuid": "df327b1b-5d9d-4a10-9dd6-c5996e24d01a"}}}
2020-02-03T11:14:42.902+0100	INFO	[beat]	instance/beat.go:911	Build info	{"system_info": {"build": {"commit": "9ba65d864ca37cd32c25b980dbb4020975288fc0", "libbeat": "7.2.0", "time": "2019-06-20T15:07:31.000Z", "version": "7.2.0"}}}
2020-02-03T11:14:42.902+0100	INFO	[beat]	instance/beat.go:914	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
2020-02-03T11:14:42.905+0100	INFO	[beat]	instance/beat.go:918	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-20T23:50:42.05+01:00","name":"SGSOQ01","ip":["192.136.19.133/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.14393.3442 (rs1_release.191219-1727)","mac":["00:50:56:aa:50:cd"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2016 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.3443"},"timezone":"CET","timezone_offset_sec":3600,"id":"b64f60a7-5d8f-4832-94d1-0c636151a7e5"}}}
2020-02-03T11:14:42.907+0100	INFO	[beat]	instance/beat.go:947	Process info	{"system_info": {"process": {"cwd": "C:\\metricbeat-7.2.0", "exe": "C:\\metricbeat-7.2.0\\metricbeat.exe", "name": "metricbeat.exe", "pid": 184, "ppid": 6648, "start_time": "2020-02-03T11:14:41.992+0100"}}}
2020-02-03T11:14:42.908+0100	INFO	instance/beat.go:292	Setup Beat: metricbeat; Version: 7.2.0
2020-02-03T11:14:42.908+0100	INFO	[index-management]	idxmgmt/std.go:178	Set output.elasticsearch.index to 'metricbeat-7.2.0' as ILM is enabled.
2020-02-03T11:14:42.910+0100	INFO	elasticsearch/client.go:166	Elasticsearch url: http://SELC01.groups.local:9200
2020-02-03T11:14:42.910+0100	INFO	elasticsearch/client.go:166	Elasticsearch url: http://SELC02.groups.local:9200
2020-02-03T11:14:42.910+0100	INFO	elasticsearch/client.go:166	Elasticsearch url: http://SELC03.groups.local:9200
2020-02-03T11:14:42.911+0100	INFO	[publisher]	pipeline/module.go:97	Beat name: SGSOQ01
2020-02-03T11:14:42.911+0100	INFO	elasticsearch/client.go:166	Elasticsearch url: http://SELC01.groups.local:9200
2020-02-03T11:14:42.911+0100	INFO	elasticsearch/client.go:166	Elasticsearch url: http://SELC02.groups.local:9200
2020-02-03T11:14:42.911+0100	INFO	elasticsearch/client.go:166	Elasticsearch url: http://SELC03.groups.local:9200
2020-02-03T11:14:42.997+0100	INFO	elasticsearch/client.go:735	Attempting to connect to Elasticsearch version 7.2.0
2020-02-03T11:14:43.109+0100	INFO	[index-management]	idxmgmt/std.go:252	Auto ILM enable success.
2020-02-03T11:14:43.219+0100	INFO	[index-management]	idxmgmt/std.go:265	ILM policy successfully loaded.
2020-02-03T11:14:43.219+0100	INFO	[index-management]	idxmgmt/std.go:394	Set setup.template.name to '{metricbeat-7.2.0 {now/d}-000001}' as ILM is enabled.
2020-02-03T11:14:43.219+0100	INFO	[index-management]	idxmgmt/std.go:399	Set setup.template.pattern to 'metricbeat-7.2.0-*' as ILM is enabled.
2020-02-03T11:14:43.219+0100	INFO	[index-management]	idxmgmt/std.go:433	Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.2.0 {now/d}-000001} as ILM is enabled.
2020-02-03T11:14:43.219+0100	INFO	[index-management]	idxmgmt/std.go:437	Set settings.index.lifecycle.name in template to {metricbeat-7.2.0 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-02-03T11:14:43.268+0100	INFO	template/load.go:169	Existing template will be overwritten, as overwrite is enabled.
2020-02-03T11:14:43.456+0100	INFO	template/load.go:108	Try loading template metricbeat-7.2.0 to Elasticsearch
2020-02-03T11:14:43.583+0100	INFO	template/load.go:100	template with name 'metricbeat-7.2.0' loaded.
2020-02-03T11:14:43.594+0100	INFO	[index-management]	idxmgmt/std.go:289	Loaded index template.
2020-02-03T11:14:44.033+0100	INFO	[index-management]	idxmgmt/std.go:300	Write alias successfully generated.
  1. If the templates/mappings are already setup then the problem lies in the data shipiing from Filebeat to Logastsh (need to verify that) or from Logstash to ES. So can you check the logs and see if sth goes wrong there?
  • from Filebeat to logstash :

I have some errors but i don't think it's linked to metricbeat (monitored log are not parsed as JSON)

2020-02-03T11:22:04.200+0100	ERROR	readjson/json.go:52	Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}
2020-02-03T11:22:14.217+0100	ERROR	readjson/json.go:52	Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}
2020-02-03T11:22:30.627+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":751546,"time":{"ms":31}},"total":{"ticks":1204655,"time":{"ms":47},"value":1204655},"user":{"ticks":453109,"time":{"ms":16}}},"handles":{"open":238},"info":{"ephemeral_id":"d255fe53-e599-495a-845b-2c5c5434d89f","uptime":{"ms":1164552693}},"memstats":{"gc_next":4440784,"memory_alloc":2220264,"memory_total":50651051672,"rss":1069056},"runtime":{"goroutines":52}},"filebeat":{"events":{"added":3,"done":3},"harvester":{"open_files":7,"running":7}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":3,"batches":3,"total":3},"read":{"bytes":24},"write":{"bytes":2201}},"pipeline":{"clients":1,"events":{"active":0,"published":3,"total":3},"queue":{"acked":3}}},"registrar":{"states":{"current":7,"update":3},"writes":{"success":3,"total":3}}}}}
2020-02-03T11:23:00.634+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":751578,"time":{"ms":32}},"total":{"ticks":1204703,"time":{"ms":48},"value":1204703},"user":{"ticks":453125,"time":{"ms":16}}},"handles":{"open":238},"info":{"ephemeral_id":"d255fe53-e599-495a-845b-2c5c5434d89f","uptime":{"ms":1164582699}},"memstats":{"gc_next":4440784,"memory_alloc":2380816,"memory_total":50651212224},"runtime":{"goroutines":52}},"filebeat":{"harvester":{"open_files":7,"running":7}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":7}}}}}
2020-02-03T11:23:30.627+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":751609,"time":{"ms":31}},"total":{"ticks":1204734,"time":{"ms":31},"value":1204734},"user":{"ticks":453125}},"handles":{"open":235},"info":{"ephemeral_id":"d255fe53-e599-495a-845b-2c5c5434d89f","uptime":{"ms":1164612692}},"memstats":{"gc_next":4440784,"memory_alloc":2523552,"memory_total":50651354960},"runtime":{"goroutines":52}},"filebeat":{"harvester":{"open_files":7,"running":7}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":7}}}}}
2020-02-03T11:24:00.629+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":751640,"time":{"ms":31}},"total":{"ticks":1204780,"time":{"ms":46},"value":1204780},"user":{"ticks":453140,"time":{"ms":15}}},"handles":{"open":235},"info":{"ephemeral_id":"d255fe53-e599-495a-845b-2c5c5434d89f","uptime":{"ms":1164642695}},"memstats":{"gc_next":4440784,"memory_alloc":2669840,"memory_total":50651501248,"rss":16384},"runtime":{"goroutines":52}},"filebeat":{"harvester":{"open_files":7,"running":7}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":7}}}}}
  • from Logstash to ES :

nothing interesting in the logfile

4

Hi!

Thank you for elaborating more!

We should investigate the ERROR we see regarding json.
Also could you share Metricbeat's configuration? What kind of metrics are you trying to collect? With what modules? (If no Modules, then try to enable system module)

Last but not least, I would suggest you enable console output and verify that events are being created by Metricbeat.

C.

5

We should investigate the ERROR we see regarding json.

We can but i don't have a lot of power on this. Maybe I can temporary remove filebeat inputs of log types to see if the problem is linked :slight_smile:

Also could you share Metricbeat's configuration?

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
  host: "http://SKIB01.groups.local:5601"
  username: "######"
  password: "######"
output.logstash:
  hosts: ["slog02.groups.local:5046"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
logging.level: info

What kind of metrics are you trying to collect? With what modules? (If no Modules, then try to enable system module)

The system's module is the only enabled module with the default configuration :

- module: system
  period: 10s
  metricsets:
    - cpu
    - memory
    - network
    - process
    - process_summary
    - socket_summary
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

- module: system
  period: 15m
  metricsets:
    - uptime

Last but not least, I would suggest you enable console output and verify that events are being created by Metricbeat.

I setup the console output for Metricbeat and filebeat. I removed output.logstash from the original configuration file . You can find the results below :

Filebeat :

2020-02-03T14:20:17.808+0100	INFO	instance/beat.go:606	Home path: [C:\filebeat-7.2.0] Config path: [C:\filebeat-7.2.0] Data path: [C:\filebeat-7.2.0\data] Logs path: [C:\filebeat-7.2.0\logs]
2020-02-03T14:20:17.817+0100	INFO	instance/beat.go:614	Beat ID: 0ff5d9db-48be-4a8c-88de-91572785c119
2020-02-03T14:20:17.824+0100	INFO	[beat]	instance/beat.go:902	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\filebeat-7.2.0", "data": "C:\\filebeat-7.2.0\\data", "home": "C:\\filebeat-7.2.0", "logs": "C:\\filebeat-7.2.0\\logs"}, "type": "filebeat", "uuid": "0ff5d9db-48be-4a8c-88de-91572785c119"}}}
2020-02-03T14:20:17.824+0100	INFO	[beat]	instance/beat.go:911	Build info	{"system_info": {"build": {"commit": "9ba65d864ca37cd32c25b980dbb4020975288fc0", "libbeat": "7.2.0", "time": "2019-06-20T15:05:29.000Z", "version": "7.2.0"}}}
2020-02-03T14:20:17.824+0100	INFO	[beat]	instance/beat.go:914	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
2020-02-03T14:20:17.828+0100	INFO	[beat]	instance/beat.go:918	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-20T23:50:42.04+01:00","name":"SGSOQ01","ip":["192.136.19.133/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.14393.3442 (rs1_release.191219-1727)","mac":["00:50:56:aa:50:cd"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2016 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.3443"},"timezone":"CET","timezone_offset_sec":3600,"id":"b64f60a7-5d8f-4832-94d1-0c636151a7e5"}}}
2020-02-03T14:20:17.831+0100	INFO	[beat]	instance/beat.go:947	Process info	{"system_info": {"process": {"cwd": "C:\\filebeat-7.2.0", "exe": "C:\\filebeat-7.2.0\\filebeat.exe", "name": "filebeat.exe", "pid": 11136, "ppid": 9504, "start_time": "2020-02-03T14:20:17.759+0100"}}}
2020-02-03T14:20:17.831+0100	INFO	instance/beat.go:292	Setup Beat: filebeat; Version: 7.2.0
2020-02-03T14:20:17.832+0100	INFO	[publisher]	pipeline/module.go:97	Beat name: SGSOQ01
2020-02-03T14:20:17.834+0100	INFO	instance/beat.go:421	filebeat start running.
2020-02-03T14:20:17.834+0100	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-02-03T14:20:17.844+0100	INFO	registrar/registrar.go:145	Loading registrar data from C:\filebeat-7.2.0\data\registry\filebeat\data.json
2020-02-03T14:20:17.844+0100	INFO	registrar/registrar.go:152	States Loaded from registrar: 7
2020-02-03T14:20:17.844+0100	WARN	beater/filebeat.go:358	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-03T14:20:17.844+0100	INFO	crawler/crawler.go:72	Loading Inputs: 1
2020-02-03T14:20:17.844+0100	INFO	log/input.go:148	Configured paths: [D:\Applications\GSO\auth-server\auth-server.out.log]
2020-02-03T14:20:17.844+0100	INFO	input/input.go:114	Starting input of type: log; ID: 7594395317448897889 
2020-02-03T14:20:17.845+0100	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2020-02-03T14:20:17.845+0100	INFO	cfgfile/reload.go:172	Config reloader started
2020-02-03T14:20:17.845+0100	INFO	cfgfile/reload.go:227	Loading of config files completed.
2020-02-03T14:20:20.827+0100	INFO	add_cloud_metadata/add_cloud_metadata.go:347	add_cloud_metadata: hosting provider type not detected.
2020-02-03T14:20:47.838+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":78,"time":{"ms":78}},"total":{"ticks":124,"time":{"ms":124},"value":124},"user":{"ticks":46,"time":{"ms":46}}},"handles":{"open":175},"info":{"ephemeral_id":"214e4a02-30ef-41d7-afe8-79ddb239d88e","uptime":{"ms":30049}},"memstats":{"gc_next":4194304,"memory_alloc":2445576,"memory_total":6280304,"rss":23449600},"runtime":{"goroutines":21}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"console"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":7,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4}}}}}
2020-02-03T14:21:17.839+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":78},"total":{"ticks":124,"value":124},"user":{"ticks":46}},"handles":{"open":177},"info":{"ephemeral_id":"214e4a02-30ef-41d7-afe8-79ddb239d88e","uptime":{"ms":60049}},"memstats":{"gc_next":4194304,"memory_alloc":2510568,"memory_total":6345296,"rss":16384},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":7}}}}}
2020-02-03T14:21:37.855+0100	INFO	log/harvester.go:253	Harvester started for file: D:\Applications\GSO\auth-server\auth-server.out.log

Metricbeat :

2020-02-03T14:18:17.841+0100	INFO	instance/beat.go:606	Home path: [C:\metricbeat-7.2.0] Config path: [C:\metricbeat-7.2.0] Data path: [C:\metricbeat-7.2.0\data] Logs path: [C:\metricbeat-7.2.0\logs]
2020-02-03T14:18:17.852+0100	INFO	instance/beat.go:614	Beat ID: df327b1b-5d9d-4a10-9dd6-c5996e24d01a
2020-02-03T14:18:17.859+0100	INFO	[beat]	instance/beat.go:902	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\metricbeat-7.2.0", "data": "C:\\metricbeat-7.2.0\\data", "home": "C:\\metricbeat-7.2.0", "logs": "C:\\metricbeat-7.2.0\\logs"}, "type": "metricbeat", "uuid": "df327b1b-5d9d-4a10-9dd6-c5996e24d01a"}}}
2020-02-03T14:18:17.859+0100	INFO	[beat]	instance/beat.go:911	Build info	{"system_info": {"build": {"commit": "9ba65d864ca37cd32c25b980dbb4020975288fc0", "libbeat": "7.2.0", "time": "2019-06-20T15:07:31.000Z", "version": "7.2.0"}}}
2020-02-03T14:18:17.859+0100	INFO	[beat]	instance/beat.go:914	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
2020-02-03T14:18:17.863+0100	INFO	[beat]	instance/beat.go:918	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-20T23:50:42.03+01:00","name":"SGSOQ01","ip":["192.136.19.133/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.14393.3442 (rs1_release.191219-1727)","mac":["00:50:56:aa:50:cd"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2016 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.3443"},"timezone":"CET","timezone_offset_sec":3600,"id":"b64f60a7-5d8f-4832-94d1-0c636151a7e5"}}}
2020-02-03T14:18:17.865+0100	INFO	[beat]	instance/beat.go:947	Process info	{"system_info": {"process": {"cwd": "C:\\metricbeat-7.2.0", "exe": "C:\\metricbeat-7.2.0\\metricbeat.exe", "name": "metricbeat.exe", "pid": 4760, "ppid": 6648, "start_time": "2020-02-03T14:18:16.991+0100"}}}
2020-02-03T14:18:17.865+0100	INFO	instance/beat.go:292	Setup Beat: metricbeat; Version: 7.2.0
2020-02-03T14:18:17.866+0100	INFO	[publisher]	pipeline/module.go:97	Beat name: SGSOQ01
2020-02-03T14:18:17.866+0100	INFO	instance/beat.go:421	metricbeat start running.
2020-02-03T14:18:17.866+0100	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-02-03T14:18:17.869+0100	INFO	helper/privileges_windows.go:79	Metricbeat process and system info: {"OSVersion":{"Major":6,"Minor":2,"Build":9200},"Arch":"amd64","NumCPU":4,"User":{"SID":"S-1-5-21-169686320-2045781430-181542594-31085","Account":"lamquing_adm","Domain":"INTERDOM","Type":1},"ProcessPrivs":{"SeBackupPrivilege":{"enabled":false},"SeChangeNotifyPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateGlobalPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePagefilePrivilege":{"enabled":false},"SeCreateSymbolicLinkPrivilege":{"enabled":false},"SeDebugPrivilege":{"enabled":true},"SeDelegateSessionUserImpersonatePrivilege":{"enabled":false},"SeImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseBasePriorityPrivilege":{"enabled":false},"SeIncreaseQuotaPrivilege":{"enabled":false},"SeIncreaseWorkingSetPrivilege":{"enabled":false},"SeLoadDriverPrivilege":{"enabled":false},"SeManageVolumePrivilege":{"enabled":false},"SeProfileSingleProcessPrivilege":{"enabled":false},"SeRemoteShutdownPrivilege":{"enabled":false},"SeRestorePrivilege":{"enabled":false},"SeSecurityPrivilege":{"enabled":false},"SeShutdownPrivilege":{"enabled":false},"SeSystemEnvironmentPrivilege":{"enabled":false},"SeSystemProfilePrivilege":{"enabled":false},"SeSystemtimePrivilege":{"enabled":false},"SeTakeOwnershipPrivilege":{"enabled":false},"SeTimeZonePrivilege":{"enabled":false},"SeUndockPrivilege":{"enabled":false}}}
2020-02-03T14:18:17.869+0100	INFO	helper/privileges_windows.go:87	SeDebugPrivilege is enabled. SeDebugPrivilege=(Enabled)
2020-02-03T14:18:17.872+0100	INFO	cfgfile/reload.go:172	Config reloader started
2020-02-03T14:18:17.874+0100	INFO	cfgfile/reload.go:227	Loading of config files completed.
2020-02-03T14:18:20.873+0100	INFO	add_cloud_metadata/add_cloud_metadata.go:347	add_cloud_metadata: hosting provider type not detected.
2020-02-03T14:18:47.868+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":109,"time":{"ms":109}},"total":{"ticks":296,"time":{"ms":296},"value":0},"user":{"ticks":187,"time":{"ms":187}}},"handles":{"open":201},"info":{"ephemeral_id":"87e80cc2-d7a0-4094-a684-6e1392aece00","uptime":{"ms":30853}},"memstats":{"gc_next":10286368,"memory_alloc":8293544,"memory_total":18581880,"rss":37154816},"runtime":{"goroutines":36}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":52,"batches":5,"total":52},"type":"console","write":{"bytes":87072}},"pipeline":{"clients":3,"events":{"active":0,"published":52,"total":52},"queue":{"acked":52}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":2,"success":2},"fsstat":{"events":1,"success":1},"memory":{"events":3,"success":3},"network":{"events":6,"success":6},"process":{"events":30,"success":30},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":4}}}}}
6

Hi again!

It looks weird to me that Metricbeat logs do not show any emitted event. Could you confirm that you don't see any events in this output?

Also could you confirm that system module is actually enabled: ./metricbeat modules list?

C.

7

It looks weird to me that Metricbeat logs do not show any emitted event. Could you confirm that you don't see any events in this output?

How can you tell that no data are emitted? My index is fulfilled and the following lines appears :

2020-02-03T14:32:47.903+0100|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1750,"time":{"ms":47}},"total":{"ticks":3171,"time":{"ms":47},"value":3171},"user":{"ticks":1421}},"handles":{"open":221},"info":{"ephemeral_id":"87e80cc2-d7a0-4094-a684-6e1392aece00","uptime":{"ms":870867}},"memstats":{"gc_next":10096336,"memory_alloc":5982360,"memory_total":181447592,"rss":159744},"runtime":{"goroutines":36}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":51,"batches":6,"total":51},"write":{"bytes":86561}},"pipeline":{"clients":3,"events":{"active":0,"published":51,"total":51},"queue":{"acked":51}}},"metricbeat":{"system":{"cpu":{"events":4,"success":4},"filesystem":{"events":2,"success":2},"fsstat":{"events":1,"success":1},"memory":{"events":3,"success":3},"network":{"events":6,"success":6},"process":{"events":30,"success":30},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}}}}}
2020-02-03T14:33:17.903+0100|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1765,"time":{"ms":15}},"total":{"ticks":3202,"time":{"ms":31},"value":3202},"user":{"ticks":1437,"time":{"ms":16}}},"handles":{"open":224},"info":{"ephemeral_id":"87e80cc2-d7a0-4094-a684-6e1392aece00","uptime":{"ms":900866}},"memstats":{"gc_next":10104784,"memory_alloc":7796384,"memory_total":186912264,"rss":-299008},"runtime":{"goroutines":36}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":47,"batches":6,"total":47},"write":{"bytes":82708}},"pipeline":{"clients":3,"events":{"active":0,"published":47,"total":47},"queue":{"acked":47}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":6,"success":6},"process":{"events":29,"success":29},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3},"uptime":{"events":1,"success":1}}}}}}
Name Health Status Primaries Replicas Docs Storage
metricbeat-qa-7.2.0-2020.02.04 green open 1 1 48 34.9mb
metricbeat-qa-7.2.0-2020.02.03 green open 1 1 76913 63mb

Also could you confirm that system module is actually enabled: ./metricbeat modules list ?

PS C:\metricbeat-7.2.0> .\metricbeat.exe modules list
Enabled:
system

Thanks for you help :slight_smile:

8

Hi!

I was meaning log like this:

2020-02-04T17:20:37.063+0200	DEBUG	[processors]	processing/processors.go:186	Publish event: {
  "@timestamp": "2020-02-04T15:20:36.983Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "8.0.0"
  },
  "system": {
    "process": {
      "state": "running",
      "memory": {
        "size": 7833358336,
        "rss": {
          "bytes": 530190336,
          "pct": 0.0154
        },
        "share": 0
      },
      "cmdline": "/Applications/zoom.us.app/Contents/MacOS/zoom.us",
      "cpu": {
        "total": {
          "value": 0,
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "start_time": "2020-02-04T14:04:33.544Z"
      }
    }
  },
  "agent": {
    "type": "metricbeat",
    "ephemeral_id": "7eeeba46-0258-4e26-ba0c-45427e39e2c2",
    "hostname": "some",
    "id": "bd12ee45-881f-48e4-af20-13b139548607",
    "version": "8.0.0"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "host": {
    "id": "883134FF-0EC4-5E1B-9F9E-FD06FB681D84",
    "hostname": "some",
    "architecture": "x86_64",
    "name": "some",
    "os": {
      "name": "Mac OS X",
      "kernel": "18.7.0",
      "build": "18G95",
      "platform": "darwin",
      "version": "10.14.6",
      "family": "darwin"
    }
  },
  "event": {
    "dataset": "system.process",
    "module": "system",
    "duration": 79563192
  },
  "metricset": {
    "period": 10000,
    "name": "process"
  },
  "service": {
    "type": "system"
  },
  "process": {
    "args": [
      "/Applications/zoom.us.app/Contents/MacOS/zoom.us"
    ],
    "name": "zoom.us",
    "pid": 17955,
    "ppid": 1,
    "pgid": 17955,
    "executable": "/Applications/zoom.us.app/Contents/MacOS/zoom.us"
  },
  "user": {
    "name": "some"
  }
}

So system module specific events. The other lines you see are for Metricbeat's monitoring itself.

C.

9

sorry for the misunderstand. Yes i have an output. Here is the result :

{
  "@timestamp": "2020-02-04T15:37:35.962Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.0"
  },
  "metricset": {
    "name": "uptime"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "uptime": {
      "duration": {
        "ms": 310816562962
      }
    }
  },
  "agent": {
    "version": "7.2.0",
    "type": "metricbeat",
    "ephemeral_id": "3438a502-cd07-4216-8ce5-afd6d85aad14",
    "hostname": "SGSOQ01",
    "id": "df327b1b-5d9d-4a10-9dd6-c5996e24d01a"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "architecture": "x86_64",
    "os": {
      "family": "windows",
      "name": "Windows Server 2016 Standard",
      "kernel": "10.0.14393.3442 (rs1_release.191219-1727)",
      "build": "14393.3443",
      "platform": "windows",
      "version": "10.0"
    },
    "id": "b64f60a7-5d8f-4832-94d1-0c636151a7e5",
    "name": "SGSOQ01",
    "hostname": "SGSOQ01"
  },
  "event": {
    "dataset": "system.uptime",
    "module": "system"
  }
}
{
  "@timestamp": "2020-02-04T15:37:35.962Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.0"
  },
  "system": {
    "memory": {
      "total": 17179398144,
      "used": {
        "bytes": 12424290304,
        "pct": 0.7232
      },
      "free": 4755107840,
      "actual": {
        "used": {
          "bytes": 12424290304,
          "pct": 0.7232
        },
        "free": 4755107840
      },
      "swap": {
        "total": 19729534976,
        "used": {
          "bytes": 12980355072,
          "pct": 0.6579
        },
        "free": 6749179904
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "hostname": "SGSOQ01",
    "architecture": "x86_64",
    "name": "SGSOQ01",
    "os": {
      "build": "14393.3443",
      "platform": "windows",
      "version": "10.0",
      "family": "windows",
      "name": "Windows Server 2016 Standard",
      "kernel": "10.0.14393.3442 (rs1_release.191219-1727)"
    },
    "id": "b64f60a7-5d8f-4832-94d1-0c636151a7e5"
  },
  "agent": {
    "id": "df327b1b-5d9d-4a10-9dd6-c5996e24d01a",
    "version": "7.2.0",
    "type": "metricbeat",
    "ephemeral_id": "3438a502-cd07-4216-8ce5-afd6d85aad14",
    "hostname": "SGSOQ01"
  },
  "event": {
    "dataset": "system.memory",
    "module": "system"
  },
  "metricset": {
    "name": "memory"
  },
  "service": {
    "type": "system"
  }
}
10

Thank you for clarifying this!

So since Metricbeat is collecting metrics with the Module, sth is missing in the shipping of the metrics to Elasticsearch.

I would suggest to ship them directly to Elasticsearch putting Logstash out for now, so as to possibly isolate the issue in Metricbeat->Logstash communication.

C.

11

Thank you. It is working now and the problem seems to be linked with Logstash.
Any idea of what could be wrong?

12

Hey!

You should check in Metricbeats logs (in debug mode like ./metricbeat -e -d "*") and what errors give when trying to send events to Logstash. It might be a connectivity error. If events are normally sent to Logstash then look into the Logstash logs to see if something is failing while processing the events.

C.

13

I think the problem is linked to the index definition in logstash :

index => "%{[@metadata][beat]}-qa-%{[@metadata][version]}-%{+YYYY.MM.dd}"

As you can see i add "-qa" in the index name. If i remove it, it works.
Unfortunately I didn't see anything related to this in the logs.

Is it possible to push metrics throught a custom index?

14

Nevermind, i see we can "play" with tags in metricbeat. It will do the trick.

Thank you very much for your help. I appreciate.

Have a good day :slight_smile:

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.