Central Management output to Elasticsearch

Rasmus_Fredensborg_J · January 2, 2019, 1:29pm

Hi,
Thank you for your amazing products. I just started using Central management for beats and got everything working except for output to Elasticsearch cloud. I looked at this issue but it doesn't seem to be related to my issue, as I'm trying to output to cloud. After configuring beat in central management, the management.yml on the filebeat file looks like this:

configok: true
configs:
- type: filebeat.inputs
  blocks:
  - raw:
      enabled: true
      multiline.match: after
      multiline.negate: false
      multiline.pattern: ^[[:space:]]
      paths:
      - /emento/logs/*/system.sync.log
      pipeline: system.sync.log
      type: log
  - raw:
      enabled: true
      paths:
      - /emento/logs/*/audit.sync.log
      pipeline: audit.sync.log
      type: log
  - raw:
      enabled: true
      paths:
      - /emento/logs/*/access.sync.log
      pipeline: access.sync.log
      type: log
  - raw:
      enabled: true
      multiline.match: after
      multiline.negate: false
      multiline.pattern: ^[[:space:]]
      paths:
      - /emento/logs/*/sensitivity.log
      pipeline: sensitivity.log
      type: log
- type: output
  blocks:
  - raw:
      elasticsearch:
        hosts:
        - https://***.eu-central-1.aws.cloud.es.io:9243
        password: ****
        username: elastic
      output: elasticsearch

(I masked the elasticsearch host url and password with *'s )

The filebeat log does not reveal any issues whatsoever, when receiving the new config:

2019-01-02T13:07:28.861Z	INFO	[centralmgmt]	management/manager.go:176	New configurations retrieved
2019-01-02T13:07:28.861Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for filebeat.inputs
2019-01-02T13:07:28.872Z	INFO	log/input.go:138	Configured paths: [/emento/logs/*/system.sync.log]
2019-01-02T13:07:28.872Z	INFO	input/input.go:114	Starting input of type: log; ID: 1093171485359595733 
2019-01-02T13:07:28.873Z	INFO	log/input.go:138	Configured paths: [/emento/logs/*/audit.sync.log]
2019-01-02T13:07:28.873Z	INFO	input/input.go:114	Starting input of type: log; ID: 11179040793685463107 
2019-01-02T13:07:28.873Z	INFO	log/input.go:138	Configured paths: [/emento/logs/*/access.sync.log]
2019-01-02T13:07:28.873Z	INFO	input/input.go:114	Starting input of type: log; ID: 4488876039308697840 
2019-01-02T13:07:28.874Z	INFO	log/input.go:138	Configured paths: [/emento/logs/*/sensitivity.log]
2019-01-02T13:07:28.874Z	INFO	input/input.go:114	Starting input of type: log; ID: 11261006042038773783 
2019-01-02T13:07:28.874Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for output
2019-01-02T13:07:28.877Z	INFO	elasticsearch/client.go:163	Elasticsearch url: https://****.eu-central-1.aws.cloud.es.io:9243
2019-01-02T13:07:28.880Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for filebeat.modules
2019-01-02T13:07:28.880Z	INFO	[centralmgmt]	management/manager.go:149	Storing new state
2019-01-02T13:07:28.894Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/central/access.sync.log
2019-01-02T13:07:28.895Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/config/system.sync.log
2019-01-02T13:07:28.896Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/central/audit.sync.log
2019-01-02T13:07:28.896Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/central/sensitivity.log
2019-01-02T13:07:28.896Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/contenteditor/system.sync.log
2019-01-02T13:07:28.901Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/config/audit.sync.log
2019-01-02T13:07:28.905Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/nocourse/sensitivity.log
2019-01-02T13:07:28.905Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/contenteditor/audit.sync.log
2019-01-02T13:07:28.905Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/nocourse/system.sync.log
2019-01-02T13:07:28.909Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/nocourse/audit.sync.log
2019-01-02T13:07:28.912Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/nophone/audit.sync.log
2019-01-02T13:07:28.913Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/nophone/system.sync.log
2019-01-02T13:07:28.913Z	INFO	log/harvester.go:254	Harvester started for file: /emento/logs/central/system.sync.log
2019-01-02T13:07:29.913Z	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(https://****.eu-central-1.aws.cloud.es.io:9243))
2019-01-02T13:07:29.974Z	INFO	elasticsearch/client.go:713	Connected to Elasticsearch version 6.5.1

The setup works when manually configuring the filebeat without central management. Let me know if I'm doing anything wrong.

Thanks

pierhugues · January 3, 2019, 1:35pm

Could you bump the debug level to debug I wonder if something happen and we do not expose the error correctly.

Rasmus_Fredensborg_J · January 4, 2019, 9:51am

Sure thing! Here's (what I think is) the relevant part of the log:

2019-01-04T09:35:09.534Z	DEBUG	[centralmgmt]	management/manager.go:164	Retrieving new configurations from Kibana
2019-01-04T09:35:10.368Z	DEBUG	[centralmgmt]	management/manager.go:172	configuration didn't change, sleeping
2019-01-04T09:35:10.368Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for output
2019-01-04T09:35:10.368Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for filebeat.inputs
2019-01-04T09:35:10.368Z	DEBUG	[centralmgmt]	cfgfile/list.go:62	Starting reload procedure, current runners: 0
2019-01-04T09:35:10.369Z	DEBUG	[centralmgmt]	cfgfile/list.go:80	Start list: 0, Stop list: 0
2019-01-04T09:35:10.369Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for filebeat.modules
2019-01-04T09:35:10.369Z	DEBUG	[centralmgmt]	cfgfile/list.go:62	Starting reload procedure, current runners: 0
2019-01-04T09:35:10.369Z	DEBUG	[centralmgmt]	cfgfile/list.go:80	Start list: 0, Stop list: 0
2019-01-04T09:35:39.535Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":4}},"total":{"ticks":80,"time":{"ms":92},"value":80},"user":{"ticks":80,"time":{"ms":88}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":7},"info":{"ephemeral_id":"0108f1a9-93aa-457a-9910-e82b662d4af6","uptime":{"ms":30081}},"memstats":{"gc_next":5955824,"memory_alloc":3075360,"memory_total":9718352,"rss":25477120}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":1},"load":{"1":0.47,"15":0.09,"5":0.24,"norm":{"1":0.47,"15":0.09,"5":0.24}}}}}}
2019-01-04T09:36:09.535Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0},"total":{"ticks":80,"value":80},"user":{"ticks":80}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":7},"info":{"ephemeral_id":"0108f1a9-93aa-457a-9910-e82b662d4af6","uptime":{"ms":60081}},"memstats":{"gc_next":5955824,"memory_alloc":3330408,"memory_total":9973400}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.29,"15":0.09,"5":0.22,"norm":{"1":0.29,"15":0.09,"5":0.22}}}}}}
2019-01-04T09:36:10.369Z	DEBUG	[centralmgmt]	management/manager.go:164	Retrieving new configurations from Kibana
2019-01-04T09:36:11.197Z	INFO	[centralmgmt]	management/manager.go:176	New configurations retrieved
2019-01-04T09:36:11.197Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for filebeat.inputs
2019-01-04T09:36:11.197Z	DEBUG	[centralmgmt]	cfgfile/list.go:62	Starting reload procedure, current runners: 0
2019-01-04T09:36:11.197Z	DEBUG	[centralmgmt]	cfgfile/list.go:80	Start list: 4, Stop list: 0
2019-01-04T09:36:11.198Z	DEBUG	[processors]	processors/processor.go:66	Processors: 
2019-01-04T09:36:11.198Z	DEBUG	[input]	log/config.go:200	recursive glob enabled
2019-01-04T09:36:11.198Z	DEBUG	[input]	log/input.go:147	exclude_files: []. Number of stats: 0
2019-01-04T09:36:11.198Z	DEBUG	[input]	log/input.go:168	input with previous states loaded: 0
2019-01-04T09:36:11.198Z	INFO	log/input.go:138	Configured paths: [/emento/logs/*/system.sync.log]
2019-01-04T09:36:11.198Z	DEBUG	[centralmgmt]	cfgfile/list.go:101	Starting runner: input [type=log, ID=1093171485359595733]
2019-01-04T09:36:11.198Z	INFO	input/input.go:114	Starting input of type: log; ID: 1093171485359595733 
2019-01-04T09:36:11.198Z	DEBUG	[processors]	processors/processor.go:66	Processors: 
2019-01-04T09:36:11.198Z	DEBUG	[input]	log/config.go:200	recursive glob enabled
2019-01-04T09:36:11.198Z	DEBUG	[input]	log/input.go:147	exclude_files: []. Number of stats: 0
2019-01-04T09:36:11.198Z	DEBUG	[input]	log/input.go:168	input with previous states loaded: 0
2019-01-04T09:36:11.198Z	INFO	log/input.go:138	Configured paths: [/emento/logs/*/audit.sync.log]
2019-01-04T09:36:11.198Z	DEBUG	[centralmgmt]	cfgfile/list.go:101	Starting runner: input [type=log, ID=11179040793685463107]
2019-01-04T09:36:11.198Z	INFO	input/input.go:114	Starting input of type: log; ID: 11179040793685463107 
2019-01-04T09:36:11.199Z	DEBUG	[processors]	processors/processor.go:66	Processors: 
2019-01-04T09:36:11.199Z	DEBUG	[input]	log/config.go:200	recursive glob enabled
2019-01-04T09:36:11.199Z	DEBUG	[input]	log/input.go:147	exclude_files: []. Number of stats: 0
2019-01-04T09:36:11.199Z	DEBUG	[input]	log/input.go:168	input with previous states loaded: 0
2019-01-04T09:36:11.199Z	INFO	log/input.go:138	Configured paths: [/emento/logs/*/access.sync.log]
2019-01-04T09:36:11.199Z	DEBUG	[centralmgmt]	cfgfile/list.go:101	Starting runner: input [type=log, ID=4488876039308697840]
2019-01-04T09:36:11.199Z	INFO	input/input.go:114	Starting input of type: log; ID: 4488876039308697840 
2019-01-04T09:36:11.199Z	DEBUG	[processors]	processors/processor.go:66	Processors: 
2019-01-04T09:36:11.199Z	DEBUG	[input]	log/config.go:200	recursive glob enabled
2019-01-04T09:36:11.199Z	DEBUG	[input]	log/input.go:147	exclude_files: []. Number of stats: 0
2019-01-04T09:36:11.199Z	DEBUG	[input]	log/input.go:168	input with previous states loaded: 0
2019-01-04T09:36:11.199Z	INFO	log/input.go:138	Configured paths: [/emento/logs/*/sensitivity.log]
2019-01-04T09:36:11.199Z	DEBUG	[centralmgmt]	cfgfile/list.go:101	Starting runner: input [type=log, ID=11261006042038773783]
2019-01-04T09:36:11.199Z	INFO	input/input.go:114	Starting input of type: log; ID: 11261006042038773783 
2019-01-04T09:36:11.199Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for output
2019-01-04T09:36:11.200Z	INFO	elasticsearch/client.go:163	Elasticsearch url: https://b1263c164b14417d8e1e5b340cc18392.eu-central-1.aws.cloud.es.io:9243
2019-01-04T09:36:11.200Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for filebeat.modules

Let me know if you need more of the log entries. The log does not reveal any complications, as far as I can see

However, I just noticed that after a few minutes the beat receives new configurations from Kibana again, even though I didn't change anything in Kibana:

2019-01-04T09:38:12.293Z	DEBUG	[centralmgmt]	management/manager.go:164	Retrieving new configurations from Kibana
2019-01-04T09:38:12.884Z	INFO	[centralmgmt]	management/manager.go:176	New configurations retrieved
2019-01-04T09:38:12.884Z	INFO	[centralmgmt]	management/manager.go:213	Applying settings for output

Hope this helps

Rasmus_Fredensborg_J · January 8, 2019, 8:13am

Do you need anything else, @pierhugues?

Rasmus_Fredensborg_J · January 18, 2019, 11:26am

Any updates on this, @pierhugues?

pierhugues · January 18, 2019, 11:12pm

Sorry for the delay looking at the log, I see it it did receive a few configuration, the output is configured and the input is received (4?). In the log do you see it tried to open the files?

I don't see any errors, do some happen after?

Rasmus_Fredensborg_J · January 22, 2019, 11:18am

I can see in the log, that it starts harvesting all the right files, but nothing is received in Kibana. No errors happen after

system · February 19, 2019, 11:18am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.