Filebeat log sending failed

[brunoju (zhuwenqing)]

EK + Filebeat

But filebeat log sending to Kibana failed with unknown reason

Here is the conf

==================================
filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  _source.enabled: true

setup.dashboards.enabled: true
setup.kibana:
  host: "localhost:5601"
  space.id:

output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "http"
  username: "elastic"
  password: "NhUYVPIsnPLsfI9e"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
===========================

Here is the partial log:

=================================
2019-04-30T15:41:59.315+0800	INFO	instance/beat.go:280	Setup Beat: filebeat; Version: 7.0.0
2019-04-30T15:41:59.315+0800	INFO	[index-management]	idxmgmt/std.go:165	Set output.elasticsearch.index to 'filebeat-7.0.0' as ILM is enabled.
2019-04-30T15:41:59.317+0800	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
2019-04-30T15:41:59.317+0800	INFO	[publisher]	pipeline/module.go:97	Beat name: Matrix
2019-04-30T15:41:59.318+0800	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
2019-04-30T15:41:59.321+0800	INFO	elasticsearch/client.go:734	Attempting to connect to Elasticsearch version 7.0.0
2019-04-30T15:41:59.348+0800	INFO	add_cloud_metadata/add_cloud_metadata.go:345	add_cloud_metadata: hosting provider type detected as ecs, metadata={"availability_zone":"cn-shanghai-b","instance":{"id":"i-uf65w0dtnuug27oeq6ih"},"provider":"ecs","region":"cn-shanghai"}
2019-04-30T15:41:59.358+0800	INFO	[index-management]	idxmgmt/std.go:223	Auto ILM enable success.
2019-04-30T15:41:59.391+0800	INFO	[index-management]	idxmgmt/std.go:238	ILM policy successfully loaded.
2019-04-30T15:41:59.391+0800	INFO	[index-management]	idxmgmt/std.go:361	Set setup.template.name to '{filebeat-7.0.0 {now/d}-000001}' as ILM is enabled.
2019-04-30T15:41:59.391+0800	INFO	[index-management]	idxmgmt/std.go:366	Set setup.template.pattern to 'filebeat-7.0.0-*' as ILM is enabled.
2019-04-30T15:41:59.391+0800	INFO	[index-management]	idxmgmt/std.go:400	Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.0.0 {now/d}-000001} as ILM is enabled.
2019-04-30T15:41:59.391+0800	INFO	[index-management]	idxmgmt/std.go:404	Set settings.index.lifecycle.name in template to {filebeat-7.0.0 map[policy:{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}]} as ILM is enabled.
2019-04-30T15:41:59.395+0800	INFO	template/load.go:82	Loading template for Elasticsearch version: 7.0.0
2019-04-30T15:41:59.395+0800	INFO	template/load.go:84	Existing template will be overwritten, as overwrite is enabled.
2019-04-30T15:41:59.559+0800	INFO	template/load.go:145	Elasticsearch template with name 'filebeat-7.0.0' loaded
2019-04-30T15:41:59.559+0800	INFO	[index-management]	idxmgmt/std.go:272	Loaded index template.
2019-04-30T15:41:59.560+0800	INFO	[index-management]	idxmgmt/std.go:283	Write alias successfully generated.
2019-04-30T15:41:59.560+0800	INFO	kibana/client.go:118	Kibana url: http://localhost:5601
2019-04-30T15:41:59.731+0800	INFO	kibana/client.go:118	Kibana url: http://localhost:5601
2019-04-30T15:42:35.671+0800	INFO	instance/beat.go:741	Kibana dashboards successfully loaded.
2019-04-30T15:42:35.672+0800	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
2019-04-30T15:42:35.675+0800	INFO	elasticsearch/client.go:734	Attempting to connect to Elasticsearch version 7.0.0
2019-04-30T15:42:35.694+0800	INFO	kibana/client.go:118	Kibana url: http://localhost:5601

===================================

[Mario_Castro]

Hi @brunoju

Please, use markdown formatting so that we can read you configs easily.

You are not describing which module of Filebeat you are using. Please, also attach the filebeat config. Can you also describe a bit more the problem? You cannot see logs in Kibana but, can you see them in Elasticsearch connecting with a simple HTTP request?

Ensure that a Elasticsearch instance is running in your localhost too. I would also recommend that you change the password of that cluster, because you pasted it here.

[brunoju (zhuwenqing)]

Hi Mario,

Thanks a lot for your answer. I have found out the problem.
Just follow the command and open up the debug model to find the root cause:

sudo filebeat -e -d "*"