Filebeat is not creating index in ES?

vikas_gopal · September 4, 2017, 9:08pm

Hi Experts,

For Demo I am ingesting Apache data using filebeat into ES. My filebeat.yml is

filebeat.modules:
- module: apache2
  access:
   var.paths: ["E:/ELK/Module1/apache1.log"]
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  template.name: "filebeat"
  template.path: "filebeat.template.json"
  template.overwrite: false

My Configuration is ok as I am getting below in PS

PS E:\ELK\filebeat> .\filebeat.exe -c filebeat.yml -configtest
Config OK

I got no error while starting filebeat service as below

> PS E:\ELK\filebeat> Start-Service filebeat

After above my Filebeat log says

> 2017-09-05T02:36:57+05:30 INFO Home path: [E:\ELK\filebeat] Config path: [E:\ELK\filebeat] Data path: [C:\\ProgramData\\filebeat] Logs path: [E:\ELK\filebeat\logs]
> 2017-09-05T02:36:57+05:30 INFO Metrics logging every 30s
> 2017-09-05T02:36:58+05:30 INFO Setup Beat: filebeat; Version: 5.5.2
> 2017-09-05T02:36:58+05:30 INFO Loading template enabled. Reading template file: E:\ELK\filebeat\filebeat.template.json
> 2017-09-05T02:36:58+05:30 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: E:\ELK\filebeat\filebeat.template-es2x.json
> 2017-09-05T02:36:58+05:30 INFO Loading template enabled for Elasticsearch 6.x. Reading template file: E:\ELK\filebeat\filebeat.template-es6x.json
> 2017-09-05T02:36:58+05:30 INFO Elasticsearch url: http://localhost:9200
> 2017-09-05T02:36:58+05:30 INFO Activated elasticsearch as output plugin.
> 2017-09-05T02:36:58+05:30 INFO Publisher name: DESKTOP-CVGTF3S
> 2017-09-05T02:36:58+05:30 INFO Flush Interval set to: 1s
> 2017-09-05T02:36:58+05:30 INFO Max Bulk Size set to: 50
> 2017-09-05T02:36:58+05:30 INFO filebeat start running.
> 2017-09-05T02:36:58+05:30 INFO Elasticsearch url: http://localhost:9200
> 2017-09-05T02:36:58+05:30 INFO Connected to Elasticsearch version 5.5.2
> 2017-09-05T02:36:58+05:30 INFO Registry file set to: C:\ProgramData\filebeat\registry
> 2017-09-05T02:36:58+05:30 INFO Loading registrar data from C:\ProgramData\filebeat\registry
> 2017-09-05T02:36:58+05:30 INFO States Loaded from registrar: 1
> 2017-09-05T02:36:58+05:30 INFO Loading Prospectors: 2
> 2017-09-05T02:36:58+05:30 INFO Starting Registrar
> 2017-09-05T02:36:58+05:30 INFO Start sending events to output
> 2017-09-05T02:36:58+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
> 2017-09-05T02:36:58+05:30 INFO Prospector with previous states loaded: 0
> 2017-09-05T02:36:58+05:30 INFO Starting prospector of type: log; id: 1449209540265381168 
> 2017-09-05T02:36:58+05:30 INFO Prospector with previous states loaded: 0
> 2017-09-05T02:36:58+05:30 INFO Starting prospector of type: log; id: 3723623454282510377 
> 2017-09-05T02:36:58+05:30 INFO Loading and starting Prospectors completed. Enabled prospectors: 2
> 2017-09-05T02:36:58+05:30 INFO Harvester started for file: E:\ELK\Module1\apache1.log
> 2017-09-05T02:36:58+05:30 INFO Connected to Elasticsearch version 5.5.2
> 2017-09-05T02:36:58+05:30 INFO Trying to load template for client: http://localhost:9200
> 2017-09-05T02:36:58+05:30 INFO Template already exists and will not be overwritten.

Not sure what could be the problem here . Can someone please help me .

Regards
VG

vikas_gopal · September 4, 2017, 9:29pm

I waited for couple of more min and now I see some new lines added to the filebeat log . It says

> 2017-09-05T02:55:27+05:30 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.es.call_count.PublishEvents=1 libbeat.es.publish.read_bytes=2638 libbeat.es.publish.write_bytes=30350 libbeat.publisher.published_events=2047
> 2017-09-05T02:55:57+05:30 INFO No non-zero metrics in the last 30s
> 2017-09-05T02:55:57+05:30 INFO Error publishing events (retrying): temporary bulk send failure
> 2017-09-05T02:55:58+05:30 INFO Connected to Elasticsearch version 5.5.2
> 2017-09-05T02:55:58+05:30 INFO Trying to load template for client: http://localhost:9200
> 2017-09-05T02:55:58+05:30 INFO Template already exists and will not be overwritten.
> 2017-09-05T02:56:27+05:30 INFO Non-zero metrics in the last 30s: libbeat.es.call_count.PublishEvents=1 libbeat.es.publish.read_bytes=971 libbeat.es.publish.write_bytes=29745 libbeat.es.published_but_not_acked_events=50
> 2017-09-05T02:56:57+05:30 INFO No non-zero metrics in the last 30s
> 2017-09-05T02:56:58+05:30 INFO Error publishing events (retrying): temporary bulk send failure
> 2017-09-05T02:57:00+05:30 INFO Connected to Elasticsearch version 5.5.2
> 2017-09-05T02:57:00+05:30 INFO Trying to load template for client: http://localhost:9200
> 2017-09-05T02:57:00+05:30 INFO Template already exists and will not be overwritten.
> 2017-09-05T02:57:27+05:30 INFO Non-zero metrics in the last 30s: libbeat.es.call_count.PublishEvents=1 libbeat.es.publish.read_bytes=971 libbeat.es.publish.write_bytes=29745 libbeat.es.published_but_not_acked_events=50
> 2017-09-05T02:57:57+05:30 INFO No non-zero metrics in the last 30s
> 2017-09-05T02:58:00+05:30 INFO Error publishing events (retrying): temporary bulk send failure
> 2017-09-05T02:58:04+05:30 INFO Connected to Elasticsearch version 5.5.2
> 2017-09-05T02:58:04+05:30 INFO Trying to load template for client: http://localhost:9200
> 2017-09-05T02:58:04+05:30 INFO Template already exists and will not be overwritten.
> 2017-09-05T02:58:27+05:30 INFO Non-zero metrics in the last 30s: libbeat.es.call_count.PublishEvents=1 libbeat.es.publish.read_bytes=972 libbeat.es.publish.write_bytes=29745 libbeat.es.published_but_not_acked_events=50
> 2017-09-05T02:58:57+05:30 INFO No non-zero metrics in the last 30s
> 2017-09-05T02:59:04+05:30 INFO Error publishing events (retrying): temporary bulk send failure

vikas_gopal · September 4, 2017, 9:39pm

Even I removed the template stenza because logs are keep on saying cannot overwrite template . Also I changed the file path and name. Still this does not resolve the issue , still I don't see any Index in ES or in Kibana.New filebeat.yml is like

filebeat.modules:
- module: apache2
  access:
   var.paths: ["E:/ELK/abc.log"]
output.elasticsearch:
  hosts: ["localhost:9200"]

vikas_gopal · September 4, 2017, 10:45pm

Well after lots of efforts I was able to resolve the issue . It was to do with registry file, so I mentioned the path and things work perfectly.So this is what my new yml looks like.

filebeat.modules:
- module: apache2
  access:
   var.paths: ["E:/ELK/Module1/apache.log"]
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
filebeat.registry_file: E:/ELK/registry

Hope It will help someone.

Regards
VG

system · October 2, 2017, 10:46pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.