Filebeat cannot able to index into elasticsearch

Sujith · March 1, 2017, 7:47am

please find the error below, ES ping not happening. Please suggest

2017-03-01T13:14:44+05:30 DBG ES Ping(url=http://10.209.68.81:9201, timeout=1m30s)
2017-03-01T13:14:45+05:30 DBG Ping request failed with: Get http://10.209.68.81:9201: dial tcp 10.209.68.81:9201: connectex: No connection could be made because the target machine actively refused it.
2017-03-01T13:14:45+05:30 ERR Connecting error publishing events (retrying): Get http://10.209.68.81:9201: dial tcp 10.209.68.81:9201: connectex: No connection could be made because the target machine actively refused it.
2017-03-01T13:14:45+05:30 DBG send fail
2017-03-01T13:14:46+05:30 DBG Flushing spooler because of timeout. Events flushed: 0

Sujith · March 1, 2017, 7:51am

Also find below error too

2017-03-01T13:20:28+05:30 DBG ES Ping(url=http://10.209.68.81:9201, timeout=1m30s)
2017-03-01T13:20:28+05:30 DBG Ping request failed with: 401 Unauthorized
2017-03-01T13:20:28+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2017-03-01T13:20:28+05:30 DBG send fail
2017-03-01T13:20:30+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
2017-03-01T13:20:30+05:30 DBG Run prospector
2017-03-01T13:20:30+05:30 DBG Start next scan

dadoonet · March 1, 2017, 8:18am

Moving your question to #beats:filebeat

I'd say that probably elasticsearch is not running at 10.209.68.81:9201

BTW I'd use 10.209.68.81:9200 instead

Sujith · March 1, 2017, 8:24am

will the ES port number matters?

Because i have added custom port number in ES.yml file where ES is working fine, But i couldnt able to index filebeat into ES.

Please help

steffens · March 1, 2017, 5:49pm

you getting 401 Unauthorized from elasticsearch. did you correctly configure the username and password?

Sujith · March 2, 2017, 8:01am

Yes Steffens, I ahve configured proper username and password.

By in case if i have not done properly, let me know what needs to be done please ?

Sujith · March 2, 2017, 12:22pm

Hi Steffens, Please can you let us know what needs to be done for 401 unauthorized error.?

steffens · March 2, 2017, 1:05pm

are you using xpack or some proxy asking for authentication? Can you share your config (redact the password please)? Which filebeat version are you using?

Sujith · March 2, 2017, 1:13pm

Im using x pack 5.1.1

filebeat version 5.1.1

May i know which config file i need to share please?

Sujith · March 2, 2017, 1:32pm

Please find the below filebeat config file

filebeat.prospectors:

input_type: log
paths:
- /var/log/*.log
- D:\Team\logs*.log
  document_type: log
  output.elasticsearch:
Array of hosts to connect to.
hosts: ["bngwidap107.aonnet.aon.net:9200"]

logging.level: debug
logging.selectors: ["*"]

please let us know if anything needed.

Sujith · March 2, 2017, 2:15pm

Hi steffens, Please let us know what misght be causing the error.?

Sujith · March 3, 2017, 7:20am

please can someone help me in this issue, we are awaiting for your response.

Sujith · March 3, 2017, 11:39am

Hi steffen, Let us know what is the issue causing for us.

Christian_Dahlqvist · March 3, 2017, 12:03pm

Please be patient as this forum is manned by volunteers. As you have secured your cluster with X-Pack, you will need to configure Beats to take this into account as well.

Sujith · March 3, 2017, 12:53pm

Hi Christian,

Yes we have configured with beats security since then same issue we are facing.

For now i have made xpack.security.enabled: false in elasticsearch config and tried to index filebeat into elasticsearch, still filebeat not indexing.

Please find filebeat log below where no error appersa i seems to be.

2017-03-03T18:18:55+05:30 DBG Prospector states cleaned up. Before: 18, After: 18
2017-03-03T18:18:56+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
2017-03-03T18:19:01+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
2017-03-03T18:19:05+05:30 DBG Run prospector
2017-03-03T18:19:05+05:30 DBG Start next scan
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.03.03_12.49.43.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.03.03_12.49.43.log, offset: 903525
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut_17.03.03_12.49.43.log
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.05.33.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.05.33.log, offset: 1045061
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut_17.03.03_13.05.33.log
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.16.09.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.16.09.log, offset: 931302
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut_17.03.03_13.16.09.log
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.47.27.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.47.27.log, offset: 1039071
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut_17.03.03_13.47.27.log

also please find elasticsearch log below for reference

[2017-03-03T17:54:20,639][ERROR][o.e.x.m.AgentService ] [bngwidap107.aonnet.aon.net] exception when exporting documents
org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:148) ~[x-pack-5.1.1.jar:5.1.1]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.close(ExportBulk.java:77) ~[x-pack-5.1.1.jar:5.1.1]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:194) ~[x-pack-5.1.1.jar:5.1.1]
at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:208) [x-pack-5.1.1.jar:5.1.1]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?]
... 4 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default_local] reports failures when exporting documents
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:121) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:111) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?]
... 4 more
[2017-03-03T17:54:20,658][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]
[2017-03-03T17:54:21,921][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]
[2017-03-03T17:54:21,926][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]
[2017-03-03T17:54:24,334][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]
[2017-03-03T17:54:44,172][ERROR][o.e.x.m.c.c.ClusterStateCollector] [bngwidap107.aonnet.aon.net] collector [cluster-state-collector] timed out when collecting data
[2017-03-03T17:54:45,479][INFO ][o.e.c.r.a.AllocationService] [bngwidap107.aonnet.aon.net] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
[2017-03-03T17:54:45,636][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]

steffens · March 3, 2017, 5:30pm

Please format logs and config files with </> button.

You config is kind of incomplete... by redacting username/password I didn't mean to drop them.

Plus, how did you create your user for writing to ES?

No idea if/how logs from ES are related to your problem.

when posting logs, read them first. The filebeat log says nothing about faild send-attempts. But files not being updated... did you change any logs? Have you tried to delete the registry file?

I adapted the configuration from Christians link a little to create me a beat_user for writing to filebeat-*, metricbeat-* and packetbeat-* index (if you did just copy the samples as is, you would have no credentials for filebeat):

POST _xpack/security/role/beat_writer
{
  "cluster": ["manage_index_templates", "monitor"],
  "indices": [
    {
      "names": [ "filebeat-*", "metricbeat-*", "packetbeat-*" ], 
      "privileges": ["read","write","create_index"]
    }
  ]
}

POST /_xpack/security/user/beat_user
{
  "password" : "changeme",
  "roles" : [ "beat_writer"],
  "full_name" : "Internal Beat User"
}

And the beats output configuration:

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "beat_user"
  password: "changeme"

Sujith · March 6, 2017, 11:33am

Hi Steffens,

Thanks for the above input.

Please can you let me know how to format logs and config files with </> button.

also i have configured elastic user in filebeat output.elaticsearch. please let me is that the proper configuration which user writing to ES.?

As you mentioned samples above, i have created an beat_writer role and beat_user, still i couldnt able to index filebeat into ES.

Please do help.

Sujith · March 9, 2017, 1:59pm

Hi steffens,

continously im getting the below error in filebeat logs.

2017-03-09T18:43:24+05:30 DBG send completed
2017-03-09T18:43:24+05:30 DBG output worker: publish 50 events
2017-03-09T18:43:24+05:30 DBG PublishEvents: 50 events have been published to elasticsearch in 1.9989ms.
2017-03-09T18:43:24+05:30 WARN Can not index event (status=404): {"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"filebeat-2017.03.09","index_uuid":"na","index":"filebeat-2017.03.09"}
2017-03-09T18:43:24+05:30 WARN Can not index event (status=404): {"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"filebeat-2017.03.09","index_uuid":"na","index":"filebeat-2017.03.09"}
2017-03-09T18:43:24+05:30 WARN Can not index event (status=404): {"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"filebeat-2017.03.09","index_uuid":"na","index":"filebeat-2017.03.09"}

please let us know how do we index filebeat into elasticsearch.?

ruflin · March 10, 2017, 11:48pm

Could you quickly try to ingest with the admin user to see if it is an user access issue or not?

Sujith · March 11, 2017, 11:03am

Ruflin, May i know what admin user you are talking about?

Topic		Replies	Views
Filebeat - Error connecting to publish events Beats filebeat	7	935	November 22, 2017
Index not creating Beats	6	1490	July 29, 2016
Filebeat unable to send logs to logstash Beats filebeat	5	1988	July 3, 2017
Filebeat not sending logs to Elasticsearch Beats filebeat	5	3509	July 6, 2016
Couldn't connect to elasticsearch cluster hosted in the cloud Elasticsearch	2	428	October 2, 2018

Filebeat cannot able to index into elasticsearch

Array of hosts to connect to.

Related Topics