Filebeat cannot able to index into elasticsearch

Elastic Stack Beats
#1

Filebeat cannot able to index into elasticsearch

please find the error below, ES ping not happening. Please suggest

2017-03-01T13:14:44+05:30 DBG ES Ping(url=http://10.209.68.81:9201, timeout=1m30s)
2017-03-01T13:14:45+05:30 DBG Ping request failed with: Get http://10.209.68.81:9201: dial tcp 10.209.68.81:9201: connectex: No connection could be made because the target machine actively refused it.
2017-03-01T13:14:45+05:30 ERR Connecting error publishing events (retrying): Get http://10.209.68.81:9201: dial tcp 10.209.68.81:9201: connectex: No connection could be made because the target machine actively refused it.
2017-03-01T13:14:45+05:30 DBG send fail
2017-03-01T13:14:46+05:30 DBG Flushing spooler because of timeout. Events flushed: 0

#2

Also find below error too

2017-03-01T13:20:28+05:30 DBG ES Ping(url=http://10.209.68.81:9201, timeout=1m30s)
2017-03-01T13:20:28+05:30 DBG Ping request failed with: 401 Unauthorized
2017-03-01T13:20:28+05:30 ERR Connecting error publishing events (retrying): 401 Unauthorized
2017-03-01T13:20:28+05:30 DBG send fail
2017-03-01T13:20:30+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
2017-03-01T13:20:30+05:30 DBG Run prospector
2017-03-01T13:20:30+05:30 DBG Start next scan

#3

Moving your question to #beats:filebeat

I'd say that probably elasticsearch is not running at 10.209.68.81:9201

BTW I'd use 10.209.68.81:9200 instead

#4

will the ES port number matters?

Because i have added custom port number in ES.yml file where ES is working fine, But i couldnt able to index filebeat into ES.

Please help

#5

you getting 401 Unauthorized from elasticsearch. did you correctly configure the username and password?

#6

Yes Steffens, I ahve configured proper username and password.

By in case if i have not done properly, let me know what needs to be done please ?

#7

Hi Steffens, Please can you let us know what needs to be done for 401 unauthorized error.?

#8

are you using xpack or some proxy asking for authentication? Can you share your config (redact the password please)? Which filebeat version are you using?

#9

Im using x pack 5.1.1

filebeat version 5.1.1

May i know which config file i need to share please?

#10

Please find the below filebeat config file

filebeat.prospectors:

  • input_type: log
    paths:
    • /var/log/*.log
    • D:\Team\logs*.log
      document_type: log
      output.elasticsearch:

    Array of hosts to connect to.

    hosts: ["bngwidap107.aonnet.aon.net:9200"]

logging.level: debug
logging.selectors: ["*"]

please let us know if anything needed.

#11

Hi steffens, Please let us know what misght be causing the error.?

#12

please can someone help me in this issue, we are awaiting for your response.

#13

Hi steffen, Let us know what is the issue causing for us.

#14

Please be patient as this forum is manned by volunteers. As you have secured your cluster with X-Pack, you will need to configure Beats to take this into account as well.

#15

Hi Christian,

Yes we have configured with beats security since then same issue we are facing.

For now i have made xpack.security.enabled: false in elasticsearch config and tried to index filebeat into elasticsearch, still filebeat not indexing.

Please find filebeat log below where no error appersa i seems to be.

2017-03-03T18:18:55+05:30 DBG Prospector states cleaned up. Before: 18, After: 18
2017-03-03T18:18:56+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
2017-03-03T18:19:01+05:30 DBG Flushing spooler because of timeout. Events flushed: 0
2017-03-03T18:19:05+05:30 DBG Run prospector
2017-03-03T18:19:05+05:30 DBG Start next scan
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.03.03_12.49.43.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.03.03_12.49.43.log, offset: 903525
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut_17.03.03_12.49.43.log
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.05.33.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.05.33.log, offset: 1045061
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut_17.03.03_13.05.33.log
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.16.09.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.16.09.log, offset: 931302
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut_17.03.03_13.16.09.log
2017-03-03T18:19:05+05:30 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.47.27.log
2017-03-03T18:19:05+05:30 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.03.03_13.47.27.log, offset: 1039071
2017-03-03T18:19:05+05:30 DBG File didn't change: D:\Team\logs\SystemOut_17.03.03_13.47.27.log

also please find elasticsearch log below for reference

[2017-03-03T17:54:20,639][ERROR][o.e.x.m.AgentService ] [bngwidap107.aonnet.aon.net] exception when exporting documents
org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:148) ~[x-pack-5.1.1.jar:5.1.1]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.close(ExportBulk.java:77) ~[x-pack-5.1.1.jar:5.1.1]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:194) ~[x-pack-5.1.1.jar:5.1.1]
at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:208) [x-pack-5.1.1.jar:5.1.1]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?]
... 4 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default_local] reports failures when exporting documents
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:121) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:111) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?]
... 4 more
[2017-03-03T17:54:20,658][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]
[2017-03-03T17:54:21,921][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]
[2017-03-03T17:54:21,926][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]
[2017-03-03T17:54:24,334][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]
[2017-03-03T17:54:44,172][ERROR][o.e.x.m.c.c.ClusterStateCollector] [bngwidap107.aonnet.aon.net] collector [cluster-state-collector] timed out when collecting data
[2017-03-03T17:54:45,479][INFO ][o.e.c.r.a.AllocationService] [bngwidap107.aonnet.aon.net] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
[2017-03-03T17:54:45,636][INFO ][o.e.c.m.MetaDataMappingService] [bngwidap107.aonnet.aon.net] [winlogbeat-2017.02.10/FgUngVG-Q0C-HeRUh19QBQ] update_mapping [wineventlog]

#16

Please format logs and config files with </> button.

You config is kind of incomplete... by redacting username/password I didn't mean to drop them.

Plus, how did you create your user for writing to ES?

No idea if/how logs from ES are related to your problem.

when posting logs, read them first. The filebeat log says nothing about faild send-attempts. But files not being updated... did you change any logs? Have you tried to delete the registry file?

I adapted the configuration from Christians link a little to create me a beat_user for writing to filebeat-*, metricbeat-* and packetbeat-* index (if you did just copy the samples as is, you would have no credentials for filebeat):

POST _xpack/security/role/beat_writer
{
  "cluster": ["manage_index_templates", "monitor"],
  "indices": [
    {
      "names": [ "filebeat-*", "metricbeat-*", "packetbeat-*" ], 
      "privileges": ["read","write","create_index"]
    }
  ]
}

POST /_xpack/security/user/beat_user
{
  "password" : "changeme",
  "roles" : [ "beat_writer"],
  "full_name" : "Internal Beat User"
}

And the beats output configuration:

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "beat_user"
  password: "changeme"
#17

Hi Steffens,

Thanks for the above input.

Please can you let me know how to format logs and config files with </> button.

also i have configured elastic user in filebeat output.elaticsearch. please let me is that the proper configuration which user writing to ES.?

As you mentioned samples above, i have created an beat_writer role and beat_user, still i couldnt able to index filebeat into ES.

Please do help.

#18

Hi steffens,

continously im getting the below error in filebeat logs.

2017-03-09T18:43:24+05:30 DBG send completed
2017-03-09T18:43:24+05:30 DBG output worker: publish 50 events
2017-03-09T18:43:24+05:30 DBG PublishEvents: 50 events have been published to elasticsearch in 1.9989ms.
2017-03-09T18:43:24+05:30 WARN Can not index event (status=404): {"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"filebeat-2017.03.09","index_uuid":"na","index":"filebeat-2017.03.09"}
2017-03-09T18:43:24+05:30 WARN Can not index event (status=404): {"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"filebeat-2017.03.09","index_uuid":"na","index":"filebeat-2017.03.09"}
2017-03-09T18:43:24+05:30 WARN Can not index event (status=404): {"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"filebeat-2017.03.09","index_uuid":"na","index":"filebeat-2017.03.09"}

please let us know how do we index filebeat into elasticsearch.?

#19

Could you quickly try to ingest with the admin user to see if it is an user access issue or not?

#20

Ruflin, May i know what admin user you are talking about?