Filebeat error connecting to Elasticsearch

Elastic Stack Beats
1

I have Filebeat on a Windows 10 machine sending log data to Elasticsearch on a Windows Server 2016. The services are running correctly. As I'd posted earlier, I'm using basic authentication.
I've created a user named filebeat_internal for this purpose.
In the filebeat.yml file:
<
output.elasticsearch:
#Array of hosts to connect to.
#hosts: ["localhost:9200"]
hosts: ["192.168.103.84:9200"]

#Optional protocol and basic auth credentials.
#protocol: "https"
username: "filebeat_internal"
password: "password"

/>

Also, on the server machine, I've logged into Elasticsearch and Kibana using the built-in user - kibana.

But my logs are not reaching Elasticsearch. Please see some Filebeat log file entries below:

2019-12-26T17:58:38.719+0530 INFO [index-management] idxmgmt/std.go:178 Set output.elasticsearch.index to 'filebeat-7.4.2' as ILM is enabled.
2019-12-26T17:58:38.719+0530 INFO elasticsearch/client.go:170 Elasticsearch url: http://192.168.103.84:9200
2019-12-26T17:58:38.719+0530 INFO [publisher] pipeline/module.go:97 Beat name: Sucharit
2019-12-26T17:58:38.721+0530 INFO instance/beat.go:422 filebeat start running.
2019-12-26T17:58:38.721+0530 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2019-12-26T17:58:38.721+0530 INFO registrar/registrar.go:145 Loading registrar data from C:\ProgramData\filebeat\registry\filebeat\data.json
2019-12-26T17:58:38.721+0530 INFO registrar/registrar.go:152 States Loaded from registrar: 7
2019-12-26T17:58:38.721+0530 INFO crawler/crawler.go:72 Loading Inputs: 1
2019-12-26T17:58:38.722+0530 INFO log/input.go:152 Configured paths: [C:\ProgramData\LogTest*]
2019-12-26T17:58:38.722+0530 INFO input/input.go:114 Starting input of type: log; ID: 15460377364020981247
2019-12-26T17:58:38.722+0530 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-12-26T17:58:38.722+0530 INFO cfgfile/reload.go:171 Config reloader started
2019-12-26T17:58:38.722+0530 INFO cfgfile/reload.go:226 Loading of config files completed.
2019-12-26T17:58:38.723+0530 INFO log/harvester.go:251 Harvester started for file: C:\ProgramData\LogTest\CW2.log
2019-12-26T17:58:41.693+0530 INFO add_cloud_metadata/add_cloud_metadata.go:87 add_cloud_metadata: hosting provider type not detected.
2019-12-26T17:58:42.694+0530 INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://192.168.103.84:9200))
2019-12-26T17:58:46.415+0530 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://192.168.103.84:9200)): Get http://192.168.103.84:9200: dial tcp 192.168.103.84:9200: connectex: No connection could be made because the target machine actively refused it.
2019-12-26T17:58:46.415+0530 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://192.168.103.84:9200)) with 1 reconnect attempt(s)

Appreciate any help in resolving this.
Thanks.

2

Thanks for posting your Filebeat log. The line that stands out is this one:

For some reason it appears that Filebeat is unable to connect to Elasticsearch. I'm not too familar with Windows tooling, but would it possible for you to try and make an HTTP request from your Filebeat machine to your Elasticsearch cluster using something like cURL (or whatever the equivalent is for Windows)? The cURL command would be:

curl -v http://192.168.103.84:9200/

Note the -v flag for verbose output. So you'd want to run that or the Windows equivalent of it from your Filebeat machine. The output might give us some information on why the connection is not successful.

Also, Googling for "No connection could be made because the target machine actively refused it" seems to suggest that there might be a firewall in place between the Filebeat machine and the Elasticsearch machine or the right ports (in this case 9200) might not be open on the Elasticsearch machine. Could either of these be the case for your setup?

3

1.) Here's the output of running the curl command on my Windows 10 client
C:\WINDOWS\system32>curl -v http://192.168.103.84:9200/

  • Trying 192.168.103.84...
  • TCP_NODELAY set
  • connect to 192.168.103.84 port 9200 failed: Connection refused
  • Failed to connect to 192.168.103.84 port 9200: Connection refused
  • Closing connection 0
    curl: (7) Failed to connect to 192.168.103.84 port 9200: Connection refused

2.) Yes, the Firewall settings are correctly in place. There are firewall rules set on the server machine to allow inbound connections to port 9200.

3.) Just pinging the IP address from my Windows 10 client works fine:
C:\WINDOWS\system32>ping 192.168.103.84

Pinging 192.168.103.84 with 32 bytes of data:
Reply from 192.168.103.84: bytes=32 time=406ms TTL=126
Reply from 192.168.103.84: bytes=32 time=318ms TTL=126
Reply from 192.168.103.84: bytes=32 time=317ms TTL=126
Reply from 192.168.103.84: bytes=32 time=316ms TTL=126

Ping statistics for 192.168.103.84:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 316ms, Maximum = 406ms, Average = 339ms

4.) So, I'm not sure why the connection is being refused. Could it be because of built in user and passwords that I've set?
On the Windows server machine, the URL
http://localhost:9200
in a browser, asks for a login. So, when I enter "elastic" and its corresponding password, the page loads.
But when I enter
http://192.168.103.84:9200
in a browser on the client, it just says connection refused.

Thanks again.

4

I've been reading in some places that Elasticsearch needs to be configured to accept remote connections. If that is the case, do I need this setting in the elasticsearch.yml file?

network.host: 0.0.0.0

5

Yes, the fact that you are able to reach the Elasticsearch API with http://localhost:9200 but not with http://192.168.103.84:9200 tells me that its probably because you are using the default configuration of Elasticsearch, which is to bind to loopback addresses only.

Try changing that setting to 192.168.103.84 and see if it helps.

1 Like
6

Yes, that solves that connection error. But now I get some more connection errors.
I'm using X-Pack security for basic authentication.

This is the error in the filebeat log file:

|2019-12-27T14:53:56.977+0530|ERROR|pipeline/output.go:100|Failed to connect to backoff(elasticsearch(http://192.168.103.84:9200)): Connection marked as failed because the onConnect callback failed: failed to create alias: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [filebeat_internal]"}],"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [filebeat_internal]"},"status":403}: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [filebeat_internal]"}],"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [filebeat_internal]"},"status":403}|

|2019-12-27T14:53:56.977+0530|INFO|pipeline/output.go:93|Attempting to reconnect to backoff(elasticsearch(http://192.168.103.84:9200)) with 5 reconnect attempt(s)|

Please let me know what additional privileges or settings are needed to fix this.
Thanks.

7

Depending on what you're trying to accomplish with Filebeat (setup, writing events, etc.) you might need to setup one or more X-Pack Security roles and then assign those roles to your filebeat_internal user. You can read more about this at https://www.elastic.co/guide/en/beats/filebeat/current/feature-roles.html.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.