Filebeats can not connect to ElasticSearch

Elastic Stack Beats
1

In version 8.5 of Elastic Search and Filebeats
I can not ship the logs from Filebeats to Elasticsearch

Here is the error log

{"log.level":"info","@timestamp":"2022-11-10T04:23:19.993Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":106},"message":"add_cloud_metadata: hosting provider type detected as aws, metadata={\"cloud\":{\"account\":{\"id\":\"281771844648\"},\"availability_zone\":\"ap-southeast-1a\",\"image\":{\"id\":\"ami-07651f0c4c315a529\"},\"instance\":{\"id\":\"i-0d68d0d1348edcc56\"},\"machine\":{\"type\":\"t2.medium\"},\"provider\":\"aws\",\"region\":\"ap-southeast-1\",\"service\":{\"name\":\"EC2\"}}}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T04:23:19.994Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":235},"message":"error connecting to Elasticsearch at http://18.139.158.62:9200: Get \"http://18.139.158.62:9200\": EOF","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T04:23:19.994Z","log.origin":{"file.name":"instance/beat.go","file.line":1056},"message":"Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at http://18.139.158.62:9200: Get \"http://18.139.158.62:9200\": EOF]","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at http://18.139.158.62:9200: Get "http://18.139.158.62:9200": EOF]

Here is the config of Filebeats output

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["18.139.158.62:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "passmeifyoucan(example not real pass)"

There is no big changes on Elasticsearch config, I just changed the network hosts to 0.0.0.0.

2

From the filebeat server run this

curl -v -u elastic http://18.139.158.62:9200

3

here is the output for curl

root@ip-10-10-10-7:/home/ubuntu# curl -v -u elastic http://18.139.158.62:9200
Enter host password for user 'elastic':
*   Trying 18.139.158.62:9200...
* Connected to 18.139.158.62 (18.139.158.62) port 9200 (#0)
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Host: 18.139.158.62:9200
> Authorization: Basic ZWxhc3RpYzpadUhpM2IxZ2c0V3hJPXRYS2htWg==
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server
root@ip-10-10-10-7:/home/ubuntu#
4

This is ss -nltp from elastic side

root@ip-10-10-10-26:/home/ubuntu# ss -nltp
State         Recv-Q        Send-Q               Local Address:Port                Peer Address:Port        Process                                           
LISTEN        0             4096                 127.0.0.53%lo:53                       0.0.0.0:*            users:(("systemd-resolve",pid=436,fd=14))        
LISTEN        0             128                        0.0.0.0:22                       0.0.0.0:*            users:(("sshd",pid=697,fd=3))                    
LISTEN        0             511                        0.0.0.0:5601                     0.0.0.0:*            users:(("node",pid=1451,fd=21))                  
LISTEN        0             4096                             *:9300                           *:*            users:(("java",pid=779,fd=406))                  
LISTEN        0             128                           [::]:22                          [::]:*            users:(("sshd",pid=697,fd=4))                    
LISTEN        0             4096                             *:9200                           *:*            users:(("java",pid=779,fd=415))                  
root@ip-10-10-10-26:/home/ubuntu#
5

Try https

curl -v -k -u elastic https://18.139.158.62:9200

6

here is the output

root@ip-10-10-10-7:/home/ubuntu# curl -v -k -u elastic https://18.139.158.62:9200

Enter host password for user 'elastic':
*   Trying 18.139.158.62:9200...
* Connected to 18.139.158.62 (18.139.158.62) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=ip-10-10-10-26
*  start date: Nov  8 08:19:23 2022 GMT
*  expire date: Nov  7 08:19:23 2024 GMT
*  issuer: CN=Elasticsearch security auto-configuration HTTP CA
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Server auth using Basic with user 'elastic'
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: 18.139.158.62:9200
> Authorization: Basic ZWxhc3RpYzpadUhpM2IxZ2c0V3hJPXRYS2htWg==
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 537
< 
{
  "name" : "ip-10-10-10-26",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "b5U3qoGCQAys9Bl7aM2_Aw",
  "version" : {
    "number" : "8.5.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "c94b4700cda13820dad5aa74fae6db185ca5c304",
    "build_date" : "2022-10-24T16:54:16.433628434Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host 18.139.158.62 left intact
root@ip-10-10-10-7:/home/ubuntu#
7

So that tells us your elasticsearch has https enabled.

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["18.139.158.62:9200"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"
  ssl.verifcation_mode: none

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "passmeifyoucan(example
8

@Zay_Lin_Htun please be more careful with where you put your topics, Filebeat issues do not belong in #elastic-stack:kibana for eg.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.