Error connection between filebeat and logstash

Elastic Stack Beats
1

Hello, everyone,
I have a problem with the communication of filebeat and elasticsearch.

I have recently configured SSL for communication between filebeat and logstash.
As from the site the certificates have been set correctly :slight_smile:
curl -v --cacert /etc/filebeat/logstash.crt https://10.164.88.7:504

  • Rebuilt URL to: https://10.164.88.7:5044/
  • Trying 10.164.88.7...
  • TCP_NODELAY set
  • Connected to 10.164.88.7 (10.164.88.7) port 5044 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/filebeat/logstash.crt
    CApath: /etc/ssl/certs
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
  • ALPN, server did not agree to a protocol
  • Server certificate:
  • subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
  • start date: May 13 09:19:34 2019 GMT
  • expire date: May 12 09:19:34 2020 GMT
  • subjectAltName: host "10.164.88.7" matched cert's IP address!
  • issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
  • SSL certificate verify ok.

GET / HTTP/1.1
Host: 10.164.88.7:5044
User-Agent: curl/7.58.0
Accept: /

  • TLSv1.2 (IN), TLS alert, Client hello (1):
  • Empty reply from server
  • Connection #0 to host 10.164.88.7 left intact
    curl: (52) Empty reply from server

But when I try to visualize the data on Kibana, the logs do not arrive
So I tried to understand the problem.

2019/05/13 10:04:31.080358 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2019/05/13 10:04:31.080397 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.16
2019/05/13 10:04:31.080497 metrics.go:23: INFO Metrics logging every 30s
2019/05/13 10:04:31.080754 logstash.go:91: INFO Max Retries set to: 3
2019/05/13 10:04:31.080810 outputs.go:108: INFO Activated logstash as output plugin.
2019/05/13 10:04:31.080893 publish.go:300: INFO Publisher name: client8818
2019/05/13 10:04:31.081083 async.go:63: INFO Flush Interval set to: 1s
2019/05/13 10:04:31.081097 async.go:64: INFO Max Bulk Size set to: 2048
2019/05/13 10:04:31.081346 beat.go:233: INFO filebeat start running.
2019/05/13 10:04:31.081388 registrar.go:85: INFO Registry file set to: /var/lib/filebeat/registry
2019/05/13 10:04:31.081421 registrar.go:106: INFO Loading registrar data from /var/lib/filebeat/registry
2019/05/13 10:04:31.081796 registrar.go:123: INFO States Loaded from registrar: 10
2019/05/13 10:04:31.081837 crawler.go:38: INFO Loading Prospectors: 1
2019/05/13 10:04:31.081927 registrar.go:236: INFO Starting Registrar
2019/05/13 10:04:31.081936 sync.go:41: INFO Start sending events to output
2019/05/13 10:04:31.081978 prospector_log.go:65: INFO Prospector with previous states loaded: 10
2019/05/13 10:04:31.082073 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2019/05/13 10:04:31.082104 prospector.go:124: INFO Starting prospector of type: log; id: 17005676086519951868
2019/05/13 10:04:31.082127 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2019/05/13 10:04:51.083662 log.go:91: INFO Harvester started for file: /var/log/auth.log
2019/05/13 10:05:01.080934 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=1425 libbeat.logstash.publish.write_bytes=746 libbeat.logstash.published_and_acked_events=3 libbeat.publisher.published_events=3 publish.events=14 registrar.states.current=10 registrar.states.update=14 registrar.writes=2

I don't understand why, I have activated both on SSL logstash and on the Filebeat configuration file.

Filebeat.yml:

</>###################### Filebeat Configuration Example #########################

This file is an example configuration file highlighting only the most common

options. The filebeat.full.yml file from the same directory contains all the

supported options with more comments. You can use it as a reference.

You can find the full configuration reference here:

Filebeat Reference | Elastic

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • input_type: log

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /var/log/*.log
      #- c:\programdata\elasticsearch\logs*

#================================ Outputs =====================================

Configure what outputs to use when sending the data collected by the beat.

Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:

Array of hosts to connect to.

#hosts: ["localhost:9200"]

Optional protocol and basic auth credentials.

#protocol: "https"
#username: "elastic"
#password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["10.164.88.7:5044"]
ssl.certificate_authorities: ["/etc/filebeat/logstash.crt"]

ssl.certificate_authorities: ["/etc/ca.crt"]

ssl.certificate: "/etc/beat.crt"

ssl.key: "/etc/beat.key"

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/ca.crt"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/beat.crt"

Client Certificate Key

#ssl.key: "/etc/beat.key"

#================================ Logging =====================================

Sets log level. The default log level is info.

Available log levels are: critical, error, warning, info, debug

#logging.level: debug

At debug level, you can selectively enable logging only for some components.

To enable all selectors use ["*"]. Examples of other selectors are "beat",

"publish", "service".

#logging.selectors: ["*"]

02-beats-input.conf:
</> input {
beats {
port => 5044
codec => "json_lines"
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
client_inactivity_timeout => "86400"
}
}

Could you help me? Thank you.

2

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image.jpg764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

I'm moving your post to #beats:filebeat

3

Please properly format your post, it's really hard to follow.

Filebeat did actually send events. And 3 have been ACKed by Logstash.

From logs it looks like you are using Filebeat 1.x or 5.x. Both are End of Life. I'd recomment to switch to Filebeat 7 or 6.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.