Auditbeat vs testing ES output

Elastic Stack Beats
1

trying to test auditbeat connectivity to elastic, connectivity seems fine only service is unavailable.
Wondering where I possible could see the RC, tried ingesting nodes' elasticsearch + audit logs, beat log, but nothing seems logged regarding this and tcp/ssldump is no good as I'm using https.

Appreciate any hints, TIA!

# /usr/share/auditbeat/bin/auditbeat test output -c /etc/auditbeat/auditbeat.yml 
elasticsearch: https://<redacted>...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: <redacted>
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR Get https://<redacted>: Service Unavailable
2

Wondering if would be an auditing issue... though I'm confident I'm using properly credentials, but maybe I should turn on auditing event logging, as I read from doc:

xpack.security.audit.enabled

Set to true to enable auditing on the node. The default value is false . This puts the auditing events in a dedicated file named <clustername>_audit.json on each node.

3

Increased beat's log level from warning to debug and just gets this:

2020-02-28T11:07:42.651+0100	DEBUG	[keystore]	keystore/keystore.go:125	accessing key 'INGEST_PROTOCOL' from the keystore
2020-02-28T11:07:42.651+0100	DEBUG	[keystore]	keystore/keystore.go:125	accessing key 'INGEST_USER' from the keystore
2020-02-28T11:07:42.651+0100	DEBUG	[keystore]	keystore/keystore.go:125	accessing key 'INGEST_PWD' from the keystore
2020-02-28T11:07:42.652+0100	DEBUG	[keystore]	keystore/keystore.go:125	accessing key 'INGEST_URL' from the keystore
2020-02-28T11:07:42.652+0100	INFO	elasticsearch/client.go:174	Elasticsearch url: https://<redacted>
elasticsearch: https://<redacted>...

  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 62.243.41.249
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
2020-02-28T11:07:55.188+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:733	ES Ping(url=https://<redacted>)
2020-02-28T11:07:55.196+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:737	Ping request failed with: Get https://<redacted>: Service Unavailable
  talk to server... ERROR Get https://<redacted>: Service Unavailable
4

Anyone?

'Service unavailable' wouldn't that possible imply some kind of permission issue on elastic side, and if how to get details logged somehow?

I'm puzzled by this, user attempted with has a role which got create_index,index,write on auditbeat-* same role which work for another user working for Windows auditbeats.

5

Hey @stefws,

"Service unavailable" seems like an error on the server side, it doesn't look like a permissions issue.

Could you check the Elasticsearch logs?

6

Thanks, already tried ingesting nodes' elasticsearch + audit logs, but nothing in these, wondering if I should enable auditing logs as I think it mighty be a permission issue for the user...

7

:thinking: I would expect a different error if the user wouldn't have enough privileges.

Do you have any proxy between auditbeat and elasticsearch?

8

Yes a HAproxy, but got +350 Windows using the same LB VIP through same HAproxy->ES ingest nodes without any issue.

9

Can you see anything in the haproxy access logs? Maybe it is the one generating this "Service unavailable" errors?

10

No nothing in HAproxy log, also got +350 Windows Computer running both Winlog+Audit beats through same HAproxy just fine, so it's seems weird...

11

Do you mean that normal ingestion works, but auditbeat test output fails?

12

Normal ingestion from other [windows] clients works through the same HAproxy, now I'm trying from a Linux client which fails.

13

Are both Windows and Linux machines using the same Beats versions and output configs?

14

Same beat versions 7.6.0 and same output config except with different credentials (as I have forgot the password using in Windows keystores).
Keystores doesn't seem compatible among Wintel/Lintel clients :slight_smile: So I created another User with same roles as the working Wintel clients' User and entered this in a Lintel keystore. If I hardwire known wrong credentials in yml' output section, I get the same error, so I think it's some kind on authentication error from Elastic.

Also tried with known 'Super User' credentials hardwired into output, but gives the same error with Service Unavailable.

15

Found the Wintel credential, but even that isn't wokring on my Lintel client.

If I enter bad password in Wintel I get this error:

talk to server... ERROR 401 Unauthorized:

But not on my Lintel, hm wondering why, netstat shows an established tcp socket to expected destination as also indicated by test output, so what may hinder the 'ping' test to fail with 'Service Unavailable' if it's not a authentication failure...

16

:wink: but running auditbeat it sends data just fine, so is it only the test output that fails...

17

This 401 error is what I would expect if the problem were in the credentials used. Service Unavailable looks like a network or server-side problem, in Elasticsearch, or in Haproxy.

Would it be possible to try to do the output test directly from auditbeat to Elasticsearch? So we can discard some problem with the connectivity through haproxy.

18

This also points to some problem in the connectivity from auditbeat to Elasticsearch through haproxy.

19

No no, if I actually just starts the Auditbeat service on my Lintel box then it actually ships data just fine to Elastic through HAproxy like my Wintel clients do, even that 'test output' says 'Service Unavailable', very weird. What does test output ping actually do compared to sending data that might be different?

20

Oh ok, that's interesting, because actually any connection does the same initial ping test, also when connecting to ship data.

There might be some difference in the way Beats sets up the connection when testing, but this is going to require further investigation.

Could you share your output configuration?

In any case it'd be good if you could make the test with the same configuration but directly to Elasticsearch, without Haproxy.