Filebeat vs elastic putput

stefws · March 12, 2023, 5:13pm

Hi

Need a wall to play against here TIA

What am I missing out on, I wonder, when trying to launch filebeat.service. I'm see these errors:

==> /var/log/filebeat/filebeat-20230312.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:23.444+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}
[root@exrhel0311 filebeat]$ tail /var/log/filebeat/filebeat-20230312*
==> /var/log/filebeat/filebeat-20230312-1.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:23.814+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-2.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:24.064+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-3.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:24.321+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-4.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:24.567+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-5.ndjson <==
{"log.level":"warn","@timestamp":"2023-03-12T17:57:11.526+0100","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-12T17:57:11.526+0100","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-12T17:57:11.528+0100","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-12T17:57:11.537+0100","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-6.ndjson <==
{"log.level":"warn","@timestamp":"2023-03-12T17:58:15.342+0100","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-12T17:58:15.342+0100","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}

wondering as testing seems to be just fine:

[root@exrhel0311 filebeat]$ filebeat test output
elasticsearch: https://<redacted>:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.83.67.157
    dial up... OK
  TLS...
    security... WARN server's certificate chain verification is disabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 8.6.1
[root@exrhel0311 filebeat]$ filebeat test config
Config OK

stephenb · March 12, 2023, 5:50pm

Hi @stefws

Please share your filebeat.yml

Did you iterate on that?

Does it actually work now when you run the service?

What is the output of

journalct -u filebeat -f

stefws · March 12, 2023, 5:54pm

Yeap sorry

Output section says:

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["${ESURL}"]

  # Protocol - either `http` (default) or `https`.
  protocol: "${ESPROTOCOL}"
  ssl:
    #certificate_authorities: ["elasticsearch-ca.pem"]
    #verification_mode: "certificate"
    verification_mode: "none"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "${ESUSER}"
  password: "${ESPWD}"

Keyastore has these entries:

[root@exrhel0311 ~]$ pjp_filebeat keystore list
ESPROTOCOL
ESUSER
JBOSS_USER
JBOSS_PWD
AMQ_PWD
AMQ_USER
CAMEL_PWD
CAMEL_USER
ESPWD
ESURL

Problem is the service doesn't run:

[root@exrhel0311 ~]$ journalctl -u filebeat -f
-- Logs begin at Fri 2023-03-03 13:06:46 CET. --
Mar 12 17:55:24 exrhel0311 filebeat[593]: Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')
Mar 12 17:55:24 exrhel0311 systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE
Mar 12 17:55:24 exrhel0311 systemd[1]: Unit filebeat.service entered failed state.
Mar 12 17:55:24 exrhel0311 systemd[1]: filebeat.service failed.
Mar 12 17:55:24 exrhel0311 systemd[1]: filebeat.service holdoff time over, scheduling restart.
Mar 12 17:55:24 exrhel0311 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Mar 12 17:55:24 exrhel0311 systemd[1]: start request repeated too quickly for filebeat.service
Mar 12 17:55:24 exrhel0311 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
Mar 12 17:55:24 exrhel0311 systemd[1]: Unit filebeat.service entered failed state.
Mar 12 17:55:24 exrhel0311 systemd[1]: filebeat.service failed.

stephenb · March 12, 2023, 5:59pm

Thanks I know you can't share the actual values, but just the field names doesn't help a whole lot... It's the values that matter

What is the value in protocol??

Did you put the protocol leading on the ESURL as well?

Do they conflict?

If you put HTTPS in the ESURL, then you don't need the protocol line.

Please show the journalctl output That is where the real logs are at.

stefws · March 12, 2023, 6:02pm

I'm successfully using the same keystore and output section in multiple metricbeat instances for months by now, so I just grabbed the keystore and output configuration into filebeat.

No I'm not specifying the protocol on the URL, and also test output should be using the filebeat.yml:output.elasticsearch section ImHO.

stephenb · March 12, 2023, 6:06pm

Hi @stefws

Then it should be working shouldn't it?

You ask for help.. I'm trying to help

As I'm sure you know, when you run into these things, it should obviously work. It sometimes takes two eyes and asking basic questions.

Can you show the journal control logs?

You shouldn't move the keystore between different beats... Not sure if you mean you literally moved the keystore... Or you recreated it?

stefws · March 12, 2023, 6:07pm

Just grabbed a copy from metricbeat keystore, will try to recreated it...

Also appreciate you playing my wall here

stephenb · March 12, 2023, 6:10pm

Yeah no worries I do it all the time!

One thing you can do is work backwards hardcode all the values. Make sure it works then put them in the newly created keystore

stefws · March 12, 2023, 6:27pm

Hardcoding works, recreating a new keystore still fails

stephenb · March 12, 2023, 6:41pm

Ok progress....

Is the Keystore in the right place... I think it moved between 7.x and 8.x

Filebeat creates the keystore in the directory defined by the path.data configuration setting.

How did you create it?

subtle difference

filebeat keystore create <!--- Correct

./filebeat keystore create <-- InCorrect

So you reconfigured the logging or something because journalctl -u filebeat -f should show the detailed filebeat logs .. it does not, so you must have made some changes to the filebeat log config.

So I would set logging.level: debug in the filebeat.yml and then tail filebeat logs wherever you put them and look for a detailed explanation.

What is the latest error in the filebeat logs?

This kinda looks like it can't find the keystore ... the debug logs will help

error initializing publisher: missing field accessing 'output.elasticsearch.protocol'

stefws · March 12, 2023, 9:47pm

You hit the nail thanks, I've missed my relocation of path.data in the service definition file, thus filebeat service couldn't find the keystore (a copy from metricbeat keystore works fine then

system · April 9, 2023, 11:48pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.