# Filebeat vs elastic putput

**URL:** https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506
**Category:** Beats
**Tags:** filebeat
**Created:** [March 12, 2023, 5:13pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506 "2023-03-12T17:13:25Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)
#### Post date: [March 12, 2023, 5:13pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/1 "2023-03-12T17:13:25Z")

</div>

Hi

Need a wall to play against here TIA 🙂

What am I missing out on, I wonder, when trying to launch filebeat.service. I'm see these errors:

```auto
==> /var/log/filebeat/filebeat-20230312.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:23.444+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}
[root@exrhel0311 filebeat]$ tail /var/log/filebeat/filebeat-20230312*
==> /var/log/filebeat/filebeat-20230312-1.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:23.814+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-2.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:24.064+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-3.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:24.321+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-4.ndjson <==
{"log.level":"error","@timestamp":"2023-03-12T17:55:24.567+0100","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-5.ndjson <==
{"log.level":"warn","@timestamp":"2023-03-12T17:57:11.526+0100","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-12T17:57:11.526+0100","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-12T17:57:11.528+0100","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-12T17:57:11.537+0100","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}

==> /var/log/filebeat/filebeat-20230312-6.ndjson <==
{"log.level":"warn","@timestamp":"2023-03-12T17:58:15.342+0100","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-12T17:58:15.342+0100","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}

```

wondering as testing seems to be just fine:

```auto
[root@exrhel0311 filebeat]$ filebeat test output
elasticsearch: https://<redacted>:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.83.67.157
    dial up... OK
  TLS...
    security... WARN server's certificate chain verification is disabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 8.6.1
[root@exrhel0311 filebeat]$ filebeat test config
Config OK

```

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [March 12, 2023, 5:50pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/2 "2023-03-12T17:50:12Z")

</div>

Hi @stefws

Please share your filebeat.yml

Did you iterate on that?

Does it actually work now when you run the service?

What is the output of

`journalct -u filebeat -f`

---

<div class="post-metadata">

### Author: ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)
#### Post date: [March 12, 2023, 5:54pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/3 "2023-03-12T17:54:30Z")

</div>

Yeap sorry 🙂

Output section says:

```auto
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["${ESURL}"]

  # Protocol - either `http` (default) or `https`.
  protocol: "${ESPROTOCOL}"
  ssl:
    #certificate_authorities: ["elasticsearch-ca.pem"]
    #verification_mode: "certificate"
    verification_mode: "none"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "${ESUSER}"
  password: "${ESPWD}"

```

Keyastore has these entries:

```auto
[root@exrhel0311 ~]$ pjp_filebeat keystore list
ESPROTOCOL
ESUSER
JBOSS_USER
JBOSS_PWD
AMQ_PWD
AMQ_USER
CAMEL_PWD
CAMEL_USER
ESPWD
ESURL

```

Problem is the service doesn't run:

```auto
[root@exrhel0311 ~]$ journalctl -u filebeat -f
-- Logs begin at Fri 2023-03-03 13:06:46 CET. --
Mar 12 17:55:24 exrhel0311 filebeat[593]: Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.protocol' (source:'/etc/filebeat/filebeat.yml')
Mar 12 17:55:24 exrhel0311 systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE
Mar 12 17:55:24 exrhel0311 systemd[1]: Unit filebeat.service entered failed state.
Mar 12 17:55:24 exrhel0311 systemd[1]: filebeat.service failed.
Mar 12 17:55:24 exrhel0311 systemd[1]: filebeat.service holdoff time over, scheduling restart.
Mar 12 17:55:24 exrhel0311 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Mar 12 17:55:24 exrhel0311 systemd[1]: start request repeated too quickly for filebeat.service
Mar 12 17:55:24 exrhel0311 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
Mar 12 17:55:24 exrhel0311 systemd[1]: Unit filebeat.service entered failed state.
Mar 12 17:55:24 exrhel0311 systemd[1]: filebeat.service failed.

```

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [March 12, 2023, 5:59pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/4 "2023-03-12T17:59:31Z")

</div>

Thanks I know you can't share the actual values, but just the field names doesn't help a whole lot... It's the values that matter 🙂

What is the value in protocol??

Did you put the protocol leading on the ESURL as well?

Do they conflict?

If you put HTTPS in the ESURL, then you don't need the protocol line.

Please show the journalctl output That is where the real logs are at.

---

<div class="post-metadata">

### Author: ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)
#### Post date: [March 12, 2023, 6:02pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/5 "2023-03-12T18:02:30Z")

</div>

I'm successfully using the same keystore and output section in multiple metricbeat instances for months by now, so I just grabbed the keystore and output configuration into filebeat.

No I'm not specifying the protocol on the URL, and also test output should be using the filebeat.yml:output.elasticsearch section ImHO.

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [March 12, 2023, 6:06pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/6 "2023-03-12T18:06:51Z")

</div>

Hi @stefws

Then it should be working shouldn't it? 🙂

You ask for help.. I'm trying to help 🙂

As I'm sure you know, when you run into these things, it should obviously work. It sometimes takes two eyes and asking basic questions.

Can you show the journal control logs?

> [@stefws](#):
>
> so I just grabbed the keystore and output configuration into filebeat.

You shouldn't move the keystore between different beats... Not sure if you mean you literally moved the keystore... Or you recreated it?

---

<div class="post-metadata">

### Author: ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)
#### Post date: [March 12, 2023, 6:07pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/7 "2023-03-12T18:07:55Z")

</div>

Just grabbed a copy from metricbeat keystore, will try to recreated it...

Also appreciate you playing my wall here 😉

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [March 12, 2023, 6:10pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/8 "2023-03-12T18:10:15Z")

</div>

Yeah no worries I do it all the time!

One thing you can do is work backwards hardcode all the values. Make sure it works then put them in the newly created keystore

---

<div class="post-metadata">

### Author: ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)
#### Post date: [March 12, 2023, 6:27pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/9 "2023-03-12T18:27:59Z")

</div>

Hardcoding works, recreating a new keystore still fails 😕

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [March 12, 2023, 6:41pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/10 "2023-03-12T18:41:09Z")

</div>

Ok progress....

Is the Keystore in the right place... I think it moved between 7.x and 8.x

> Filebeat creates the keystore in the directory defined by the path.data configuration setting.

How did you create it?

subtle difference

```auto
filebeat keystore create <!--- Correct

./filebeat keystore create <-- InCorrect

```

So you reconfigured the logging or something because `journalctl -u filebeat -f` should show the detailed filebeat logs .. it does not, so you must have made some changes to the filebeat log config.

So I would set `logging.level: debug` in the filebeat.yml and then tail filebeat logs wherever you put them and look for a detailed explanation.

What is the latest error in the filebeat logs?

This kinda looks like it can't find the keystore ... the debug logs will help

`error initializing publisher: missing field accessing 'output.elasticsearch.protocol'`

---

<div class="post-metadata">

### Author: ![stefws](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefws/32/6442_2.png) [@stefws](https://discuss.elastic.co/u/stefws)
#### Post date: [March 12, 2023, 9:47pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/11 "2023-03-12T21:47:10Z")

</div>

You hit the nail thanks, I've missed my relocation of path.data in the service definition file, thus filebeat service couldn't find the keystore (a copy from metricbeat keystore works fine then 😉

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 9, 2023, 11:48pm UTC](https://discuss.elastic.co/t/filebeat-vs-elastic-putput/327506/12 "2023-04-09T23:48:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.