connecting filebeat with stream API

Elastic Stack Beats
1

Dear filebeat community,

I am new to elastic and filebeat and need some help.

I am trying to make filebeat to get log events from a hashicorp Nomad server stream api (Events - HTTP API | Nomad | HashiCorp Developer) the idea behind is I can hook to this stream api and the Nomad server will send my http client events in json format.

I can replicate this functionality using curl like curl -s -N --cert /home/user/client.crt --key /home/user/client.key --cacert /home/user/ca.pem --header "X-Nomad-Token: --REDACTED-- https://myserver.local:4646/v1/event/stream?namespace=*

I am now trying to replicate filebeat so it can receive notifications from this streaming api and print them into stdout.

So far I have not been successful. I tried the http input with the configuration below:

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: myserver.local
  listen_port: 4646
  url: "/v1/event/stream?namespace=*"
  secret.header: "X-Nomad-Token"
  secret.value: "--REDACTED--"
  ssl.enabled: true
  ssl.certificate: "/home/user/client.crt"
  ssl.key: "/home/user/client.key"
  ssl.certificate_authority: "/home/user/ca.pem"
  ssl.verification_mode: "full"

output.console:
  pretty: true  # Optional, makes the output more readable

and then run filebeat like:

sudo filebeat -e -c ~/filebeat_2.config

filebeat starts and I get the following messages:

{"log.level":"info","@timestamp":"2024-11-14T00:05:38.048+0100","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start","file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-14T00:05:38.048+0100","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).startInput","file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.listen_address filebeat.inputs.0.listen_port filebeat.inputs.0.secret.header filebeat.inputs.0.secret.value filebeat.inputs.0.ssl.certificate filebeat.inputs.0.ssl.certificate_authority filebeat.inputs.0.ssl.enabled filebeat.inputs.0.ssl.key filebeat.inputs.0.ssl.verification_mode filebeat.inputs.0.type filebeat.inputs.0.url]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-14T00:05:38.057+0100","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).startInput","file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 16785407196084471138)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-14T00:05:38.057+0100","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start","file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-14T00:05:38.057+0100","log.logger":"input.http_endpoint","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1","file.name":"compat/compat.go","file.line":127},"message":"Input 'http_endpoint' starting","service.name":"filebeat","id":"E8F1BEE974E68562","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-14T00:05:38.057+0100","log.logger":"metric_registry","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry","file.name":"inputmon/input.go","file.line":63},"message":"registering","service.name":"filebeat","input_type":"http_endpoint","id":"E8F1BEE974E68562","key":"E8F1BEE974E68562","uuid":"cc216ea1-c1b2-4a0d-895a-30b9e2168784","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-14T00:05:38.057+0100","log.logger":"input.http_endpoint","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/http_endpoint.(*pool).serve","file.name":"http_endpoint/input.go","file.line":212},"message":"Starting HTTPS server on myserver.local:4646 with /v1/event/stream?namespace=* end point","service.name":"filebeat","id":"E8F1BEE974E68562","address":"myserver.local:4646","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-14T00:05:38.098+0100","log.logger":"metric_registry","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry.func1","file.name":"inputmon/input.go","file.line":70},"message":"unregistering","service.name":"filebeat","input_type":"http_endpoint","id":"E8F1BEE974E68562","key":"E8F1BEE974E68562","uuid":"cc216ea1-c1b2-4a0d-895a-30b9e2168784","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-11-14T00:05:38.098+0100","log.logger":"input.http_endpoint","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1","file.name":"compat/compat.go","file.line":140},"message":"Input 'http_endpoint' failed with: unable to start server due to error: listen tcp 10.0.0.101:4646: bind: cannot assign requested address","service.name":"filebeat","id":"E8F1BEE974E68562","ecs.version":"1.6.0"}

I would like to ask for help and if there is a workaround to solve this issue and be able to receive notifications

thank you

2

Hello and welcome,

I think there is a little confusion here.

It seems that you want to get logs from Nomad by connecting to an API endpoint, right?

In the example you shared you use curl to talk to an API endpoint.

The http_endpoint input is used to make fileabeat listen for incoming requests, not to get data from other places.

As the documentation says:

This input can for example be used to receive incoming webhooks from a third-party application or service.

From what I understand you want the opposite, so you need to use the httpjson input.

But I'm not sure this will work also as the httpjson input makes requests in a specific interval and it seems that the endpoint you want to use will send the data as a stream.