Http requests as a beat

Elastic Stack Beats
1

I have a small golang application that listens to http post requests on various routes. The data received represents messages on a messaging bus. Our current solution is to write json to a file that's tail'd by filebeat.

I would like to send the json encoded messages to logstash or elasticsearch directly.

Is something like this achievable with libbeat?

Any advice for a new beats developer?

2

https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-http.html might do what you want?

3

I don't think so.

These are requests to my application. The http module is for making outbound requests.

But maybe missing something. I'm here obviously.

4

You can call your application as long as it has a HTTP API;

The HTTP module is a Metricbeat module used to call arbitrary HTTP endpoints for which a dedicated Metricbeat module is not available.

5

I'm not trying to call an API. I'm trying to listen to requests coming in to a web server.

I would like the route handlers to write directly to elasticsearch or logstash using libbeat.

The http module seems like a "curlbeat". Polling an API resource. I don't want to poll a resource.

It's a push mechanism. Not a poll.

6

Would https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-http-options.html do it then?

7

Packetbeat. Interesting thought. Didn't think about that.

Can custom modules be written for packetbeat?

8

Do you mean protocols, or modules like filebeat ones?

9

I guess so.

I'll read up on packetbeat.

10

@warkolm I'm not sure packetbeat solves what we are trying to achieve at this time. It could work (and be really cool) if we were to write a protocol or configure http protocol to parse Multipart/Mixed content type and ship the results.

I found GitHub - martinhoefling/saltbeat: Saltbeat is a forwarder for messages from the salt master event bus to logstash or elasticsearch which seems similar in concept.

Saltbeat is a forwarder for messages from the salt master event bus to logstash or elasticsearch

I'm going to check out that project.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.