Http requests as a beat


(Matt) #1

I have a small golang application that listens to http post requests on various routes. The data received represents messages on a messaging bus. Our current solution is to write json to a file that's tail'd by filebeat.

I would like to send the json encoded messages to logstash or elasticsearch directly.

Is something like this achievable with libbeat?

Any advice for a new beats developer?


(Mark Walkom) #2

https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-http.html might do what you want?


(Matt) #3

I don't think so.

These are requests to my application. The http module is for making outbound requests.

But maybe missing something. I'm here obviously.


(Mark Walkom) #4

You can call your application as long as it has a HTTP API;

The HTTP module is a Metricbeat module used to call arbitrary HTTP endpoints for which a dedicated Metricbeat module is not available.


(Matt) #5

I'm not trying to call an API. I'm trying to listen to requests coming in to a web server.

I would like the route handlers to write directly to elasticsearch or logstash using libbeat.

The http module seems like a "curlbeat". Polling an API resource. I don't want to poll a resource.

It's a push mechanism. Not a poll.


(Mark Walkom) #6

Would https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-http-options.html do it then?


(Matt) #7

Packetbeat. Interesting thought. Didn't think about that.

Can custom modules be written for packetbeat?


(Mark Walkom) #8

Do you mean protocols, or modules like filebeat ones?


(Matt) #9

I guess so.

I'll read up on packetbeat.


(Matt) #10

@warkolm I'm not sure packetbeat solves what we are trying to achieve at this time. It could work (and be really cool) if we were to write a protocol or configure http protocol to parse Multipart/Mixed content type and ship the results.

I found https://github.com/martinhoefling/saltbeat which seems similar in concept.

Saltbeat is a forwarder for messages from the salt master event bus to logstash or elasticsearch

I'm going to check out that project.


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.