Allow logstash-output-http to send binary data

Elastic Stack Elasticsearch
1

Hello there,

I'm trying to process binary data through Logstash to upload it to an HTTP endpoint (via PUT). My input data contains some fields I want to index into ElasticSearch and a field containing binary data encoded in base64, togehther with a file name. My goal is to submit the base64 encoded data to ElasticSearch (so the attachment ingest can index metadata/containing text) and to upload a decoded version to a HTTP endpoint, so I can retrieve the whole file later.

What I did now is to write a Logstash plugin which expects three options: Source field, target field and operation (encode/decode). I use it to decode the base64 from the source field to a new field, with a forced "BINARY" encoding.

My plan was to use the output-http plugin to push the files onto a WebDAV share. For this, I have configured it like this:

output {
    http {
        url => "http://webdav/%{filename}"
        http_method => "put"
        content_type => "application/octet-stream"
        format => "message"
        message => "%{data_decoded}"
    }
}

This works as long as the decoded data is actually plaintext but fails for arbitrary binary data. My temporary solution is to patch the http output to use event.get(@message) instead of event.sprintf and to specify a field instead of a format string in the message parameter.

My question is: Would it be possible to get this upstreamed somehow? I see you can't change message's behavior just like that, but maybe introducing a new format or another message parameter would work out. Alternatively, do you have any other ideas how I could improve the import process?

Thank you very much in advance!