Hello there,
I'm trying to process binary data through Logstash to upload it to an HTTP endpoint (via PUT). My input data contains some fields I want to index into ElasticSearch and a field containing binary data encoded in base64, togehther with a file name. My goal is to submit the base64 encoded data to ElasticSearch (so the attachment ingest can index metadata/containing text) and to upload a decoded version to a HTTP endpoint, so I can retrieve the whole file later.
What I did now is to write a Logstash plugin which expects three options: Source field, target field and operation (encode/decode). I use it to decode the base64 from the source field to a new field, with a forced "BINARY" encoding.
My plan was to use the output-http plugin to push the files onto a WebDAV share. For this, I have configured it like this:
output {
http {
url => "http://webdav/%{filename}"
http_method => "put"
content_type => "application/octet-stream"
format => "message"
message => "%{data_decoded}"
}
}
This works as long as the decoded data is actually plaintext but fails for arbitrary binary data. My temporary solution is to patch the http output to use event.get(@message) instead of event.sprintf and to specify a field instead of a format string in the message parameter.
My question is: Would it be possible to get this upstreamed somehow? I see you can't change message's behavior just like that, but maybe introducing a new format or another message parameter would work out. Alternatively, do you have any other ideas how I could improve the import process?
Thank you very much in advance!