Logstash-filter-http Post Body error

[zurb3]

I'm trying to get some kind of data enrichment using an API call to Abuse.ch as documents come into the pipeline. The logstash-filter-http plugin seems like the right answer, however it isn't working for what I need it to do. I've tried reinstalling the plugin and it still seems to be giving me the same kind of error. Any help is appreciated!

Fresh install:

Debug pipeline:
sudo bin/logstash --debug -e ' input { stdin{} } filter { http { url => "https://mb-api.abuse.ch/api/v1/" verb => "POST" body => {"query" => "get_info" "hash" => "7de2c1bf58bce09eecc70476747d88a26163c3d6bb1d85235c24a558d1f16754"} }} output { stdout { codec => rubydebug }}'
I intend to have it sprintf part of the document into the hash field of the request as well.

The error:

A wget example from the API docs:
wget --post-data "query=get_info&hash=7de2c1bf58bce09eecc70476747d88a26163c3d6bb1d85235c24a558d1f16754" https://mb-api.abuse.ch/api/v1/

I'm not sure why this is happening as I assume the syntax is similar to the older logstash-filter-rest. This brings me to another question, are there any up to date examples for this plugin? The documentation here is pretty sparse for examples. The github where the plugin is hosted has an out of date readme without relevant examples.

Are there any alternatives if this plugin won't work?

[Badger]

Please do not post pictures of text. Some people cannot read them, and they cannot be searched.

[zurb3]

Sure thing, my mistake. The install image isn't really important.
The error screenshot just says this:
:client_error => "Undefined method encoding for #Hash:

[Badger]

You are hitting this issue. Change the body_format to json, or use a string instead of a hash.

[zurb3]

Thanks! I'll give it a try real quick.

[zurb3]

Adding the body_format fixed the initial issue but now there's something weirder.
When it sends the request the abuse.ch API endpoint says its missing a query even though it is in the json hash. I've run the wget example from the same host and it returns the data. What about the http filter plugin is different?

The debug log is here: Logstash Log - Pastebin.com

[Badger]

It is not clear to me that the Malware Bazaar API is willing to accept the form_data as JSON in the body of a POST. Try putting it in the URL, as you do with wget.

[zurb3]

I tried that as well, no different.
Command: sudo bin/logstash --debug -e 'input { stdin{} } filter { http { url => "https://mb-api.abuse.ch/api/v1/" verb => "POST" query => {"query" => "get_info" "hash" => "7de2c1bf58bce09eecc70476747d88a26163c3d6bb1d85235c24a558d1f16754"} }} output { stdout { codec => rubydebug }}'
Log here: Logstash Log 2 - Pastebin.com

According to the wget docs --post-data transmits it in the body of the request.

I'm going to try and manually format it with body_format => string.

[zurb3]

Using string body didn't work either.
Command: sudo bin/logstash --debug -e 'input { stdin{} } filter { http { url => "https://mb-api.abuse.ch/api/v1/" verb => "POST" body => "query=get_info&hash=7de2c1bf58bce09eecc70476747d88a26163c3d6bb1d85235c24a558d1f16754" body_format => "text" headers => {"content-type" => "application/x-www-form-urlencoded"}}} output { stdout { codec => rubydebug }}'
Debug log: Logstash Log 3 - Pastebin.com

I've also noticed that it's overwriting my content-type header for some reason. I'm not sure if that would cause the API endpoint to reject it or not. Strange.

[zurb3]

Got it!
The Content-Type is very important. The plugin is case-sensitive for headers, I needed to use this command: sudo bin/logstash --debug -e 'input { stdin{} } filter { http { url => "https://mb-api.abuse.ch/api/v1/" verb => "POST" body => "query=get_info&hash=7de2c1bf58bce09eecc70476747d88a26163c3d6bb1d85235c24a558d1f16754" headers => {"Content-Type" => "application/x-www-form-urlencoded"}}} output { stdout { codec => rubydebug }}'

body_format doesn't matter if the type is string, it works with and without it.

Thanks Badger!