String interpolation for HTTP output message format

Elastic Stack Logstash
1

Hello,

I am trying to send some data gathered with the JDBC input plugin to an HTTP endpoint. I have to make my POST body match a certain specification, and so I use message format for the HTTP output plugin.

This is the message format:

message =>  '[{"headers":{"timestamp":"%{@timestamp}","toplevel_hostgroup":"%{toplevel_hostgroup}"},"body":%{body}}]'

In the body, I have to send the whole event as a JSON, in string. To make the whole event into a string I use the ruby filter plugin:

code => "event['body'] = event.to_json"

I check the events with the stdout rubydebug codec, they look like this:

{
                 "relid" => 16479,
            "schemaname" => "public",
               "relname" => "latest_reports",
        "heap_blks_read" => 2565,
         "heap_blks_hit" => 33262,
         "idx_blks_read" => 3666,
          "idx_blks_hit" => 11562145,
       "toast_blks_read" => 0,
        "toast_blks_hit" => 0,
        "tidx_blks_read" => 0,
         "tidx_blks_hit" => 0,
              "@version" => "1",
            "@timestamp" => "2015-11-18T10:05:52.577Z",
    "toplevel_hostgroup" => "punch",
                  "body" => "{\"relid\":16479,\"schemaname\":\"public\",\"relname\":\"latest_reports\",\"heap_blks_read\":2565,\"heap_blks_hit\":33262,\"idx_blks_read\":3666,\"idx_blks_hit\":11562145,\"toast_blks_read\":0,\"toast_blks_hit\":0,\"tidx_blks_read\":0,\"tidx_blks_hit\":0,\"@version\":\"1\",\"@timestamp\":\"2015-11-18T10:05:52.577Z\",\"toplevel_hostgroup\":\"punch\"}"
}

As you can see above, the whole event is nicely written in the body field, exactly how I would need it. Unfortunately the webserver receives the POST as follows:

[{"headers":{"timestamp":"2015-11-18T10:05:52.577Z","toplevel_hostgroup":"punch"},"body":{"relid":16479,"schemaname":"public","relname":"latest_reports","heap_blks_read":2565,"heap_blks_hit":33262,"idx_blks_read":3666,"idx_blks_hit":11562145,"toast_blks_read":0,"toast_blks_hit":0,"tidx_blks_read":0,"tidx_blks_hit":0,"@version":"1","@timestamp":"2015-11-18T10:05:52.577Z","toplevel_hostgroup":"punch"}}]

What I would need, and expect is this:

[{"headers":{"timestamp":"2015-11-18T10:05:52.577Z","toplevel_hostgroup":"punch"},"body":"{\"relid\":16479,\"schemaname\":\"public\",\"relname\":\"latest_reports\",\"heap_blks_read\":2565,\"heap_blks_hit\":33262,\"idx_blks_read\":3666,\"idx_blks_hit\":11562145,\"toast_blks_read\":0,\"toast_blks_hit\":0,\"tidx_blks_read\":0,\"tidx_blks_hit\":0,\"@version\":\"1\",\"@timestamp\":\"2015-11-18T10:05:52.577Z\",\"toplevel_hostgroup\":\"punch\"}"}]

Somehow when LogStash::StringInterpolation.evaluate is called for message it doesn't keep he body field as a string. I tried to use

code => "event['body'] = event.to_json.inspect"

but it doesn't work either, the body field will have double backslashes, and the HTTP endpoint won't take it. Unfortunately I don't have control over the HTTP endpoint, so I have to solve this on my end, with Logstash. Any ideas how could I get my desired output?

Cheers,
Akos

2

Forgot to mention this is Logstash 2.0, with Ruby 1.8.7, on a RHEL6 machine.

3

I ran into this myself a while back, I never did find a solution though :frowning:

4

So is this a bug then? From the Project Principles: Community: If a newbie has a bad time, it's a bug.

I am having a bad time. :wink:

5

I think I just managed to solve it. Seems it was a user error after all. :blush: For others looking for the same problem, the correct configuration was:

ruby filter

code => "event['body'] = event.to_json.inspect"

message format

message =>  '[{"headers":{"timestamp":"%{@timestamp}","toplevel_hostgroup":"%{toplevel_hostgroup}"},"body":%{body}]'

After this I got a POST with the following body:

[{"headers":{"timestamp":"2015-11-20T10:04:19.632Z","toplevel_hostgroup":"punch"},"body":"{\"relid\":301380,\"schemaname\":\"public\",\"relname\":\"fact_values\",\"heap_blks_read\":2231575,\"heap_blks_hit\":24258876,\"idx_blks_read\":3905200,\"idx_blks_hit\":47881150,\"toast_blks_read\":37502,\"toast_blks_hit\":92782,\"tidx_blks_read\":13535,\"tidx_blks_hit\":444310,\"@version\":\"1\",\"@timestamp\":\"2015-11-20T10:04:19.632Z\",\"toplevel_hostgroup\":\"punch\"}"]