Http filter plugin authorization as parameter values

Elastic Stack Logstash
1

Hello,

I try to pass user and password to http filter plugin as parameter values like.

user => "%{user_value}"
password => "%{password_value}"

both values come from an input plugin as tag values.

However, I get httprequest failure error.

When I substitute parameter values with actual values it works. Also, when header with authorization used with param value it works.

Any idea what is wrong here?

2

If you use this syntax, it means it will search for the fields user_value and password_value in the event itself.

Tags are usually a list of strings, not an hash.
Would it be possible to share the input section and the http filter section?

In addition, the best way to access would be using %{[user_value]} and %{[password_value]}

In case those credentials are fixed/static, I suggest to use:

3

@Luca_Belluccini,

Thanks. Tried as array, it did not help.

For the input, I use exec plugin and execute a linux command. I see output from the input plugin as
"message"=>"xxx"
"user_value"=>"xxx",
"password_value"=>"xxx",

Those values are correct and when copy/pasted into http filter plugin it works. Here is the http plugin
http {
url => "xxx"
user => "%{[user_value]}"
password => "%{[password_value]}"
}

4

Hello @alxsss

Would it be possible to share the pipeline?

The exec input usually returns only the message field, except if you parse it as json (and your script returns json) or you process the message.

5

@Luca_Belluccini
I use grok filter to parse values.

Unfortunately, I cannot put the whole pipeline for security reasons. I wanted to let you know that if I use http filter plugin with
headers => {
Accept => "application/json"
"Authorization"=>"%{auth_value}"
}

where auth_value is another parameter that comes from input plugin->grok filter, it works. Basically it is base64 encoded value of user and password.

So, I wondered if user_value and password_value must be encoded when they are used instead of headers?

6

I also tried with
mutate
{

  add_field => { "user_value" => "xxx" }    
  add_field => { "password_value" => "xxx" }           
}

It is still failing with httprequestfaliure

7

Hello @alxsss,

I've put in place this basic pipeline to test out.

To check if the credentials are sent, I'm relying on https://postman-echo.com/post, which is an echo service: it sends back what we sent out.

input {
    exec {
        command => 'echo "myusername mypassword something else as message"'
        interval => 30
    }
}
filter {
    grok {
        match => {
            "message" => "(?<username>.*?) (?<password>.*?) %{GREEDYDATA:message}"
        }
        overwrite => [ "message" ]
    }
    http {
        url => "https://postman-echo.com/post"
        verb => "POST"
        body_format => text
        body => "%{[message]}"
        headers => {
            "mytest" => "logstash"
        }
        user => "%{[username]}"
        password => "%{[password]}"
    }
}
output {
    stdout {
        codec => rubydebug
    }
}

Result:

{
       "headers" => {
                  "date" => "Mon, 11 May 2020 23:14:19 GMT",
          "content-type" => "application/json; charset=utf-8",
        "content-length" => "466",
            "connection" => "keep-alive",
                  "etag" => "W/\"1d2-HVEOO8HN2rGf3X1N7ql/SDl9d+k\"",
                  "vary" => "Accept-Encoding",
            "set-cookie" => "sails.sid=s%3Aqvmn6ZsRnmikZbk8GJVYjvE8kLoCPWTE.dFGnbRZ66lezZI9nn8vo38phZdpI6kMgZDZulvsMHnQ; Path=/; HttpOnly"
    },
       "command" => "echo \"myusername mypassword something else as message\"",
      "@version" => "1",
    "@timestamp" => 2020-05-11T23:14:18.258Z,
      "username" => "myusername",
          "body" => {
        "headers" => {
              "x-amzn-trace-id" => "Root=1-5eb9dc4b-18148626079b62e5d55063f7",
              "accept-encoding" => "gzip,deflate",
               "content-length" => "26",
            "x-forwarded-proto" => "https",
                 "content-type" => "text/plain",
             "x-forwarded-port" => "443",
                         "host" => "postman-echo.com",
                       "mytest" => "logstash",
                   "user-agent" => "Manticore 0.6.4",
                "authorization" => "Basic JXtbdXNlcm5hbWVdfTole1twYXNzd29yZF19"
        },
           "data" => "something else as message\n",
           "form" => {},
           "json" => nil,
            "url" => "https://postman-echo.com/post",
           "args" => {},
          "files" => {}
    },
          "host" => "vm-VirtualBox",
      "password" => "mypassword",
       "message" => "something else as message\n"
}

The headers contain:

"authorization" => "Basic JXtbdXNlcm5hbWVdfTole1twYXNzd29yZF19"

Which actually represent:

%{[username]}:%{[password]}

This means the values are not interpolated by the plugin.

As a workaround, I might suggest to generate the base64 via a ruby filter:

input {
    exec {
        command => 'echo "myusername mypassword something else as message"'
        interval => 30
    }
}
filter {
    grok {
        match => {
            "message" => "(?<username>.*?) (?<password>.*?) %{GREEDYDATA:message}"
        }
        overwrite => [ "message" ]
    }
    ruby {
        init => "require 'base64'"
        code => "event.set('auth_header', 'Basic ' + Base64.strict_encode64(event.get('username')+':'+event.get('password')))"
    }
    http {
        url => "https://postman-echo.com/post"
        verb => "POST"
        body_format => text
        body => "%{[message]}"
        headers => {
            "authorization" => "%{[auth_header]}"
            "mytest" => "logstash"
        }
        remove_field => [ "auth_header"]
    }
}
output {
    stdout {
        codec => rubydebug
    }
}

Result:

{
      "username" => "myusername",
      "password" => "mypassword",
      "@version" => "1",
       "command" => "echo \"myusername mypassword something else as message\"",
       "headers" => {
                  "vary" => "Accept-Encoding",
          "content-type" => "application/json; charset=utf-8",
                  "date" => "Mon, 11 May 2020 23:33:09 GMT",
        "content-length" => "458",
            "set-cookie" => "sails.sid=s%3AFC-7qJuQoB_EaSOwrILfyUAI32dQRtIs.RG0Etoc%2BxtV4UHaXmR9nzGsFb7uAzOE0DgLK6M64MVg; Path=/; HttpOnly",
                  "etag" => "W/\"1ca-XAUDTnV4N9fHTUtt3M0czbNbfYU\"",
            "connection" => "keep-alive"
    },
    "@timestamp" => 2020-05-11T23:33:08.421Z,
          "body" => {
           "data" => "something else as message\n",
          "files" => {},
           "args" => {},
           "form" => {},
        "headers" => {
                "authorization" => "Basic bXl1c2VybmFtZTpteXBhc3N3b3Jk",
                       "mytest" => "logstash",
             "x-forwarded-port" => "443",
               "content-length" => "26",
                 "content-type" => "text/plain",
              "accept-encoding" => "gzip,deflate",
                   "user-agent" => "Manticore 0.6.4",
              "x-amzn-trace-id" => "Root=1-5eb9e0b5-aa646f61c470f74a1ce03573",
            "x-forwarded-proto" => "https",
                         "host" => "postman-echo.com"
        },
           "json" => nil,
            "url" => "https://postman-echo.com/post"
    },
       "message" => "something else as message\n",
          "host" => "vm-VirtualBox"
}

The headers contain:

"authorization" => "Basic bXl1c2VybmFtZTpteXBhc3N3b3Jk"

Which actually represent:

myusername:mypassword

I've opened a Github issue on the plugin repository: https://github.com/logstash-plugins/logstash-filter-http/issues/24

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.