Http poller removing tags and type

Elastic Stack Logstash
1

I am using http poller and as I was debugging I noticed some strange behavior.

I am using logstash 2.0 and when I shut down nginx at one of the endpoints being polled I get this back.

So when successful the json response parses fine and all is well. But when I turn off nginx and it gets a 502/503 the tags/type is missing.

Any thoughts?

{
               "@version" => "1",
             "@timestamp" => "2015-11-09T19:04:27.752Z",
   "http_poller_metadata" => {
                  "name" => "cnam-api03",
                  "host" => "HOSTNAME",
               "request" => {
           "method" => "get",
              "url" => "http://DOMAIN.COM/DID"
       },
       "runtime_seconds" => nil
   },
                   "tags" => [
       [0] "_http_request_failure"
   ],
   "http_request_failure" => {
               "request" => {
           "method" => "get",
              "url" => "http://DOMAIN.COM/DID"
       },
                  "name" => "cnam-api03",
                 "error" => "Connection refused",
             "backtrace" => nil,
       "runtime_seconds" => 0.08
   }
}

Its really weird because when a request is good its like

{
                "response" => {
        "code" => 200,
        "data" => {
                     "_id" => "561effd322755d907d8b4567",
                  "vendor" => "VENDOR",
                    "cnam" => "WIRELESS CALLER",
            "phone_number" => 8888888888,
              "expired_at" => "2016-10-15 00:00:00",
              "created_at" => "2015-10-15 01:22:27"
        }
    },
                 "request" => 8888888888,
                "@version" => "1",
              "@timestamp" => "2015-11-09T19:21:24.786Z",
    "http_poller_metadata" => {
                    "name" => "cnam-api03",
                    "host" => "HOSTNAME",
                 "request" => {
            "method" => "get",
               "url" => "http://DOMAIN.COM/DID"
        },
         "runtime_seconds" => 0.221,
                    "code" => 200,
        "response_headers" => {
                       "server" => "nginx/1.8.0",
                 "content-type" => "application/json",
            "transfer-encoding" => "chunked",
                   "connection" => "keep-alive",
                "cache-control" => "no-cache",
                         "date" => "Mon, 09 Nov 2015 19:20:46 GMT"
        },
        "response_message" => "OK",
           "times_retried" => 0
    },
                    "type" => "cnamhealthcheck",
                    "tags" => [
        [0] "cnam_healthcheck"
    ],
                   "@host" => "cnam-api03"
}

Input file

   input {
      http_poller {
          urls => {
            "cnam-api03" => "http://DOMAIN.COM/DID"
          }   
           automatic_retries => 0

            # Check the site every 5s
            interval => 5
    
            # data is json
            codec => "json"
    
            # Wait no longer than 2 seconds for the request to complete
            request_timeout => 2
    
            # Store metadata about the request in this field
            metadata_target => http_poller_metadata
    
            # Tag this request so that we can throttle it in a filter
            tags => cnam_healthcheck
    
            # set the data type
            type => cnamhealthcheck
        }   
    
    }
2

Hi there,

I wish I could give you a solution, but I am also similar/same behavior, although I am not parsing any JSON data (my codec is 'plain'), I'm simply reading the HTTP response code returned in the response.

In my scenario, a 503 will work correctly (presumably because I'm simply checking the code rather than parsing content), but if the web server is unreachable and http_poller times out, this is when I get the same behavior which you're seeing.

It appears as though 'type' gets cleared, and any tags are overwritten with '_http_request_failure'

Thanks