Inconsistent response code from heartbeat and curl command from same machine

Elastic Stack Beats
1

Hi, I'm facing an issue where heartbeat 7.5 reports a different response code (500) value than cUrl (302) reports on the same target URL I believe the heartbeat can me mimicked through curl -Iv.

Below the heartbeat configuration in monitor.d, the corresponding response doc and curl output. What could be causing this? BTW, I have tried both with and without a trailing slash in urls.

- type: http
  name: 'somesite.com'
  tags: ['customer']
  id: 'somesite.com'
  schedule: '@every 60s'
  urls: ["http://somesite.com:80/"]
  timeout: 5
  check.request.method: HEAD

{
  "_index": "heartbeat-7.5.0-customer-2019.12",
  "_type": "_doc",
  "_id": "4ThJ_24B-hakpvoIg4XP",
  "_version": 1,
  "_score": null,
  "_size": 1479,
  "_source": {
    "observer": {
      "ip": "###########",
      "geo": {
        "name": "AWS EU Frankfurt",
        "country_iso_code": "DE",
        "city_name": "Frankfurt",
        "continent_name": "Europe"
      },
      "hostname": "###########"
    },
    "summary": {
      "up": 0,
      "down": 1
    },
    "fields": {
      "customer": "customer",
      "logstash_used": "true",
      "index": "heartbeat-7.5.0-customer-2019.12",
      "logstash_host": "ip-172-31-24-135",
      "size": 1604
    },
    "tags": [
      "customer",
      "beats_input_raw_event"
    ],
    "host": {
      "name": "###########"
    },
    "resolve": {
      "ip": "###########",
      "rtt": {
        "us": 321170
      }
    },
    "@version": "1",
    "@timestamp": "2019-12-13T12:43:47.074Z",
    "url": {
      "scheme": "http",
      "full": "http://somesite.com:80/",
      "domain": "somesite.com",
      "path": "/",
      "port": 80
    },
    "error": {
      "type": "validate",
      "message": "500 Internal Server Error"
    },
    "ecs": {
      "version": "1.1.0"
    },
    "monitor": {
      "type": "http",
      "ip": "###########",
      "check_group": "10eedc6e-1da6-11ea-8856-029326932cd8",
      "name": "somesite.com",
      "duration": {
        "us": 357687
      },
      "id": "somesite.com",
      "status": "down"
    },
    "tcp": {
      "rtt": {
        "connect": {
          "us": 15703
        }
      }
    },
    "event": {
      "dataset": "uptime"
    },
    "agent": {
      "type": "heartbeat",
      "hostname": "###########",
      "id": "28dd306a-1424-42ce-a543-d68e85641532",
      "ephemeral_id": "cd79aee3-8de9-471d-8616-9cc991a3bbbd",
      "version": "7.5.0"
    },
    "http": {
      "response": {
        "body": {
          "bytes": 0,
          "hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
          "content": ""
        },
        "status_code": 500
      },
      "rtt": {
        "total": {
          "us": 36474
        },
        "content": {
          "us": 31
        },
        "response_header": {
          "us": 20692
        },
        "write_request": {
          "us": 25
        },
        "validate": {
          "us": 20723
        }
      }
    }
  },
  "fields": {
    "availability": [
      0
    ],
    "url.link": [
      "http://somesite.com"
    ],
    "@timestamp": [
      "2019-12-13T12:43:47.074Z"
    ],
    "url.suffix": [
      "au"
    ]
  },
  "highlight": {
    "monitor.name": [
      "@kibana-highlighted-field@somesite.com@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1576241027074
  ]
}

curl command:

curl -Iv  http://somesite.com:80
* Rebuilt URL to: http://somesite.com:80/
*   Trying ###########...
* TCP_NODELAY set
* Connected to somesite.com (###########) port 80 (#0)
> GET / HTTP/1.1
> Host: somesite.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Keep-Alive: timeout=5, max=1024
< Content-Type: text/html; charset=UTF-8
< Location: http://somesite.com/studio/agegate?rurl=/
< Server: Apache
< Date: Fri, 13 Dec 2019 12:39:21 GMT
< Content-Length: 0
<
* Connection #0 to host somesite.com left intact
2

Hi @J_Weeda,

I think this can be an issue with redirect support, it is planned to improve it in 7.6.0 after this change: https://github.com/elastic/beats/pull/14125

In the meantime you can try to disable redirections by setting max_redirects option to 0 (this will be the default in 7.6.0).

If disabling redirections doesn't fix the issue it'd be great if you could try the patch previously mentioned.

Thanks!

3

This could potentially also be the result of a different user agent.

It'd be helpful to dive into the server-side of the 500 error and see what's going on. If a server is returning a 500 that generally indicates a bug on the server side.

4

thanks for your replies...

- type: http
  name: 'somesite.com'
  tags: ['customer']
  id: 'somesite.com'
  schedule: '@every 60s'
  urls: ["http://somesite.com:80"]
  max_redirects: 0
  timeout: 5
  check.request.method: HEAD
  headers:
    'Host': 'somesite.com'
    'User-Agent': 'curl/7.58.0'

I have made above changes but still the same result. The headers are the same as the cUrl request uses. Is max_redirects: 0 supposed to work in 7.5 already or have I misread your suggestion @jsoriano?

5

max_redirects: 0 should work correctly in 7.5.

Can you view the logs of the server generating the 500 error? That should have more info as to the precise cause of issues here.

6

@Andrew_Cholakian1, I will try but their not under my control.

7

@J_Weeda one question that may help here: why are you manually setting the Host header in your request?

8

I wasn't initially but I wanted to explain why the cUrl request works and heartbeat didn't. These two headers are added by cUrl. I started with the user-agent and then also added host since I didn't notice any difference.

But there is some success to report back. The config I posted (in which the headers where included) was not correct. It should have been like this:

- type: http
  name: 'somesite.com'
  tags: ['customer']
  id: 'somesite.com'
  schedule: '@every 60s'
  urls: ["http://somesite.com:80"]
  max_redirects: 0
  timeout: 5
  check.request:
    method: HEAD
    headers:
      'Host': 'somesite.com'
      'User-Agent': 'curl/7.58.0'

in fact the config worked without the Host header as well. The site reported a 500 because the User-Agent header was missing (or not accepted).

@Andrew_Cholakian1, @jsoriano Thanks for your support!

1 Like
9

Ah, makes sense. Thanks for letting us know it's been resolved!

So, to summarize the issue was that the service returned a 500 for the heartbeat default user agent (which is the go user agent). This should (probably?) cease to be an issue in 7.6.0 with the new default agent. It really depends on how the service on the other side behaves.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.