Error "Panic in job" - Unable to monitor my service

Got below error in heartbeat's debug log when tried to monitor one service while it works fine for other services:

// # Configure monitors
- type: http

  # List or urls to query
  urls: [",Index=pc,CvType=persen,Frequency=2H/"]
    method: GET
    status: 200
  # body: '{"error" : null*}'
  # Configure task schedule
  schedule: '@every 10s'
    enabled: true
    verification_mode: none

2020-03-28T11:09:03.005-0400    ERROR   scheduler/scheduler.go:355      Panic in job 'http@,Index=pc,CvType=persen,Frequency=2H/'. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
2020-03-28T11:09:03.008-0400    ERROR   scheduler/scheduler.go:357      Stacktrace: goroutine 167 [running]:
runtime/debug.Stack(0x10fcf57, 0x3a, 0xc0423368d8)
        /usr/local/go/src/runtime/debug/stack.go:24 +0xae*Scheduler).runTask.func1.1(0xc042136af0, 0xc0424b7e80)
        /go/src/ +0xfe
panic(0xf4ba60, 0x1817bc0)
        /usr/local/go/src/runtime/panic.go:502 +0x237, 0xc042450a80, 0x3, 0xc0424b21a8, 0x22, 0x119e540)
        /go/src/ +0x462, 0x3, 0xc0424b21a8, 0x22, 0x8, 0xc04238a440, 0xc0424d8d70, 0xc0424d8cf8)
        /go/src/ +0xc1, 0x10bfd26, 0x3, 0xc0424b21a8, 0x22, 0x410c63, 0xc0420b4500, 0x50, 0x48)
        /go/src/ +0x55*SimpleTransport).RoundTrip(0xc0421e4a50, 0xc0421fce00, 0x0, 0x0, 0x0)
        /go/src/ +0x105
net/http.send(0xc0421fcc00, 0x1188e20, 0xc0421e4a50, 0xbf97f807b5b9dbe4, 0xb74201d71, 0x182dde0, 0xc042004138, 0xbf97f807b5b9dbe4, 0xc0424d9068, 0x1)
        /usr/local/go/src/net/http/client.go:252 +0x18c
net/http.(*Client).send(0xc0421e4a80, 0xc0421fcc00, 0xbf97f807b5b9dbe4, 0xb74201d71, 0x182dde0, 0xc042004138, 0x0, 0x1, 0x3)
        /usr/local/go/src/net/http/client.go:176 +0x101
net/http.(*Client).Do(0xc0421e4a80, 0xc0421fcc00, 0x182dde0, 0xc0421fcc00, 0x57fda2)
        /usr/local/go/src/net/http/client.go:615 +0x294, 0xc0421fcc00, 0xc0422914c0, 0xbf97f803b5b9dbe4, 0x7ba737d71, 0x182dde0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/ +0xad, 0xc0421fc900, 0x0, 0x0, 0x0, 0x3b9aca000, 0xc0422914c0, 0x0, 0x0, 0x0, ...)
        /go/src/ +0x143, 0x3, 0xc0424d96f0, 0xc0424d9758)
        /go/src/ +0x449, 0xc0424d9778, 0x410c63)
        /go/src/ +0x38, 0xf81240, 0xc0424d97a8, 0x410c63, 0xc04223e8e0, 0x20)
        /go/src/ +0x2d, 0x18, 0x20, 0x20, 0xc04223e8e0, 0x0, 0xc04205d000)
        /go/src/ +0x2e, 0xc04223e8e0, 0xc0424d97f8, 0x7cf9a8, 0xfb4920, 0xc04223e8e0)
        /go/src/ +0x51, 0x118b520, 0xc04213e3b0, 0x118b520, 0xc04223e8e0, 0x182dde0, 0x1e9614)
        /go/src/ +0x2e, 0xc0423de91b, 0x5, 0x2704b70, 0xc04209b9a8, 0x42824e)
        /go/src/ +0x940, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0420682c0, 0xc0421e58c0, 0x10c66cd, ...)
        /go/src/ +0xce, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0421e58c0, 0xc8, 0xc0421381a8, ...)
        /go/src/ +0xef*funcJob).Run(0xc04240c560, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x582169631, 0x182dde0, ...)
        /go/src/ +0x89, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc042388420, 0xc04209bf88, 0xc04209bef0, ...)
        /go/src/ +0x8d*task).prepareSchedulerJob.func1(0xc000000010, 0x110f030, 0xc042136af0)
        /go/src/ +0x84*Scheduler).runTask.func1(0xc042136af0, 0xc0424b7e80, 0xc042136af0, 0xc04240caa0)
        /go/src/ +0x62
created by*Scheduler).runTask
        /go/src/ +0x63

Based on the stacktrace it looks like you're using an old version of heartbeat. Can you upgrade to 7.6.1? It has a rewritten scheduler that should not have this bug.

Out of curiosity, which version of heartbeat was this error from?

Thanks for your reply.
The version is 6.5.0. We are using it because our ELK stack is based on 6.5.0 version. 7.6.1 was the version we tried with at the beginning, the dashboard can not be imported in Kibana 6.5.0.
As per your suggestion, in order to use heartbeat without being disturbed by this teething bug, seems upgrade the whole ELK to 7.6 the later version would be our only option here?

Looking more closely, I see this is not a scheduler bug (the scheduler is just rescuing a panic), but rather a bug elsewhere. Would you mind upgrading heartbeat to 6.8.1 and seeing if that fixes the issue? It looks like it may.

Thanks. 6.8.1 works though, failed with response body check when I uncommented in yml.

I understand from below case that check.response.body is to match the entire response body. This works great for simple examples however doesn't work when we only want to find a small string in a large body. Was the enhancement introduced?

Actually, those perform a regex match. It is case sensitive though, did you try adjusting the case?

The response would be like below, but for a valid check, only need to seek for "error": null.

body: '{"error" : null*}' doesn't work as expected. How to set correctly for matching above requirements?

  "error" : null,
  "results" : [ {
    "Id" : {
      "id" : "46245745375425",
      "metadata" : { },
      "valid" : true
    "key" : {
      "Type" : "grok.DerivedMktDataUFO.YieldCurveKey",
      "currently" : "U",
      "cType" : "Q",
      "frequency" : "2s",
      "index" : "X2"
    "objectType" : "834721935598",
    "subTag" : "Ldn",
    "tag" : "wfawfd0213",
    "transId" : "R922ff942",
    "updatedBy" : "sdsdfid",
    "userId" : "adsfghgh",
    "validFrom" : "2020-02-13T19:33:46.574",
    "validTo" : "2030-01-01T00:00:00.000"
  } ]

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.