Error "Panic in job" - Unable to monitor my service

Elastic Stack Beats
1

Got below error in heartbeat's debug log when tried to monitor one service while it works fine for other services:

hearbeat.yml:
// # Configure monitors
heartbeat.monitors:
- type: http

  # List or urls to query
  urls: ["https://dncpri1p.cny.net:21375/dataservices/v1/b15/eok/metadata/20200213/OD/grok.deoiO.YjiwG:name=jack,Index=pc,CvType=persen,Frequency=2H/"]
  check.request:
    method: GET
  check.response:
    status: 200
  # body: '{"error" : null*}'
  # Configure task schedule
  schedule: '@every 10s'
  ssl:
    enabled: true
    verification_mode: none

2020-03-28T11:09:03.005-0400    ERROR   scheduler/scheduler.go:355      Panic in job 'http@https://dncpri1p.cny.net:21375/dataservices/v1/b15/eok/metadata/20200213/OD/grok.deoiO.YjiwG:name=jack,Index=pc,CvType=persen,Frequency=2H/'. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
2020-03-28T11:09:03.008-0400    ERROR   scheduler/scheduler.go:357      Stacktrace: goroutine 167 [running]:
runtime/debug.Stack(0x10fcf57, 0x3a, 0xc0423368d8)
        /usr/local/go/src/runtime/debug/stack.go:24 +0xae
github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask.func1.1(0xc042136af0, 0xc0424b7e80)
        /go/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:357 +0xfe
panic(0xf4ba60, 0x1817bc0)
        /usr/local/go/src/runtime/panic.go:502 +0x237
github.com/elastic/beats/heartbeat/monitors/active/dialchain.TLSLayer.func1.1(0x119e540, 0xc042450a80, 0x3, 0xc0424b21a8, 0x22, 0x119e540)
        /go/src/github.com/elastic/beats/heartbeat/monitors/active/dialchain/tls.go:83 +0x462
github.com/elastic/beats/heartbeat/monitors/active/dialchain.afterDial.func1(0x10bfd26, 0x3, 0xc0424b21a8, 0x22, 0x8, 0xc04238a440, 0xc0424d8d70, 0xc0424d8cf8)
        /go/src/github.com/elastic/beats/heartbeat/monitors/active/dialchain/util.go:89 +0xc1
github.com/elastic/beats/libbeat/outputs/transport.DialerFunc.Dial(0xc04223eae0, 0x10bfd26, 0x3, 0xc0424b21a8, 0x22, 0x410c63, 0xc0420b4500, 0x50, 0x48)
        /go/src/github.com/elastic/beats/libbeat/outputs/transport/transport.go:40 +0x55
github.com/elastic/beats/heartbeat/monitors/active/http.(*SimpleTransport).RoundTrip(0xc0421e4a50, 0xc0421fce00, 0x0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/heartbeat/monitors/active/http/simple_transp.go:84 +0x105
net/http.send(0xc0421fcc00, 0x1188e20, 0xc0421e4a50, 0xbf97f807b5b9dbe4, 0xb74201d71, 0x182dde0, 0xc042004138, 0xbf97f807b5b9dbe4, 0xc0424d9068, 0x1)
        /usr/local/go/src/net/http/client.go:252 +0x18c
net/http.(*Client).send(0xc0421e4a80, 0xc0421fcc00, 0xbf97f807b5b9dbe4, 0xb74201d71, 0x182dde0, 0xc042004138, 0x0, 0x1, 0x3)
        /usr/local/go/src/net/http/client.go:176 +0x101
net/http.(*Client).Do(0xc0421e4a80, 0xc0421fcc00, 0x182dde0, 0xc0421fcc00, 0x57fda2)
        /usr/local/go/src/net/http/client.go:615 +0x294
github.com/elastic/beats/heartbeat/monitors/active/http.execRequest(0xc0421e4a80, 0xc0421fcc00, 0xc0422914c0, 0xbf97f803b5b9dbe4, 0x7ba737d71, 0x182dde0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:252 +0xad
github.com/elastic/beats/heartbeat/monitors/active/http.execPing(0xc0421e4a80, 0xc0421fc900, 0x0, 0x0, 0x0, 0x3b9aca000, 0xc0422914c0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:226 +0x143
github.com/elastic/beats/heartbeat/monitors/active/http.createPingFactory.func1(0xc0421e4900, 0x3, 0xc0424d96f0, 0xc0424d9758)
        /go/src/github.com/elastic/beats/heartbeat/monitors/active/http/task.go:171 +0x449
github.com/elastic/beats/heartbeat/monitors.MakePingIPFactory.func1.1(0x120, 0xc0424d9778, 0x410c63)
        /go/src/github.com/elastic/beats/heartbeat/monitors/util.go:194 +0x38
github.com/elastic/beats/heartbeat/monitors.MakeSimpleCont.func1(0x10, 0xf81240, 0xc0424d97a8, 0x410c63, 0xc04223e8e0, 0x20)
        /go/src/github.com/elastic/beats/heartbeat/monitors/util.go:184 +0x2d
github.com/elastic/beats/heartbeat/monitors.funcTask.Run(0xc04213e3b0, 0x18, 0x20, 0x20, 0xc04223e8e0, 0x0, 0xc04205d000)
        /go/src/github.com/elastic/beats/heartbeat/monitors/util.go:440 +0x2e
github.com/elastic/beats/heartbeat/monitors.WithFields.func1(0xc0424d9701, 0xc04223e8e0, 0xc0424d97f8, 0x7cf9a8, 0xfb4920, 0xc04223e8e0)
        /go/src/github.com/elastic/beats/heartbeat/monitors/util.go:398 +0x51
github.com/elastic/beats/heartbeat/monitors.funcTask.Run(0xc04223e8e0, 0x118b520, 0xc04213e3b0, 0x118b520, 0xc04223e8e0, 0x182dde0, 0x1e9614)
        /go/src/github.com/elastic/beats/heartbeat/monitors/util.go:440 +0x2e
github.com/elastic/beats/heartbeat/monitors.makeByHostAnyIPJob.func1(0x7, 0xc0423de91b, 0x5, 0x2704b70, 0xc04209b9a8, 0x42824e)
        /go/src/github.com/elastic/beats/heartbeat/monitors/util.go:323 +0x940
github.com/elastic/beats/heartbeat/monitors.annotated.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0420682c0, 0xc0421e58c0, 0x10c66cd, ...)
        /go/src/github.com/elastic/beats/heartbeat/monitors/util.go:140 +0xce
github.com/elastic/beats/heartbeat/monitors.MakeJob.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0421e58c0, 0xc8, 0xc0421381a8, ...)
        /go/src/github.com/elastic/beats/heartbeat/monitors/util.go:127 +0xef
github.com/elastic/beats/heartbeat/monitors.(*funcJob).Run(0xc04240c560, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x582169631, 0x182dde0, ...)
        /go/src/github.com/elastic/beats/heartbeat/monitors/util.go:438 +0x89
github.com/elastic/beats/heartbeat/monitors.(Job).Run-fm(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc042388420, 0xc04209bf88, 0xc04209bef0, ...)
        /go/src/github.com/elastic/beats/heartbeat/monitors/task.go:120 +0x8d
github.com/elastic/beats/heartbeat/monitors.(*task).prepareSchedulerJob.func1(0xc000000010, 0x110f030, 0xc042136af0)
        /go/src/github.com/elastic/beats/heartbeat/monitors/task.go:85 +0x84
github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask.func1(0xc042136af0, 0xc0424b7e80, 0xc042136af0, 0xc04240caa0)
        /go/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:362 +0x62
created by github.com/elastic/beats/heartbeat/scheduler.(*Scheduler).runTask
        /go/src/github.com/elastic/beats/heartbeat/scheduler/scheduler.go:352 +0x63
2

Based on the stacktrace it looks like you're using an old version of heartbeat. Can you upgrade to 7.6.1? It has a rewritten scheduler that should not have this bug.

Out of curiosity, which version of heartbeat was this error from?

3

Thanks for your reply.
The version is 6.5.0. We are using it because our ELK stack is based on 6.5.0 version. 7.6.1 was the version we tried with at the beginning, the dashboard can not be imported in Kibana 6.5.0.
As per your suggestion, in order to use heartbeat without being disturbed by this teething bug, seems upgrade the whole ELK to 7.6 the later version would be our only option here?

4

Looking more closely, I see this is not a scheduler bug (the scheduler is just rescuing a panic), but rather a bug elsewhere. Would you mind upgrading heartbeat to 6.8.1 and seeing if that fixes the issue? It looks like it may.

5

Thanks. 6.8.1 works though, failed with response body check when I uncommented in yml.

I understand from below case that check.response.body is to match the entire response body. This works great for simple examples however doesn't work when we only want to find a small string in a large body. Was the enhancement introduced?

6

Actually, those perform a regex match. It is case sensitive though, did you try adjusting the case?

7

The response would be like below, but for a valid check, only need to seek for "error": null.

body: '{"error" : null*}' doesn't work as expected. How to set correctly for matching above requirements?

{
  "error" : null,
  "results" : [ {
    "Id" : {
      "id" : "46245745375425",
      "metadata" : { },
      "valid" : true
    },
    "key" : {
      "Type" : "grok.DerivedMktDataUFO.YieldCurveKey",
      "currently" : "U",
      "cType" : "Q",
      "frequency" : "2s",
      "index" : "X2"
    },
    "objectType" : "834721935598",
    "subTag" : "Ldn",
    "tag" : "wfawfd0213",
    "transId" : "R922ff942",
    "updatedBy" : "sdsdfid",
    "userId" : "adsfghgh",
    "validFrom" : "2020-02-13T19:33:46.574",
    "validTo" : "2030-01-01T00:00:00.000"
  } ]
}
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.