Heartbeat error DEBUG [tcp] tcp/task.go:58 check failed with: EOF

Elastic Stack Beats
1

Hello,

I am trying to do heartbeat TCP check to my oracle database port. The database is up, but heartbeat is reporting is as down. Debug log is showing the below error:

2019-03-29T12:46:07.748+0800 DEBUG [tcp] tcp/task.go:58 check failed with: EOF

Has anyone experience this before? NC and curl is showing below result:

# curl -v -XHEAD http://10.4.3.5:1525
* About to connect() to 10.4.3.5 port 1525 (#0)
*   Trying 10.4.3.5...
* Connected to 10.4.3.5 (10.4.3.5) port 1525 (#0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.4.3.5:1525
> Accept: */*
>
* Empty reply from server
* Connection #0 to host 10.4.3.5 left intact
curl: (52) Empty reply from server


# nc -v 10.4.3.5 -port 1525
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.4.3.5:1525.
^C

Debug log of 1 iteration:

2019-03-29T13:07:22.747+0800    DEBUG   [scheduler]     scheduler/scheduler.go:343      Start job 'tcp-tcp@10.4.3.5:1525' at 2019-03-29 13:07:22.74699963 +0800 +08 m=+1280.019880263.
2019-03-29T13:07:22.747+0800    DEBUG   [scheduler]     scheduler/scheduler.go:206      Next wakeup time: 2019-03-29 13:07:27.746843045 +0800 +08
2019-03-29T13:07:22.748+0800    DEBUG   [tcp]   tcp/task.go:58  check failed with: EOF
2019-03-29T13:07:22.748+0800    DEBUG   [publish]       pipeline/processor.go:308       Publish event: {
  "@timestamp": "2019-03-29T05:07:22.747Z",
  "@metadata": {
    "beat": "heartbeat",
    "type": "doc",
    "version": "6.6.1"
  },
  "tags": [
    "heartbeat"
  ],
  "beat": {
    "name": "heartbeat-shipper",
    "hostname": "AZ1AMGTVMGTAPM01.30.tntx.gov.sg",
    "version": "6.6.1"
  },
  "host": {
    "name": "AZ1AMGTVMGTAPM01.30.tntx.gov.sg",
    "os": {
      "name": "Red Hat Enterprise Linux Server",
      "codename": "Maipo",
      "platform": "rhel",
      "version": "7.6 (Maipo)",
      "family": ""
    },
    "id": "6b200b586216467dbeb9924b91021156",
    "containerized": true,
    "architecture": "x86_64"
  },
  "meta": {
    "cloud": {
      "machine_type": "c4.2xlarge",
      "region": "ap-southeast-1",
      "availability_zone": "ap-southeast-1a",
      "provider": "ec2",
      "instance_id": "i-098c0838f3467b723"
    }
  },
  "tcp": {
    "rtt": {
      "connect": {
        "us": 639
      }
    },
    "port": 1525
  },
  "monitor": {
    "scheme": "tcp",
    "name": "tcp",
    "type": "tcp",
    "ip": "10.4.3.5",
    "duration": {
      "us": 1205
    },
    "status": "down",
    "id": "tcp-tcp@10.4.3.5:1525",
    "host": "10.4.3.5"
  },
  "error": {
    "type": "io",
    "message": "EOF"
  }
}
2019-03-29T13:07:22.748+0800    DEBUG   [scheduler]     scheduler/scheduler.go:248      Job 'tcp-tcp@10.4.3.5:1525' returned at 2019-03-29 13:07:22.748482623 +0800 +08 m=+1280.021363292 (cont=0).
2019-03-29T13:07:22.748+0800    DEBUG   [scheduler]     scheduler/scheduler.go:206      Next wakeup time: 2019-03-29 13:07:27.746843045 +0800 +08
2019-03-29T13:07:23.753+0800    DEBUG   [elasticsearch] elasticsearch/client.go:323     PublishEvents: 1 events have been published to elasticsearch in 4.659968ms.
2019-03-29T13:07:23.753+0800    DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [255: 0, 1]
2019-03-29T13:07:23.753+0800    DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=1, start-seq=256, end-seq=256

2019-03-29T13:07:23.753+0800    DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2019-03-29T13:07:23.753+0800    DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack
2

I have the exact same issue but with tcp check on postgresql. I think the main issue is that we don't know exactly what message to send via tcp in order to get an answer from postgres.

3

Hi,

You need to post your config for the monitor because we can't see how you configured it.
Also maybe this can help:

A heartbeat monitor which would do the equivalent of this:

# nc -v 10.4.3.5 -port 1525
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.4.3.5:1525.
^C

would be this:

- type: tcp
  schedule: '@every 5s'
  hosts: ["10.4.3.5"]
  ports: [1525]

But you didn't post your config so it's hard to help.

Martin

4

Seconding this. We'll need to see the config to determine the root cause here.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.