Cannot display HTTP status in kibana dash

Elastic Stack Beats
1

Hello,

I have installed heartbeat on one of the host to monitor two URL's but it doesn't seems to work correctly to monitor one of that URL. Attaching heartbeat logs

2017-08-28T11:09:59-04:00 DBG  Job 'http@http://X.X.X.X:8080/login' returned at 2017-08-28 11:09:59.430362951 -0400 EDT (cont=0).
2017-08-28T11:09:59-04:00 DBG  Next wakeup time: 2017-08-28 11:10:09.41972809 -0400 EDT
2017-08-28T11:09:59-04:00 DBG  Publish: {
  "@timestamp": "2017-08-28T15:09:59.420Z",
  "beat": {
    "hostname": "jenkins",
    "name": "jenkins",
    "version": "5.5.2"
  },
  "duration": {
    "us": 34401
  },
  "http_rtt": {
    "us": 33923
  },
  "ip": "X.X.X.X",
  "monitor": "http@http://X.X.X.X:8081/login/login.htm",
  "port": 8081,
  "response": {
    "status": 200
  },
  "rtt": {
    "us": 34323
  },
  "scheme": "http",
  "tcp_connect_rtt": {
    "us": 202
  },
  "type": "http",
  "up": true,
  "url": "http://X.X.X.X:8081/login/login.htm"
}
2017-08-28T11:09:59-04:00 DBG  Job 'http@http://X.X.X.X:8081/login/login.htm' returned at 2017-08-28 11:09:59.454686381 -0400 EDT (cont=0).
2017-08-28T11:09:59-04:00 DBG  Next wakeup time: 2017-08-28 11:10:09.41972809 -0400 EDT
2017-08-28T11:10:00-04:00 DBG  output worker: publish 2 events
2017-08-28T11:10:00-04:00 DBG  Try to publish 2 events to logstash with window size 10
2017-08-28T11:10:00-04:00 DBG  2 events out of 2 events sent to logstash. Continue sending
2017-08-28T11:10:00-04:00 DBG  send completed
2017-08-28T11:10:09-04:00 DBG  Start job 'http@http://X.X.X.X:8081/login/login.htm' at 2017-08-28 11:10:09.419933706 -0400 EDT.
2017-08-28T11:10:09-04:00 DBG  Start job 'http@http://X.X.X.X:8080/login' at 2017-08-28 11:10:09.420049433 -0400 EDT.
2017-08-28T11:10:09-04:00 DBG  Next wakeup time: 2017-08-28 11:10:19.41972809 -0400 EDT
2017-08-28T11:10:09-04:00 DBG  Publish: {
  "@timestamp": "2017-08-28T15:10:09.420Z",
  "beat": {
    "hostname": "jenkins",
    "name": "jenkins",
    "version": "5.5.2"
  },
  "duration": {
    "us": 3575
  },
  "http_rtt": {
    "us": 2908
  },
  "ip": "X.X.X.X",
  "monitor": "http@http://X.X.X.X:8080/login",
  "port": 8080,
  "response": {
    "status": 200
  },
  "rtt": {
    "us": 3227
  },
  "scheme": "http",
  "tcp_connect_rtt": {
    "us": 156
  },
  "type": "http",
  "up": true,
  "url": "http://X.X.X.X:8080/login"
}

Config -

# Configure monitors
heartbeat.monitors:
- type: http

  # List or urls to query
  urls: ["http://10.152.58.27:8080/login", "http://10.152.58.27:8081/login/login.htm"]

  # Configure task schedule
  schedule: '@every 10s'

  # Total test connection and data exchange timeout
  #timeout: 16s

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
  env: development


#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["X.X.X.X:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/heartbeat
  name: heartbeat.log
  keepfiles: 7

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
logging.selectors: ["*"]

It publishes events only for http://X.X.X.X:8081

2

Is something missing in the logs or did you stop the beat early? Normally all beats do print some internal metrics (only the changed ones) to the log at about every 30 seconds. I'm also a little surprised the logstash output reporting 2 events being reported. It's 2 events for the port 8080? Given the schedule is every 10 seconds I would expect the output did send events for both ports in this batch.

Heartbeat 5.x has an internal buffer for batching multiple events (configurable via flush_interval). Look for Flush Interval set to in the log output.

Do you have any filters in logstash, potentially filtering out events. What you use logstash for?

Btw. the beats 6.0 beta2 release should be available hopefully in a few days. A number of changes and improvements coming to heartbeat, like new event and improved event format + new publisher pipeline without flush timeout (more immediate sending).

3

@steffens the logs are from heartbeat. I have added logs only for port 8080. Heartbeat reports the URL as success for other port - 8081.

The two events are for different ports - 8080 and 8081. I'm using logstash for beats as input, filter syslog and nginx logs and output to ES.

For 8080 its showing 200, but fails to report

  "@timestamp": "2017-08-29T14:03:38.955Z",
  "beat": {
    "hostname": "jenkins",
    "name": "jenkins",
    "version": "5.5.2"
  },
  "duration": {
    "us": 4302
  },
  "http_rtt": {
    "us": 3816
  },
  "ip": "X.X.X.X",
  "monitor": "http@http://X.X.X.X:8080/login",
  "port": 8080,
  "response": {
    "status": 200
  },
  "rtt": {
    "us": 4252
  },
  "scheme": "http",
  "tcp_connect_rtt": {
    "us": 256
  },
  "type": "http",
  "up": true,
  "url": "http://X.X.X.X:8080/login"
}
4

Without complete logs I have a hard time to see anything weird. Once the event is logged in the debug logs, it is put into the publisher pipeline. Heartbeat is not filtering it out. By default heartbeat retries to send an event up to 3 times. But I didn't see any errors in your logs.

Can you try heartbeat and only port 8080 being enabled.

For testing use this logstash configuration:

input {
  beat {
    port => 5044
  }
}

output {
  stdout {}
}

No filtering, no indexing. Just for us to figure if logstash actually receives the events.

Have you also checked the logstash configs for any errors? Or the elasticsearch for errors when indexing?

5

@steffens Okay, so removed the other URL which was working so heartbeat can report only for 8080 and attached are the logs.

 2017-08-29T12:29:10-04:00 DBG  Start job 'http@http://X.X.X.X:8080/login' at 2017-08-29 12:29:10.250331409 -0400 EDT.
2017-08-29T12:29:10-04:00 DBG  Next wakeup time: 2017-08-29 12:29:20.250106799 -0400 EDT
2017-08-29T12:29:10-04:00 DBG  Publish: {
  "@timestamp": "2017-08-29T16:29:10.250Z",
  "beat": {
    "hostname": "jenkins",
    "name": "jenkins",
    "version": "5.5.2"
  },
  "duration": {
    "us": 3949
  },
  "http_rtt": {
    "us": 3297
  },
  "ip": "10.152.58.27",
  "monitor": "http@http://X.X.X.X:8080/login",
  "port": 8080,
  "response": {
    "status": 200
  },
  "rtt": {
    "us": 3681
  },
  "scheme": "http",
  "tcp_connect_rtt": {
    "us": 208
  },
  "type": "http",
  "up": true,
  "url": "http://X.X.X.X:8080/login"
}
2017-08-29T12:29:10-04:00 DBG  Job 'http@http://X.X.X.X:8080/login' returned at 2017-08-29 12:29:10.254635317 -0400 EDT (cont=0).
2017-08-29T12:29:10-04:00 DBG  Next wakeup time: 2017-08-29 12:29:20.250106799 -0400 EDT
2017-08-29T12:29:11-04:00 DBG  output worker: publish 1 events
2017-08-29T12:29:11-04:00 DBG  Try to publish 1 events to logstash with window size 5
2017-08-29T12:29:11-04:00 DBG  1 events out of 1 events sent to logstash. Continue sending
2017-08-29T12:29:11-04:00 DBG  send completed

There are no errors in LS or ES.

6

@steffens I changed my URL to http://X.X.X.X:8080/ and now it reports under beats http monitoring, but reports as up:false.

The app is up and I can access it.

I guess it's more of an URL issue instead of ELK/beats issue.

 "@timestamp": "2017-08-29T16:53:47.266Z",
  "beat": {
    "hostname": "jenkins",
    "name": "jenkins",
    "version": "5.5.2"
  },
  "duration": {
    "us": 3343
  },
  "error": {
    "message": "403 Forbidden",
    "type": "validate"
  },
  "http_rtt": {
    "us": 2741
  },
  "ip": "X.X.X.X",
  "monitor": "http@http://X.X.X.X:8080/",
  "port": 8080,
  "scheme": "http",
  "tcp_connect_rtt": {
    "us": 251
  },
  "type": "http",
  "up": false,
  "url": "http://X.X.X.X:8080/"
}
7

The log using http://X.X.X.X:8080/login clearly shows heartbeat did send the event to Logstash.
Message 1 events out of 1 events sent to logstash..
The window size of 5 makes me wonder a little. This indicates there has been another failed attempt before that failed either due to LS not running or some connection error (as some log messages are missing I can not really tell).

Have you tried to publish the data with my minimalist logstash configuration (just run in foreground with logstash -f test.conf). This should print the event to your terminal.

Have you considered having heartbeat to send directly to Elasticsaerch?

What kind of filtering do you apply in logstash?

Check the error message in your event after changing to http://X.X.X.X:8080/ :slight_smile:

  "error": {
    "message": "403 Forbidden",
    "type": "validate"
  },
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.