Hello. I'm installing HeartBeat for the first time. I've got the basic ping uptime check working. However, I would love to match on a keyword on the page. I keep coming across articles saying this will only match on the entire page body. I've been trying regular expression matches on simple keywords with no luck. Does anyone have an example of how to set this up in the YAML file? Thanks!
Hi @ozonshak There is an example give on heartbeat config docs HTTP options | Heartbeat Reference [8.11] | Elastic
something like this
check.response:
status: [200]
body:
- Saved
- saved
Hi @shahzad31. Unfortunately, that does not work. I've already poured over that documentation. For some reason, simple individual words don't see to match anything (I've also tried matching case exactly). I've also tried some regex and JSON (although the latter might be only applicable if the response is in JSON - my response is HTML). I'm not sure what I'm doing wrong.
Does this only work if the entire response matches? Or should I be able to pick out a single word or phrase in the entire response?
Hi @ozonshak It should match a single word as well, it might be a bug, can you please open an issue https://github.com/elastic/beats/issues/new?template=bug-report.md We will try to take a look at it.
Hi @ozonshak i did bit more digging, and it seems to be working as expected for me, i think one of thing you may need to notice is that make sure, whole body is recorded in response, with parameter
response.include_body_max_bytes
i think it might be a case that the text you are matching against isn't part of default first 1024 bytes, once you do that, you can verify that if text is part of the body you are receiving in uptime monitor details page in ping history table, like below.
One more thing of consideration is to make sure to check if there is any redirect involved, i was monitoring https://ir.elastic.co/ but it was actually redirecting to https://ir.elastic.co/home/default.aspx, so i was expecting text of actual redirect to match against initial request, but if you haven't set heartbeat to follow redirects , it will match it against first response.
Thanks @shahzad31. I tried to get this syntax in, but I think I have an error because every time I save the file, my heartbeat checks basically stop. The online documentation doesn't really show an example of these that I can find. I've tried the following:
#response:
#include_body_max_bytes
#include_body: "always"
#response.include_body_max_bytes
#response.include_body: "always"
When drilling into a particular ping check, it is telling me that my response body is currently not being recorded. So, I think you are on the right track here. I just need to get the syntax correct.
@ozonshak So that settings is per monitor, let me show post an example, include_body_max_bytes means number value in bytes
- type: http
id: ir-elastic-co
name: Elastic Investor Relations
urls: ["https://ir.elastic.co/home/default.aspx"]
schedule: "@every 60s"
tags: ["org:elastic"]
response.include_body_max_bytes: 99999
response.include_body: on_error
check.response:
body:
- Latest
There is definitely room in docs improvement, we will keep an eye on it.
Thanks @shahzad31. Well, I'm a little closer, but still not working. Adding the response.include_ entries seem to have been ok (didn't stop the ping). However, as soon as I add the "body:" portion, the heartbeat stopped checking. Here is that portion of my script. Are you able to see anything wrong? As soon as I comment out the "body" and "About" lines, it starts pinging again.
# Required TLS protocols
#supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"]
response.include_body_max_bytes: 99999
response.include_body: on_error
# Request settings:
check.request:
# Configure HTTP method to use. Only 'HEAD', 'GET' and 'POST' methods are allowed.
method: "GET"
# Dictionary of additional HTTP headers to send:
#headers:
# Optional request body content
#body:
# Expected response settings
check.response:
# Expected status code. If not configured or set to 0 any status code not
# being 404 is accepted.
status: 200
# Required response headers.
#headers:
# Required response contents.
body:
- About
# Parses the body as JSON, then checks against the given condition expression
#json:
#- description: Check for page keyword
# condition:
# contains:
# http.response.body: About
Looks ok to me, @andrewvc can you see what might be wrong here?
Hey, jumping in here a little late. So, the body check should match any substring within the first 100MiB.
response.include_body and response.include_body_max_bytes only effect how much of the body is sent to ES.
I double checked this functionality locally here and it worked with this config:
- type: http
id: my-monitor
name: My Monitor
urls:
- https://www.elastic.co
schedule: "@every 10s"
check.response:
status: 200
body:
- elastic
@ozonshak can you post a screenshot of the HTTP response details from the monitor overview page, or even better, post a document from ES? Is the validation succeeding or failing?
Sorry guys. Can you please guide me to this location? I'm looking on the overview page but there are no HTTP response details. When I drill down into a monitor, I'm not seeing much either. When this fails due to changes in the YML, it stops pinging, so there are no further entries.
Overview page:
Drill-down:
I'm relatively new to working with these tools, so I could also use some assistance in pulling a document out of the heartbeat index. Thanks!
To get a document , you can go to devtools and use query like this
GET heartbeat-*/_search
{
"size": 2,
"query": {
"match_all": {}
}
}
Thanks @shahzad31. So, this is returning the oldest 2 documents that were "down" failures while I was trying to set this up. Do you have a quick sort that you can please add to that search? I've been trying several ways, but it keeps telling me it can't find a mapping for any fields I try and sort on. Thanks.
FYI - I was able to get a response body by changing the response.include_body to "always" instead of "on_error". I see a ton coming back but it won't let me scroll so I can't tell how much of the page it is capturing. I tried using a random string that I could see in the response body, but the heartbeat still stopped pinging entirely as soon as I add the body portion of the YML. Not sure what is wrong with this, but it is not taking any values without crashing the process.
Hi @shahzad31. Any further thoughts as to why I'm having this issue? I'm at a total loss. Every time I uncomment the "body" section, it basically stops heartbeat from pinging anything. I haven't been able to see any errors - the entire beat just stops working. Here is that section of the YML file:
response.include_body_max_bytes: 99999
response.include_body: always
#response.include_body: on_error
# Request settings:
check.request:
# Configure HTTP method to use. Only 'HEAD', 'GET' and 'POST' methods are allowed.
method: "GET"
# Dictionary of additional HTTP headers to send:
#headers:
# Optional request body content
#body:
# Expected response settings
check.response:
# Expected status code. If not configured or set to 0 any status code not
# being 404 is accepted.
status: 200
# Required response headers.
#headers:
# Required response contents.
body:
- Heart
If I comment out the "body" and "heart" lines above, Heartbeat starts working again.
Hi @ozonshak, when heartbeat stops working, it must shows you some error in console that why it's not starting.
Thanks @shahzad31! I totally forgot you can run this in console mode. I was checking /var/log and output from journalctl, but wasn't seeing any errors. The console showed me an issue with the syntax. Once I searched for the error, it was because I had used a tab character to indent the "-" line, which it doesn't like. Once I changed it to spaces, it started working! Thanks for your help!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.