We have bunch of apps monitored via HTTP endpoints.
However, for one of them, heartbeat writes wrong to data to elasticsearch. Data in tls.server.x509 contains old outdated certificate, however it was changed many months ago.
Tried to fetch certificate using curl from heartbeat host, it returns proper latest valid certificate info.
Where the problem can be? Is there some internal cache in heartbeat or else?
1 Like
Did you try to restart the heartbeat or run it locally? It looks like a potential issue with heartbeat.
Hi!
Yes, I tried to restart it, makes no difference.
I will try to run on my local machine, but I assume everything will work fine.
And this might be related to heartbeat upgrade from 7.4.0 to 7.9.2. Before the upgrade we received correct data, but after it we started to see outdated info.
Also, field was at tls.certificate_not_valid_after in 7.4.0 and now in 7.9.2 we have it in two fields (probably for back-compatibility) in tls.certificate_not_valid_after and tls.server.x509.not_after.
Does it makes sense to try to delete and reinstall it? I see that data folder is at /var/lib/heartbeat, but there is just empty heartbeat.lock and meta.json with one UUID, so I'm not sure where else it could save/cache some data.
Ok, I tried to run both versions on local PC with clean data folders.
And weirdly enough, 7.4.0 returns proper data, and 7.9.2 returns expired certificate.
And looks like I found the reason, looks like 7.4.0 checks only lowest level certificate expiration date, and 7.9.2 checks entire chain.
Expiration date does not match the old certificate, it matches the expiration date of root certificate which should be updated everywhere already.
I checked site using https://www.sslshopper.com/ssl-checker.html and it shows "USERTrust RSA Certification Authority" as expired.
Interesting that it shows as valid on IIS.
It looks like the problem is in "USERTrust RSA Certification Authority" issued by "AddTrust External CA Root" which expired on 30.05.2020 and root certificate is not properly updated everywhere.
https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020
It shows as valid in browsers and in windows certificate manager, but shows up as expired when using heartbeat or openssl (it also shows old root cert in a chain when executing openssl s_client -showcerts -servername $HOST -connect $HOST:443).
Does heartbeat use openssl internally? Does it use some other CA certificate storage than the one in OS?
When checking CA roots in SSL folder, I see "USERTrust" one is valid:
~$ ls /etc/ssl/certs -ag | grep USERTrust
lrwxrwxrwx 1 root 76 Sep 3 2019 USERTrust_ECC_Certification_Authority.pem -> /usr/share/ca-certificates/mozilla/USERTrust_ECC_Certification_Authority.crt
lrwxrwxrwx 1 root 76 Sep 3 2019 USERTrust_RSA_Certification_Authority.pem -> /usr/share/ca-certificates/mozilla/USERTrust_RSA_Certification_Authority.crt
lrwxrwxrwx 1 root 41 Sep 3 2019 f30dd6ad.0 -> USERTrust_ECC_Certification_Authority.pem
lrwxrwxrwx 1 root 41 Sep 3 2019 fc5a8f99.0 -> USERTrust_RSA_Certification_Authority.pem
~$ openssl x509 -in /etc/ssl/certs/USERTrust_RSA_Certification_Authority.pem -noout -subject -dates
subject=C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
notBefore=Feb 1 00:00:00 2010 GMT
notAfter=Jan 18 23:59:59 2038 GMT
But when I connect and displaying the chain, it's not:
Dump each one in chain:
~$ openssl s_client -showcerts -verify 5 -connect $HOST:443 < /dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/){a++}; out="cert"a".crt"; print >out}'
We need last one:
~$ openssl x509 -in cert3.crt -noout -subject -dates
subject=C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
notBefore=May 30 10:48:38 2000 GMT
notAfter=May 30 10:48:38 2020 GMT
Meanwhile, can I specify heartbeat to use only site level certificate dates somehow like it was before?
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.