TLS Certificate Expired issue in Uptime

dabo · December 21, 2021, 12:53pm

Hi,

After renewing certificate on all levels for one domain we still have TLS Certificate showing Expired message in its monitor in Uptime.
Interestingly monitor is not down, but shows expired certificate.

Anyone know what is it about?

Kind regards,
Dane

Dominique_Clarke · December 21, 2021, 8:29pm

Hi @dabo. Thanks for reaching out.

We'll need additional information to fully investigate. Which version of the Elastic Stack and Heartbeat are you running?

Certificate information displayed on the Overview page is based on the most recent check for that monitor. It may be helpful to query the last check of the monitor you're having trouble with to verify the certificate information. You can do so with the following query in the Dev Tools console found in Monitor Management -> Dev Tools.

GET heartbeat-*/_search
{
  "size": 1,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "monitor.id": "YOUR_MONITOR_ID"
          }
        },
        {
          "exists": {
            "field": "summary"
          }
        }
      ]
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ],
  "fields": [
    "@timestamp",
    "tls.server.x509.not_after",
    "tls.server.x509.not_before"
  ],
  "_source": false
}

You may need to adjust the index you're querying against, depending on whether you've customized index names or utilize cross-cluster search.

Once you receive the results of that query, you can find the not_after and not_before value of the matching certificate. You can then verify if it's expired (or aging/nearing expiration based on the values configured in Uptime settings).

If the returned certificate information is indeed expiring, aging, or nearing expiration, we can follow up by investigating why Heartbeat may contain outdated certificate information. If the returned certificate is not expired, aging, or nearing expiration, we can explore why the Uptime UI continues to display the certificate as expired.

dabo · December 22, 2021, 6:15am

HI @Dominique_Clarke , thank you for the reply.

Elastic is on 7.13.1 and HBs are on 7.15 version.

I ran the query:

        "fields" : {
          "tls.server.x509.not_before" : [
            "2021-12-06T12:55:11.000Z"
          ],
          "@timestamp" : [
            "2021-12-22T06:09:45.305Z"
          ],
          "tls.server.x509.not_after" : [
            "2021-12-08T15:59:50.000Z"
          ]
        }

Everything else works fine.

Let me know if you need additional info.

Kind regards,
Dane

Dominique_Clarke · December 22, 2021, 3:53pm

Hi @dabo. Thank you for following up with that information. As you can see, based on the tls.server.x509.not_after value, the information being reported by the monitor indicates that the certificate is indeed expired, resulting in the certificate being displayed as expired on the Uptime Overview page. If you feel this information is incorrect, we can follow up with why Heartbeat may be reporting incorrect certificate information. Unfortunately, members of our team that work directly with Heartbeat are taking some time off for the holidays, but we will have someone follow up once they return.

dabo · December 23, 2021, 7:03am

Hi @Dominique_Clarke,
Certificate is not expired, this is the host: https://bamboo.fotoware.com/rest/api/latest/status
Please let me know if I can help to investigate why heartbeat is showing the wrong information.

Whenever they are back since this is not urgent. Have a nice holidays and happy Christmas.

Kind regards,
Dane

shahzad31 · January 3, 2022, 10:50am

Hello @dabo

Did you also check the certificates page in uptime. ? Does the new TLS certificate appears there? Or does it still displays the old certificate for that monitor.

Regards

dabo · January 3, 2022, 11:08am

Hello @shahzad31 ,

Certificate page shows monitors that are not expired. None of the threes below have expired certificate. But first two don't have it shown when you click on them, and third one "Bamboo" has it shown when you click on it, as written above in my first comment.

Kind regards,
Dane

shahzad31 · January 3, 2022, 4:19pm

Thank you for the information @dabo i think i have found the limitation/bug, it's an issue with query we are using, its using a collapse query inner_hits, which only fetches 3 results and also isn't sorted by timestamp.

github.com

elastic/kibana/blob/main/x-pack/plugins/uptime/common/requests/get_certs_request_body.ts#L134

    
      
            'monitor.name',
            'tls.server.x509.issuer.common_name',
            'tls.server.x509.subject.common_name',
            'tls.server.hash.sha1',
            'tls.server.hash.sha256',
            'tls.server.x509.not_after',
            'tls.server.x509.not_before',
          ],
          collapse: {
            field: 'tls.server.hash.sha256',
            inner_hits: {
              _source: {
                includes: ['monitor.id', 'monitor.name', 'url.full'],
              },
              collapse: {
                field: 'monitor.id',
              },
              name: 'monitors',
              sort: [{ 'monitor.id': 'asc' as const }],
            },
          },

I will raise the issue and we will try to fix it ASAP and hopefully it will make the next release.

Regards

dabo · January 4, 2022, 6:29am

Great thanks.
Is this bug also the reason why it is showing Expired in this monitor?

Kind regards,
Dane

shahzad31 · January 4, 2022, 2:23pm

Hello @dabo yes, it's hard to say the query can result in displaying the outdated certificate from previous pings. We are going to spend some time investigating it and see if we can reproduce it .

There seems to be two issues here, query can be definitely improve to display more related monitors for each certificate.

Will you be able to share some raw documents for each monitor, i meant where it display expired and where it doesn't.

And also share the relative masked heartbeat config. You can leverage discover/dev tools to get raw documents.

Regards

dabo · January 4, 2022, 2:55pm

Hi @shahzad31 ,
Sure but you need to be more specific or guide me.
I already shared results of one query from heartbeat index in this topic.

Do you need more from heartbeat-* index?
What specific fields are you interested in?

Kind regards,
Dane

shahzad31 · January 4, 2022, 3:45pm

Yes from the same index heartbeat-*, i think basic monitor fields monitor.id, @timestamp, monitor.timespan plus tls.* fields.

Though i will suggest running a clean setup with no security stuff, since it seems like the web links are public. So you can create heartbeat config , share those with us and also relative raw documents from heartbeat index. It will be helpful if you can share whole raw document. If it can't be masked, only basic fields will also help.

Regards

dabo · January 19, 2022, 11:09am

Hi, sorry for the long delay.

Please find attached some sample documents:

Monitor that is showing an as expired in TLS Cert page and in Monitors page:

{
  "took" : 1407,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : "heartbeat-7.15.0-2021.12.23-000004",
        "_type" : "_doc",
        "_id" : "h878cX4BAtKwMU-HPHxm",
        "_score" : null,
        "fields" : {
          "tls.version" : [
            "1.2"
          ],
          "tls.server.x509.subject.distinguished_name" : [
            "CN=*.fotoware.com"
          ],
          "tls.server.x509.serial_number" : [
            "265377785648398644839330295164343369846751"
          ],
          "tls.server.x509.issuer.common_name" : [
            "R3"
          ],
          "tls.server.x509.public_key_algorithm" : [
            "RSA"
          ],
          "tls.version_protocol" : [
            "tls"
          ],
          "monitor.timespan" : [
            {
              "lt" : "2022-01-19T10:58:40.499Z",
              "gte" : "2022-01-19T10:57:40.499Z"
            }
          ],
          "tls.cipher" : [
            "ECDHE-RSA-AES-128-GCM-SHA256"
          ],
          "tls.server.hash.sha1" : [
            "2ddd12867c8b5ba586333303adf56112efb1bc00"
          ],
          "monitor.id" : [
            "19"
          ],
          "tls.certificate_not_valid_before" : [
            "2021-12-06T12:55:11.000Z"
          ],
          "tls.certificate_not_valid_after" : [
            "2021-12-08T15:59:50.000Z"
          ],
          "tls.rtt.handshake.us" : [
            213858
          ],
          "@timestamp" : [
            "2022-01-19T10:57:39.983Z"
          ],
          "tls.server.x509.not_after" : [
            "2021-12-08T15:59:50.000Z"
          ],
          "tls.server.x509.subject.common_name" : [
            "*.fotoware.com"
          ],
          "tls.server.x509.not_before" : [
            "2021-12-06T12:55:11.000Z"
          ],
          "tls.server.x509.issuer.distinguished_name" : [
            "CN=R3,O=Let's Encrypt,C=US"
          ],
          "tls.established" : [
            true
          ],
          "tls.server.x509.public_key_exponent" : [
            65537
          ],
          "url.full" : [
            "https://bamboo.fotoware.com/rest/api/latest/status"
          ],
          "tls.server.hash.sha256" : [
            "a7c422cace90b1028d1e1e73864d1fb2441f1002f51f7b2ad22cfd5109f2f18a"
          ]
        },
        "sort" : [
          1642589859983
        ]
      }
    ]
  }
}

Monitor that is showing expired cert only in TLS Cert page:

{
  "took" : 239,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : "heartbeat-7.15.0-2021.12.23-000004",
        "_type" : "_doc",
        "_id" : "7yT-cX4BAopBheBaY4ep",
        "_score" : null,
        "fields" : {
          "tls.version" : [
            "1.2"
          ],
          "tls.server.x509.subject.distinguished_name" : [
            "CN=*.fotoware.com"
          ],
          "tls.server.x509.serial_number" : [
            "265377785648398644839330295164343369846751"
          ],
          "tls.server.x509.issuer.common_name" : [
            "R3"
          ],
          "tls.server.x509.public_key_algorithm" : [
            "RSA"
          ],
          "tls.version_protocol" : [
            "tls"
          ],
          "monitor.timespan" : [
            {
              "lt" : "2022-01-19T11:00:59.217Z",
              "gte" : "2022-01-19T10:59:59.217Z"
            }
          ],
          "tls.cipher" : [
            "ECDHE-RSA-AES-256-GCM-SHA384"
          ],
          "tls.server.hash.sha1" : [
            "2ddd12867c8b5ba586333303adf56112efb1bc00"
          ],
          "monitor.id" : [
            "14"
          ],
          "tls.certificate_not_valid_before" : [
            "2021-12-06T12:55:11.000Z"
          ],
          "tls.certificate_not_valid_after" : [
            "2022-03-06T12:55:10.000Z"
          ],
          "tls.rtt.handshake.us" : [
            1350813
          ],
          "@timestamp" : [
            "2022-01-19T10:59:56.726Z"
          ],
          "tls.server.x509.not_after" : [
            "2022-03-06T12:55:10.000Z"
          ],
          "tls.server.x509.subject.common_name" : [
            "*.fotoware.com"
          ],
          "tls.server.x509.not_before" : [
            "2021-12-06T12:55:11.000Z"
          ],
          "tls.server.x509.issuer.distinguished_name" : [
            "CN=R3,O=Let's Encrypt,C=US"
          ],
          "tls.established" : [
            true
          ],
          "tls.server.x509.public_key_exponent" : [
            65537
          ],
          "url.full" : [
            "https://trk.fotoware.com/ping"
          ],
          "tls.server.hash.sha256" : [
            "a7c422cace90b1028d1e1e73864d1fb2441f1002f51f7b2ad22cfd5109f2f18a"
          ]
        },
        "sort" : [
          1642589996726
        ]
      }
    ]
  }
}

Monitor that is fine:

{
  "took" : 247,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : "heartbeat-7.15.0-2021.12.23-000004",
        "_type" : "_doc",
        "_id" : "syUBcn4BAopBheBaFrgC",
        "_score" : null,
        "fields" : {
          "tls.version" : [
            "1.2"
          ],
          "tls.server.x509.subject.distinguished_name" : [
            "CN=fotoware.com"
          ],
          "tls.server.x509.serial_number" : [
            "298984076984006459550215663264375414305827"
          ],
          "tls.server.x509.issuer.common_name" : [
            "R3"
          ],
          "tls.server.x509.public_key_algorithm" : [
            "RSA"
          ],
          "tls.version_protocol" : [
            "tls"
          ],
          "monitor.timespan" : [
            {
              "lt" : "2022-01-19T11:03:58.963Z",
              "gte" : "2022-01-19T11:02:58.963Z"
            }
          ],
          "tls.cipher" : [
            "ECDHE-RSA-AES-256-GCM-SHA384"
          ],
          "tls.server.hash.sha1" : [
            "36ef4040a8b0be56eb7952c90ec94a66af419474"
          ],
          "monitor.id" : [
            "41"
          ],
          "tls.certificate_not_valid_before" : [
            "2021-12-06T12:54:05.000Z"
          ],
          "tls.certificate_not_valid_after" : [
            "2022-03-06T12:54:04.000Z"
          ],
          "tls.rtt.handshake.us" : [
            1430244
          ],
          "@timestamp" : [
            "2022-01-19T11:02:56.769Z"
          ],
          "tls.server.x509.not_after" : [
            "2022-03-06T12:54:04.000Z"
          ],
          "tls.server.x509.subject.common_name" : [
            "fotoware.com"
          ],
          "tls.server.x509.not_before" : [
            "2021-12-06T12:54:05.000Z"
          ],
          "tls.server.x509.issuer.distinguished_name" : [
            "CN=R3,O=Let's Encrypt,C=US"
          ],
          "tls.established" : [
            true
          ],
          "tls.server.x509.public_key_exponent" : [
            65537
          ],
          "url.full" : [
            "https://fotoware.com/"
          ],
          "tls.server.hash.sha256" : [
            "d522aaa56744ed01d8e5a45a539182f854954ed3d91c32ad18570d5b46930ec7"
          ]
        },
        "sort" : [
          1642590176769
        ]
      }
    ]
  }
}

I've excluded only 2 fields from tls.* fields.

We don't have possibility to set up a new test heartbeat config atm.
Hope this helps.

Kind regards,
Dane

system · January 19, 2022, 11:09am

7.1 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

system · February 12, 2022, 11:09am

This topic was automatically closed 24 days after the last reply. New replies are no longer allowed.