This is regarding uptime app data compatibility in observability, We are using uptime app to monitor TLS certs expiry on Linux servers using heartbeat. however they are few servers we cannot install any agents due to OS compatibility like Unix, firmware like DataPower. Our leadership want to monitor all servers in single pane like uptime app instead of different dashboard.
As a workaround I created ingest pipeline to mimic data being sent by heartbeat by creating custom index with heartbeat document structure, templates, policies,.. etc. Ingest pipeline Data is reaching to heatbeat index same as linux instance heartbeat data document structure. however its not visible in uptime app. I am suspecting the data formatting issue here. any hints will be appreciated
curl -XPUT "http://qwerty123.containerhost:9244/_ingest/pipeline/todel_test_bloom" -H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
"description": "todel_test_bloom",
"processors": [
{
"set": {
"field": "static_template",
"value": "{ \"@metadata\": { \"beat\": \"heartbeat\", \"type\": \"_doc\", \"version\": \"8.8.1\" }, \"summary\": { \"up\": 1, \"down\": 0 }, \"state\": { \"started_at\": \"2024-03-09T16:51:58.000000000+05:30\", \"duration_ms\": \"0\", \"checks\": 1, \"up\": 1, \"down\": 0, \"id\": \"default-18d54eb3bec-2f\", \"status\": \"up\", \"flap_history\": [], \"ends\": { \"up\": 0, \"flap_history\": null, \"started_at\": \"2024-03-09T16:51:58.000+05:30\", \"duration_ms\": \"1\", \"status\": \"down\", \"ends\": null, \"id\": \"default-18d54eb3be0-2e\", \"checks\": 2, \"down\": 2 } }, \"observer\": { \"ip\": [ \"10.0.2.15\", \"fe80::a00:27ff:fe4f:af15\", \"192.168.58.1\", \"172.17.0.1\" ], \"mac\": [ \"02-42-4D-6B-64-38\", \"02-42-4D-89-3F-EB\", \"08-00-27-4F-AF-15\" ], \"hostname\": \"lab01\" }, \"tcp\": { \"rtt\": { \"connect\": { \"us\": 1660 } } }, \"tls\": { \"rtt\": { \"handshake\": { \"us\": 12134 } }, \"version_protocol\": \"tls\", \"version\": \"1.2\", \"cipher\": \"ECDHE-RSA-AES-256-GCM-SHA384\", \"server\": { \"x509\": { \"serial_number\": \"837969032333185907195096596728286312641407400400\", \"signature_algorithm\": \"SHA256-RSA\", \"not_before\": \"2024-01-25T05:38:07.000Z\", \"not_after\": \"2027-01-24T05:38:07.000Z\", \"public_key_size\": 4096, \"public_key_exponent\": 65537, \"issuer\": { \"common_name\": \"Elasticsearch security auto-configuration HTTP CA\", \"distinguished_name\": \"CN=Elasticsearch security auto-configuration HTTP CA\" }, \"subject\": { \"common_name\": \"lab01\", \"distinguished_name\": \"CN=lab01\" }, \"public_key_algorithm\": \"RSA\" }, \"hash\": { \"sha1\": \"ce6bcb292830ba01c76f789ac74496e8376774d2\", \"sha256\": \"9d47f3c3ad5d158f95d86f24efdadd6fdced1bb2b0458864d031d47c31f04df1\" } }, \"certificate_not_valid_before\": \"2024-01-25T05:38:07.000Z\", \"certificate_not_valid_after\": \"2027-01-24T05:38:07.000Z\", \"established\": true }, \"monitor\": { \"duration\": { \"us\": 14591 }, \"check_group\": \"355c93ff-be97-11ee-94aa-0800274faf15\", \"ip\": \"127.0.0.1\", \"timespan\": { \"gte\": \"2024-03-09T11:21:58.955Z\", \"lt\": \"2024-03-09T11:22:03.955Z\" }, \"name\": \"demostage01\", \"type\": \"tcp\", \"id\": \"demostage01-connective-ins01-61415635304cc8d\", \"status\": \"up\" }, \"resolve\": { \"ip\": \"127.0.0.1\", \"rtt\": { \"us\": 622 } }, \"url\": { \"domain\": \"demostagehost001\", \"port\": 6999, \"full\": \"ssl://demostagehost001:6999\", \"scheme\": \"ssl\" }, \"event\": { \"dataset\": \"tcp\" }, \"ecs\": { \"version\": \"8.0.0\" }, \"agent\": { \"ephemeral_id\": \"e3652ef1-88f9-4b78-8480-c9442427fa4e\", \"id\": \"8252dcca-41c7-47da-a58c-cfc3c7633924\", \"name\": \"lab01\", \"type\": \"heartbeat\", \"version\": \"8.8.1\" } }"
}
},
{
"json": {
"field": "static_template",
"add_to_root": true
}
},
{
"set": {
"copy_from": "temp_val.url_full",
"field": "url.full",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.url_domain",
"field": "url.domain",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.monitor_id",
"field": "url.id",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.monitor_gte",
"field": "monitor.timespan.gte",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.monitor_lt",
"field": "monitor.timespan.lt",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.monitor_name",
"field": "monitor.name",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.monitor_id",
"field": "monitor.id",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.cert_not_before",
"field": "tls.certificate_not_valid_before",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.cert_not_after",
"field": "tls.certificate_not_valid_after",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.cert_not_before",
"field": "tls.server.x509.not_before",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.cert_not_after",
"field": "tls.server.x509.not_after",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.started_at",
"field": "state.started_at",
"ignore_empty_value": true
}
},
{
"set": {
"copy_from": "temp_val.started_at",
"field": "state.ends.started_at",
"ignore_empty_value": true
}
},
{
"remove": {
"field": [
"temp_val",
"static_template"
]
}
},
{
"set": {
"field": "_source.@timestamp",
"value": "{{_ingest.timestamp}}"
}
}
]
}'
I will get the unix/datapower TLS cert details based on the temp_val json sent using curl cron jobs from source.
curl -XPOST -k -u abc:xyz"https://abc-ait.es.eu-west-1.aws.found.io:9243/heartbeat-datapower/_bulk" -H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{ "create":{ } }
{ "temp_val": { "url_full": "www.localhost001.com", "url_domain": "localhost001", "monitor_id": "localhost001-connective-ins01-61415635304cc8d", "monitor_gte": "2024-03-09T11:22:03.955Z", "monitor_lt": "2024-03-09T11:21:03.955Z", "monitor_name": "localhost001", "cert_not_before": "2024-01-27T11:38:07.000Z", "cert_not_after": "2099-01-27T05:38:07.000Z", "started_at": "2024-03-09T16:51:58.000+05:30" } }
'
heartbeat-datapower index is configured to use the above ingest pipeline as default one and applied mapping,template,policies... same as heartbeat 8.8.1 data stream.