How to send heartbeats to Elasticsearch in a secured network?

Elastic Stack Logstash
1

Hey there everyone,.. really hoping I can find some help here.

I am working in an environment where Elasticsearch, Kibana and Logstash is running on machine A. Machine B has a Heartbeat installed and is collecting heartbeat information on Service 1. The problem here is that the network architecture is set up to allow for communications from machine B to machine A, but communications from machine A to B are not allowed for security reasons.

Sending Heartbeat messages on machine B directly Elasticsearch on machine A will not work in this network configuration. However, I had an idea to add Logstash on machine B and configure it to send heartbeats to Logstash on machine A via a UDP output. On machine A, the UDP input on Logstash will use the JSON filter plugin to change the heartbeat message back to JSON and send it to Elasticsearch. The problem is that the heartbeat JSON going to Elasticsearch is not the same as the heartbeat JSON being provided by Heartbeat sending information directly to Elasticsearch.

I feel like I am missing a step, or need to adjust the JSON to make it compatible.

Does anyone know what these missing steps are?

John

2

Can you elaborate on this? Are you using a data diode?

Typically if machine A can talk to machine B, it's stateful, and acks are allowed and thus the normal beats protocol should be used for machine A to talk to machine B.

Are you sure that UDP is the only allowed protocol between machine A and machine B? This would be an unusual setup even for secure enterprise environments. Unless you are using a data diode?

Can you share how the messages differ?

My initial guess is that the first logstash instance is removing the @metadata block when it outputs over UDP to the second one. So you may need a copy mutate filter on the first logstash instance that copy's @metadata to metadata_backup and a copy mutate on the destination logstash that restores metadata_backup to @metadata

3

if Machine B (heartbeat) can talk to Machine A (ELK), then what is the issue here? Heartbeat would be able to send data to Elasticsearch or Logstash.

4

I would say that this is correct, the @metadata field will not be part of the output message, they can be used to filter in the output block of a pipeline, but will not be present in the output message.

Also for communication between logstash instances, the recommendation seems to be to use the logstash input/output as described here.

5

What do you mean by that? Are you saying that B can initiate connections to A, but A cannot open a connection to B? Or are you saying that network packets cannot be sent from A to B at all?

1 Like
6

Thank you so much guys for your feedback and help!

That's what I initially thought, that the security node prevents traffic in one direction but does not interfere with communication handshaking like ACKS. But once the hardware Diode got put into place, the handshake ARP-related ACKs from A to B got shutdown,.. so no network traffic allowed from A to B... causing the HTTPS REST call from Heartbeat to Elasticsearch to fail miserably :frowning:

I like the @metadata proposal because it is the weirdest thing,.. Kibana->Observability shows the heartbeats coming in but doesn't show the info on the monitor itself (like stuff is missing).

I will try the @metadata suggestion,.. maybe that is the little piece of the puzzle that is missing :slight_smile:

Cheers,
John

1 Like
7

This is true but with him using a data diode only udp-based unidirectional protocols will suffice

1 Like
8

If you still have issues it would be useful to share a document json that does work how you expect and one that doesn't and we can help you brainstorm how to fix the remaining differences.

9

Hey there,.. so what I am seeing in heartbeats is the following.

2024-02-19 18 17 39
2024-02-19 18 17 391916×944 129 KB

So, I can see the monitors reporting that they are up but for some reason elasticsearch can't determine how many monitors there are.

Here is my logstash configuration..

For Machine B (which can send UDP data only to Machine A) this is what I have for logstash config.

input {    
  beats {
    port => 5044
    enrich => [all]
  }
}
filter {
  mutate {
    copy => {"@metadata" => "metadata_backup"}
  }
}
output {
  stdout {
    codec => rubydebug
  }
  udp {
      id => "udp_input_nifi_heartbeat"
      host => "2.8.9.133"
      port => 9757
  }
}

For Machine A which has elastricsearch and kibana, and cannot talk to machine B directly.. I have the following logstash config that takes the heartbeat data from UDP, converts it back to JSON and feeds it to elasticsearch.

input {
  udp {
    id => "udp_input_nifi_heartbeat"
    host => "2.8.9.133"
    port => 9757    
    type => heartbeat
  }
}
filter {
  json {
    source => "message"
  }
  mutate {
     copy => {"metadata_backup" => "@metadata"}
  }
  mutate {
    remove_field => ["message", "metadata_backup"]
  }
}
output {
  stdout {
    codec => rubydebug
  }  
  elasticsearch {
    hosts => ["https://2.8.9.133:9200"]
    ssl => true
    cacert => "/etc/logstash/certs/ca/ca.crt"
    ssl_certificate_verification => false
    action => "create"    
    manage_template => false
    index => "heartbeat-8.12.0-%{+YYYY.MM.dd}"
  }
}

I am hoping someone out there sees the stupid mistake I am making :slight_smile:

Cheers,
John

10

If you want the UDP message to be JSON then I think you have to specify codec => json on the output.

11

You have the rubydebug enabled which sends a copy of each message to the console.

Can you grab the same message from each logstash server and we can see if there are any differences between how the message leaves the first logstash server and how it leaves the second one?

12

Hey there, thank you so much for sticking with me on this problem! It is very much appreciated!

With debug enabled, Machine B's Logstash send out the following heartbeat message;

  {
                "monitor" => {
                   "type" => "http",
                   "name" => "Test Node Elasticsearch",
                     "ip" => "192.168.239.128",
               "timespan" => {
                 "lt" => "2024-02-21T16:03:59.688Z",
                "gte" => "2024-02-21T16:03:49.688Z"
            },
                 "status" => "up",
            "check_group" => "cd0abed7-d0d2-11ee-9833-000c291ba6dc-1",
                     "id" => "test-elasticsearch",
               "duration" => {
                "us" => 7622
            }
        },
                  "event" => {
            "dataset" => "http",
               "type" => "heartbeat/summary"
        },
                    "ecs" => {
            "version" => "8.0.0"
        },
                    "tcp" => {
            "rtt" => {
                "connect" => {
                    "us" => 257
                }
            }
        },
               "@version" => "1",
        "metadata_backup" => {
              "input" => {
                "beats" => {
                    "host" => {
                        "ip" => "127.0.0.1"
                    }
                }
            },
            "version" => "8.11.1",
               "beat" => "heartbeat",
               "type" => "_doc"
        },
                  "state" => {
                  "checks" => 16,
                    "down" => 0,
              "started_at" => "2024-02-21T11:01:19.771964557-05:00",
                    "ends" => nil,
             "duration_ms" => "149916",
                  "status" => "up",
                      "up" => 16,
                      "id" => "default-18dcc66679b-0",
            "flap_history" => []
        },
                "summary" => {
            "final_attempt" => true,
                     "down" => 0,
                   "status" => "up",
              "retry_group" => "cd0abed7-d0d2-11ee-9833-000c291ba6dc",
                       "up" => 1,
             "max_attempts" => 1,
                  "attempt" => 1
        },
               "observer" => {
                  "ip" => [
                [0] "192.168.239.128",
                [1] "fe80::20c:29ff:fe1b:a6dc"
            ],
                 "mac" => [
                [0] "00-0C-29-1B-A6-DC"
            ],
            "hostname" => "highsidemachine.com"
        },
                    "tls" => {
                                 "version" => "1.3",
                             "established" => true,
            "certificate_not_valid_before" => "2023-11-29T17:11:16.000Z",
                        "version_protocol" => "tls",
                                     "rtt" => {
                "handshake" => {
                    "us" => 5132
                }
            },
             "certificate_not_valid_after" => "2026-11-28T17:11:16.000Z",
                                  "cipher" => "TLS-AES-256-GCM-SHA384",
                                  "server" => {
                "x509" => {
                              "not_before" => "2023-11-29T17:11:16.000Z",
                                 "subject" => {
                               "common_name" => "elastic",
                        "distinguished_name" => "CN=elastic"
                    },
                               "not_after" => "2026-11-28T17:11:16.000Z",
                     "signature_algorithm" => "SHA256-RSA",
                         "public_key_size" => 2048,
                     "public_key_exponent" => 65537,
                           "serial_number" => "700267725123170265758521605690421630765307868547",
                    "public_key_algorithm" => "RSA",
                                  "issuer" => {
                               "common_name" => "Elastic Certificate Tool Autogenerated CA",
                        "distinguished_name" => "CN=Elastic Certificate Tool Autogenerated CA"
                    }
                },
                "hash" => {
                      "sha1" => "e612bbe877199e19a169ba1e87b0190d726ca541",
                    "sha256" => "102321c71fcb1d75696f99a6fa0d99cf94eb144d9bf4eaa7b032cd5773cfdc39"
                }
            }
        },
                   "tags" => [
            [0] "beats_input_raw_event"
        ],
             "@timestamp" => 2024-02-21T16:03:49.688Z,
                    "url" => {
            "scheme" => "https",
            "domain" => "192.168.239.128",
              "port" => 9200,
              "full" => "https://192.168.239.128:9200"
        },
                   "http" => {
            "response" => {
                       "body" => {
                     "hash" => "3923b0ba922ccdf6be2e5990c66867c9ff47103317e78d3424a538eebefbf79e",
                    "bytes" => 541
                },
                    "headers" => {
                       "Content-Length" => "541",
                    "X-Elastic-Product" => "Elasticsearch",
                         "Content-Type" => "application/json"
                },
                "status_code" => 200,
                  "mime_type" => "application/json"
            },
                 "rtt" => {
                       "validate" => {
                    "us" => 2022
                },
                  "write_request" => {
                    "us" => 31
                },
                          "total" => {
                    "us" => 7542
                },
                        "content" => {
                    "us" => 217
                },
                "response_header" => {
                    "us" => 1804
                }
            }
        },
                  "agent" => {
                    "type" => "heartbeat",
                 "version" => "8.11.1",
                    "name" => "highsidemachine.com",
            "ephemeral_id" => "715c32d1-7293-42fa-bed1-53a4ff9cbb53",
                      "id" => "d8856b2c-3236-4ec0-a906-590669af4242"
        }
    }

When Machine A's Logstash receives this heartbeat message, it reports the following;

 {
             "host" => {
           "ip" => "192.168.239.128"
       },
             "tags" => [
           [0] "beats_input_raw_event"
       ],
              "ecs" => {
           "version" => "8.0.0"
       },
             "http" => {
                "rtt" => {
                      "validate" => {
                   "us" => 2022
               },
                 "write_request" => {
                   "us" => 31
               },
                         "total" => {
                   "us" => 7542
               },
                       "content" => {
                   "us" => 217
               },
               "response_header" => {
                   "us" => 1804
               }
           },
           "response" => {
               "status_code" => 200,
                      "body" => {
                    "hash" => "3923b0ba922ccdf6be2e5990c66867c9ff47103317e78d3424a538eebefbf79e",
                   "bytes" => 541
               },
                 "mime_type" => "application/json",
                   "headers" => {
                   "X-Elastic-Product" => "Elasticsearch",
                        "Content-Type" => "application/json",
                      "Content-Length" => "541"
               }
           }
       },
              "tcp" => {
           "rtt" => {
               "connect" => {
                   "us" => 257
               }
           }
       },
            "state" => {
                 "checks" => 16,
           "flap_history" => [],
                     "up" => 16,
             "started_at" => "2024-02-21T11:01:19.771964557-05:00",
                   "ends" => nil,
                 "status" => "up",
                   "down" => 0,
                     "id" => "default-18dcc66679b-0",
            "duration_ms" => "149916"
       },
         "observer" => {
                 "ip" => [
               [0] "192.168.239.128",
               [1] "fe80::20c:29ff:fe1b:a6dc"
           ],
                "mac" => [
               [0] "00-0C-29-1B-A6-DC"
           ],
           "hostname" => "highsidemachine.com"
       },
             "type" => "heartbeat",
         "@version" => "1",
              "url" => {
           "scheme" => "https",
           "domain" => "192.168.239.128",
             "port" => 9200,
             "full" => "https://192.168.239.128:9200"
       },
            "event" => {
              "type" => "heartbeat/summary",
           "dataset" => "http"
       },
          "monitor" => {
                  "type" => "http",
                    "ip" => "192.168.239.128",
                  "name" => "Test Node Elasticsearch",
                "status" => "up",
           "check_group" => "cd0abed7-d0d2-11ee-9833-000c291ba6dc-1",
              "timespan" => {
               "gte" => "2024-02-21T16:03:49.688Z",
                "lt" => "2024-02-21T16:03:59.688Z"
           },
              "duration" => {
               "us" => 7622
           },
                    "id" => "test-elasticsearch"
       },
              "tls" => {
                                 "server" => {
               "x509" => {
                                 "issuer" => {
                       "distinguished_name" => "CN=Elastic Certificate Tool Autogenerated CA",
                              "common_name" => "Elastic Certificate Tool Autogenerated CA"
                   },
                    "public_key_exponent" => 65537,
                                "subject" => {
                       "distinguished_name" => "CN=elastic",
                              "common_name" => "elastic"
                   },
                             "not_before" => "2023-11-29T17:11:16.000Z",
                    "signature_algorithm" => "SHA256-RSA",
                          "serial_number" => "700267725123170265758521605690421630765307868547",
                              "not_after" => "2026-11-28T17:11:16.000Z",
                   "public_key_algorithm" => "RSA",
                        "public_key_size" => 2048
               },
               "hash" => {
                     "sha1" => "e612bbe877199e19a169ba1e87b0190d726ca541",
                   "sha256" => "102321c71fcb1d75696f99a6fa0d99cf94eb144d9bf4eaa7b032cd5773cfdc39"
               }
           },
           "certificate_not_valid_before" => "2023-11-29T17:11:16.000Z",
                                 "cipher" => "TLS-AES-256-GCM-SHA384",
                            "established" => true,
                                "version" => "1.3",
                       "version_protocol" => "tls",
                                    "rtt" => {
               "handshake" => {
                   "us" => 5132
               }
           },
            "certificate_not_valid_after" => "2026-11-28T17:11:16.000Z"
       },
            "agent" => {
                   "type" => "heartbeat",
                   "name" => "highsidemachine.com",
           "ephemeral_id" => "715c32d1-7293-42fa-bed1-53a4ff9cbb53",
                     "id" => "d8856b2c-3236-4ec0-a906-590669af4242",
                "version" => "8.11.1"
       },
       "@timestamp" => 2024-02-21T16:03:49.688Z,
          "summary" => {
             "retry_group" => "cd0abed7-d0d2-11ee-9833-000c291ba6dc",
                      "up" => 1,
                  "status" => "up",
           "final_attempt" => true,
                 "attempt" => 1,
                    "down" => 0,
            "max_attempts" => 1
       }
   }

This is with the Logstash configuration I mentioned before. The weird thing is that the outputs don't seem to line up.

13

After some investigation, I have some more information to share. I have simplified the problem down to the Beats input plugin for Logstash.

If I configure Heartbeat to send to Logstash and then configure Logstash to send directly to Elasticsearch,... the monitor information being shown in Kibana is messed up. I can see the Heartbeat messages reporting the one monitor I have is "up" but it doesn't know how to show the hostname that the monitor belongs too (this information seems to have been lost).

2024-02-21 17 02 36
2024-02-21 17 02 361872×583 49.5 KB

So, if I take away the UDP to UDP portion of the problem, the problem still exists. This is the config I have for Logstash

input {
  beats {
    port => 5044   
    enrich => [all] 
    codec => json_lines { 
      target => "[document]"
      ecs_compatibility => "v8"
    }
  }
}
filter {
  mutate {
    copy => {"@metadata" => "metadata_backup"}
  }
}
output {
  stdout {
    codec => rubydebug
  }
  if [@metadata][beat] == "heartbeat" {    
    elasticsearch {
      hosts => ["https://2.8.9.3:9200"]
      ssl => true
      cacert => "/etc/logstash/certs/ca/ca.crt"
      ssl_certificate_verification => false
      action => "create"
      user => "e"
      password => "z"
      ilm_enabled => true
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    }
  }
}

Has anyone seen this before? If I send from Heartbeat straight to Elasticsearch, everything shows up fine in Kibana,.. throwing Logstash in the mix is somehow messing things up. Anyone see what I am doing wrong in my settings????

Cheers,
John --- oh so confused :confused:

14

Can you send a heartbeat directly (one that works and shows up in the UI) and then one that doesn't and query the underlying index with via dev tools:

POST heartbeat-8*/_search
{}

And we can do a JSON diff of the working and nonworking document to identify the problem?

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.