Maps are not showing data if logstash is used

When i directly send data to Elasticsearch i am able to see the maps data in SIEM and Netflow geolocations. But if i send data through the Logstash i am missing data in SIEM maps and Netflow geoip locations.

To get data in maps when i used logstash, what option/plugin/configuration should I used?

Can you please refer any any document/blog/configuration/post ?

this is probably a better question to ask at SIEM forum but here’s a couple of pointer

  1. SIEM app uses predefined index pattern. filebeat-, winlogbeat- are some of them, but logstash-* is not one of them.
  2. if you ship logstash with default config, it won’t be included in SIEM app as it will use logstash-* as index pattern, hence it won’t show in the SIEM network map.
  3. for starters , add logstash-* index pattern to SIEM . here’s a reference on SIEM.
  4. also you need geoip plugin filter to do the geoip enrichment to be able to visualize logstash ingested logs in maps. see here for geoip documentation

Thank you Ptamba for prompt reply.
I tried by adding filebeat index(through logstash) pattern in SIEM settings, still not found the data in SIEM maps.

I will try Geo Plugin for Logstash and get back to you?

Thank you.

I tried below configuration but destination.geo.location data not found in kibanadiscovery. i am generating data using filebeat netflow module. I am seeing field "destination.ip" in kibana discovery with ip address and destination.geo.location is empty. But if i directly sent the data elasticsearch insetaed of logstash i am seeing destination.geo.location data in kibana discvery and maps. In index patter i am seeing destination.ip is IP. below is the default geo database found logstash logs. Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"}. Can you please help me to get geo location data?

input {
  beats {
    port => 5045
  }
}
filter {
 mutate {
       rename => {
           "host" => "hostnetflow"}}
geoip {
      source => "destination.ip"
    }
}
output {
  elasticsearch {
    hosts => ["http://10.252.10.76:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}
~~~~~~~~~~~~~~~~~~~~~~~~

if you want it to be in the destination.geo then you should update your geoip filter to

geoip {
      source => "destination.ip"
       target = > “[destination][geo]”
    }

if you don’t specify the target, by default the geo information will be placed under geoip. in your current setup, assuming you have no geo parser failure, you should have geoip.location populated

I updated the config as below, still no destination.geo.location data in kibana discvoery.

geoip {
      source => "destination.ip"
      target => "[destination][geo]"
      tag_on_failure => "_geoip_lookup_failure"
    }
}
output {
  elasticsearch {
    hosts => ["http://10.252.10.76:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}

can you change the output to stdout and show the output generated by logstash? also are you using role based access control? does your logstash user has privileges to write to filebeat-* indices?

Yes i have all permissions. below is the output.

{
        "service" => {
        "type" => "netflow"
    },
          "input" => {
        "type" => "netflow"
    },
         "source" => {
            "port" => 53,
              "ip" => "8.8.8.8",
        "locality" => "public",
         "packets" => 2,
           "bytes" => 346
    },
        "fileset" => {
        "name" => "log"
    },
        "netflow" => {
           "post_nat_destination_ipv4_address" => "xx.xx.xx.xx",
                         "protocol_identifier" => 17,
                         "source_ipv4_address" => "8.8.8.8",
                  "destination_transport_port" => 57272,
                "post_ip_diff_serv_code_point" => 255,
                              "application_id" => [
            [0] 20,
            [1] 0,
            [2] 0,
            [3] 48,
            [4] 68,
            [5] 0,
            [6] 0,
            [7] 0,
            [8] 0
        ],
                           "ingress_interface" => 3,
                      "flow_start_sys_up_time" => 1206986714,
                           "octet_delta_count" => 346,
                "post_nat_source_ipv4_address" => "0.0.0.0",
                      "post_octet_delta_count" => 346,
                             "flow_end_reason" => 0,
                          "packet_delta_count" => 2,
             "post_napt_source_transport_port" => 0,
                       "source_transport_port" => 53,
                     "post_packet_delta_count" => 2,
        "post_napt_destination_transport_port" => 57272,
                                    "exporter" => {
                  "address" => "192.168.252.13:3873",
                "source_id" => 4,
                  "version" => 9,
                "timestamp" => "2020-05-25T05:48:23.000Z",
            "uptime_millis" => 1207211134
        },
                            "egress_interface" => 59,
                                        "type" => "netflow_flow",
                    "destination_ipv4_address" => "10.252.10.10",
                           "forwarding_status" => 64,
                        "flow_end_sys_up_time" => 1207029624
    },
          "agent" => {
                "type" => "filebeat",
        "ephemeral_id" => "f1cc8358-cde0-44eb-a6cd-336cc4d33392",
            "hostname" => "xxxxxxx",
             "version" => "7.7.0",
                  "id" => "ee22c382-4c82-4026-b738-70164e05b6cf"
    },
       "@version" => "1",
       "observer" => {
        "ip" => "192.168.252.13"
    },
            "ecs" => {
        "version" => "1.5.0"
    },
     "@timestamp" => 2020-05-25T05:48:23.000Z,
           "flow" => {
              "id" => "MesTKQVGReI",
        "locality" => "public"
    },
        "network" => {
        "community_id" => "1:8f6hNbronq4wEqyCG7ESTdpb8Wk=",
         "iana_number" => 17,
             "packets" => 2,
           "direction" => "unknown",
               "bytes" => 346,
           "transport" => "udp"
    },
    "destination" => {
            "port" => 57272,
              "ip" => "10.252.10.10",
        "locality" => "private"
    },
          "event" => {
         "dataset" => "netflow.log",
            "kind" => "event",
           "start" => "2020-05-25T05:44:38.580Z",
          "module" => "netflow",
         "created" => "2020-05-25T05:48:23.000Z",
          "action" => "netflow_flow",
             "end" => "2020-05-25T05:45:21.490Z",
        "category" => "network_traffic",
        "duration" => 42910000000
    },
           "tags" => [
        [0] "ISSQFILE",
        [1] "INHY",
        [2] "beats_input_raw_event",
        [3] "_geoip_lookup_failure"
    ],
    "hostnetflow" => {
                   "os" => {
             "version" => "8 (Core)",
              "kernel" => "4.18.0-147.8.1.el8_1.x86_64",
            "codename" => "Core",
                "name" => "CentOS Linux",
            "platform" => "centos",
              "family" => "redhat"
        },
                 "name" => "xxxxxxx",
         "architecture" => "x86_64",
                   "ip" => [
            [0] "10.252.10.75",
            [1] "fe80::d4ee:d927:5185:8d0"
        ],
                  "mac" => [
            [0] "00:15:5d:10:0b:62"
        ],
             "hostname" => "xxxxxxx",
                   "id" => "e1cebd3d12bc4510bdecafd61726096c",
        "containerized" => false
    }
}
{
        "service" => {
        "type" => "netflow"
    },
          "input" => {
        "type" => "netflow"
    },
         "source" => {
            "port" => 58470,
              "ip" => "10.252.242.5",
        "locality" => "private",
         "packets" => 13,
           "bytes" => 6062
    },
        "fileset" => {
        "name" => "log"
    },
        "netflow" => {
           "post_nat_destination_ipv4_address" => "0.0.0.0",
                         "protocol_identifier" => 6,
                              "application_id" => [
            [0] 20,
            [1] 0,
            [2] 0,
            [3] 48,
            [4] 68,
            [5] 0,
            [6] 0,
            [7] 0,
            [8] 0
        ],
                  "destination_transport_port" => 443,
                "post_ip_diff_serv_code_point" => 255,
                         "source_ipv4_address" => "10.252.242.5",
                           "ingress_interface" => 59,
                      "flow_start_sys_up_time" => 1207213404,
                           "octet_delta_count" => 6062,
                "post_nat_source_ipv4_address" => "xx.xx.xx.xx",
                      "post_octet_delta_count" => 6062,
                             "flow_end_reason" => 3,
                          "packet_delta_count" => 13,
             "post_napt_source_transport_port" => 58470,
                            "egress_interface" => 3,
        "post_napt_destination_transport_port" => 0,
                     "post_packet_delta_count" => 13,
                       "source_transport_port" => 58470,
                                        "type" => "netflow_flow",
                                    "exporter" => {
                  "address" => "192.168.252.13:3873",
                "source_id" => 4,
                  "version" => 9,
                "timestamp" => "2020-05-25T05:48:28.000Z",
            "uptime_millis" => 1207216274
        },
                    "destination_ipv4_address" => "138.91.140.216",
                           "forwarding_status" => 64,
                        "flow_end_sys_up_time" => 1207215234
    },
          "agent" => {
                "type" => "filebeat",
        "ephemeral_id" => "f1cc8358-cde0-44eb-a6cd-336cc4d33392",
            "hostname" => "xxxxxxx",
             "version" => "7.7.0",
                  "id" => "ee22c382-4c82-4026-b738-70164e05b6cf"
    },
       "@version" => "1",
       "observer" => {
        "ip" => "192.168.252.13"
    },
            "ecs" => {
        "version" => "1.5.0"
    },
     "@timestamp" => 2020-05-25T05:48:28.000Z,
           "flow" => {
              "id" => "hY_1pkddS_o",
        "locality" => "public"
    },
        "network" => {
         "iana_number" => 6,
        "community_id" => "1:HyX3xWIqJYOix5Ha7Kwd1nJnTZA=",
             "packets" => 13,
           "direction" => "unknown",
               "bytes" => 6062,
           "transport" => "tcp"
    },
          "event" => {
         "dataset" => "netflow.log",
            "kind" => "event",
          "module" => "netflow",
           "start" => "2020-05-25T05:48:25.130Z",
         "created" => "2020-05-25T05:48:28.000Z",
          "action" => "netflow_flow",
             "end" => "2020-05-25T05:48:26.960Z",
        "category" => "network_traffic",
        "duration" => 1830000000
    },
    "destination" => {
            "port" => 443,
              "ip" => "138.91.140.216",
        "locality" => "public"
    },
           "tags" => [
        [0] "ISSQFILE",
        [1] "INHY",
        [2] "beats_input_raw_event",
        [3] "_geoip_lookup_failure"
    ],
    "hostnetflow" => {
                   "os" => {
             "version" => "8 (Core)",
              "kernel" => "4.18.0-147.8.1.el8_1.x86_64",
            "codename" => "Core",
                "name" => "CentOS Linux",
            "platform" => "centos",
              "family" => "redhat"
        },
                 "name" => "xxxxxxx",
         "architecture" => "x86_64",
                   "ip" => [
            [0] "10.252.10.75",
            [1] "fe80::d4ee:d927:5185:8d0"
        ],
             "hostname" => "xxxxxxx",
                  "mac" => [
            [0] "00:15:5d:10:0b:62"
        ],
                   "id" => "e1cebd3d12bc4510bdecafd61726096c",
        "containerized" => false
    }
}
```````````````````````````````````````````

as you can see here, the destination object is not populated with geo information and there's a lookup-failure tag. there should be an error related to geoip filter in the logstash log.

you might want to update to this:

geoip {
       source => "[destination][ip]"
       target = > “[destination][geo]”
    }

Thank you Ptamba. I am seeing source and destiation locations on Maps. I need another option how do i get connecting lines between source and destianation on Maps?

Kibana forum will probably be a more appropriate location to ask for visualization related questions :blush:

:wave:

In Elastic Maps you have to use the Point to Point layer type to draw lines that connect points on your documents. Here a quick video using Kibana Sample Flights dataset.

Peek 2020-05-26 12-02

Thank you Jsanz