When i directly send data to Elasticsearch i am able to see the maps data in SIEM and Netflow geolocations. But if i send data through the Logstash i am missing data in SIEM maps and Netflow geoip locations.
To get data in maps when i used logstash, what option/plugin/configuration should I used?
Can you please refer any any document/blog/configuration/post ?
this is probably a better question to ask at SIEM forum but here’s a couple of pointer
Thank you Ptamba for prompt reply.
I tried by adding filebeat index(through logstash) pattern in SIEM settings, still not found the data in SIEM maps.
I will try Geo Plugin for Logstash and get back to you?
Thank you.
I tried below configuration but destination.geo.location data not found in kibanadiscovery. i am generating data using filebeat netflow module. I am seeing field "destination.ip" in kibana discovery with ip address and destination.geo.location is empty. But if i directly sent the data elasticsearch insetaed of logstash i am seeing destination.geo.location data in kibana discvery and maps. In index patter i am seeing destination.ip is IP. below is the default geo database found logstash logs. Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"}. Can you please help me to get geo location data?
input {
beats {
port => 5045
}
}
filter {
mutate {
rename => {
"host" => "hostnetflow"}}
geoip {
source => "destination.ip"
}
}
output {
elasticsearch {
hosts => ["http://10.252.10.76:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
}
}
~~~~~~~~~~~~~~~~~~~~~~~~if you want it to be in the destination.geo then you should update your geoip filter to
geoip {
source => "destination.ip"
target = > “[destination][geo]”
}
if you don’t specify the target, by default the geo information will be placed under geoip. in your current setup, assuming you have no geo parser failure, you should have geoip.location populated
I updated the config as below, still no destination.geo.location data in kibana discvoery.
geoip {
source => "destination.ip"
target => "[destination][geo]"
tag_on_failure => "_geoip_lookup_failure"
}
}
output {
elasticsearch {
hosts => ["http://10.252.10.76:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
}
}
can you change the output to stdout and show the output generated by logstash? also are you using role based access control? does your logstash user has privileges to write to filebeat-* indices?
Yes i have all permissions. below is the output.
{
"service" => {
"type" => "netflow"
},
"input" => {
"type" => "netflow"
},
"source" => {
"port" => 53,
"ip" => "8.8.8.8",
"locality" => "public",
"packets" => 2,
"bytes" => 346
},
"fileset" => {
"name" => "log"
},
"netflow" => {
"post_nat_destination_ipv4_address" => "xx.xx.xx.xx",
"protocol_identifier" => 17,
"source_ipv4_address" => "8.8.8.8",
"destination_transport_port" => 57272,
"post_ip_diff_serv_code_point" => 255,
"application_id" => [
[0] 20,
[1] 0,
[2] 0,
[3] 48,
[4] 68,
[5] 0,
[6] 0,
[7] 0,
[8] 0
],
"ingress_interface" => 3,
"flow_start_sys_up_time" => 1206986714,
"octet_delta_count" => 346,
"post_nat_source_ipv4_address" => "0.0.0.0",
"post_octet_delta_count" => 346,
"flow_end_reason" => 0,
"packet_delta_count" => 2,
"post_napt_source_transport_port" => 0,
"source_transport_port" => 53,
"post_packet_delta_count" => 2,
"post_napt_destination_transport_port" => 57272,
"exporter" => {
"address" => "192.168.252.13:3873",
"source_id" => 4,
"version" => 9,
"timestamp" => "2020-05-25T05:48:23.000Z",
"uptime_millis" => 1207211134
},
"egress_interface" => 59,
"type" => "netflow_flow",
"destination_ipv4_address" => "10.252.10.10",
"forwarding_status" => 64,
"flow_end_sys_up_time" => 1207029624
},
"agent" => {
"type" => "filebeat",
"ephemeral_id" => "f1cc8358-cde0-44eb-a6cd-336cc4d33392",
"hostname" => "xxxxxxx",
"version" => "7.7.0",
"id" => "ee22c382-4c82-4026-b738-70164e05b6cf"
},
"@version" => "1",
"observer" => {
"ip" => "192.168.252.13"
},
"ecs" => {
"version" => "1.5.0"
},
"@timestamp" => 2020-05-25T05:48:23.000Z,
"flow" => {
"id" => "MesTKQVGReI",
"locality" => "public"
},
"network" => {
"community_id" => "1:8f6hNbronq4wEqyCG7ESTdpb8Wk=",
"iana_number" => 17,
"packets" => 2,
"direction" => "unknown",
"bytes" => 346,
"transport" => "udp"
},
"destination" => {
"port" => 57272,
"ip" => "10.252.10.10",
"locality" => "private"
},
"event" => {
"dataset" => "netflow.log",
"kind" => "event",
"start" => "2020-05-25T05:44:38.580Z",
"module" => "netflow",
"created" => "2020-05-25T05:48:23.000Z",
"action" => "netflow_flow",
"end" => "2020-05-25T05:45:21.490Z",
"category" => "network_traffic",
"duration" => 42910000000
},
"tags" => [
[0] "ISSQFILE",
[1] "INHY",
[2] "beats_input_raw_event",
[3] "_geoip_lookup_failure"
],
"hostnetflow" => {
"os" => {
"version" => "8 (Core)",
"kernel" => "4.18.0-147.8.1.el8_1.x86_64",
"codename" => "Core",
"name" => "CentOS Linux",
"platform" => "centos",
"family" => "redhat"
},
"name" => "xxxxxxx",
"architecture" => "x86_64",
"ip" => [
[0] "10.252.10.75",
[1] "fe80::d4ee:d927:5185:8d0"
],
"mac" => [
[0] "00:15:5d:10:0b:62"
],
"hostname" => "xxxxxxx",
"id" => "e1cebd3d12bc4510bdecafd61726096c",
"containerized" => false
}
}
{
"service" => {
"type" => "netflow"
},
"input" => {
"type" => "netflow"
},
"source" => {
"port" => 58470,
"ip" => "10.252.242.5",
"locality" => "private",
"packets" => 13,
"bytes" => 6062
},
"fileset" => {
"name" => "log"
},
"netflow" => {
"post_nat_destination_ipv4_address" => "0.0.0.0",
"protocol_identifier" => 6,
"application_id" => [
[0] 20,
[1] 0,
[2] 0,
[3] 48,
[4] 68,
[5] 0,
[6] 0,
[7] 0,
[8] 0
],
"destination_transport_port" => 443,
"post_ip_diff_serv_code_point" => 255,
"source_ipv4_address" => "10.252.242.5",
"ingress_interface" => 59,
"flow_start_sys_up_time" => 1207213404,
"octet_delta_count" => 6062,
"post_nat_source_ipv4_address" => "xx.xx.xx.xx",
"post_octet_delta_count" => 6062,
"flow_end_reason" => 3,
"packet_delta_count" => 13,
"post_napt_source_transport_port" => 58470,
"egress_interface" => 3,
"post_napt_destination_transport_port" => 0,
"post_packet_delta_count" => 13,
"source_transport_port" => 58470,
"type" => "netflow_flow",
"exporter" => {
"address" => "192.168.252.13:3873",
"source_id" => 4,
"version" => 9,
"timestamp" => "2020-05-25T05:48:28.000Z",
"uptime_millis" => 1207216274
},
"destination_ipv4_address" => "138.91.140.216",
"forwarding_status" => 64,
"flow_end_sys_up_time" => 1207215234
},
"agent" => {
"type" => "filebeat",
"ephemeral_id" => "f1cc8358-cde0-44eb-a6cd-336cc4d33392",
"hostname" => "xxxxxxx",
"version" => "7.7.0",
"id" => "ee22c382-4c82-4026-b738-70164e05b6cf"
},
"@version" => "1",
"observer" => {
"ip" => "192.168.252.13"
},
"ecs" => {
"version" => "1.5.0"
},
"@timestamp" => 2020-05-25T05:48:28.000Z,
"flow" => {
"id" => "hY_1pkddS_o",
"locality" => "public"
},
"network" => {
"iana_number" => 6,
"community_id" => "1:HyX3xWIqJYOix5Ha7Kwd1nJnTZA=",
"packets" => 13,
"direction" => "unknown",
"bytes" => 6062,
"transport" => "tcp"
},
"event" => {
"dataset" => "netflow.log",
"kind" => "event",
"module" => "netflow",
"start" => "2020-05-25T05:48:25.130Z",
"created" => "2020-05-25T05:48:28.000Z",
"action" => "netflow_flow",
"end" => "2020-05-25T05:48:26.960Z",
"category" => "network_traffic",
"duration" => 1830000000
},
"destination" => {
"port" => 443,
"ip" => "138.91.140.216",
"locality" => "public"
},
"tags" => [
[0] "ISSQFILE",
[1] "INHY",
[2] "beats_input_raw_event",
[3] "_geoip_lookup_failure"
],
"hostnetflow" => {
"os" => {
"version" => "8 (Core)",
"kernel" => "4.18.0-147.8.1.el8_1.x86_64",
"codename" => "Core",
"name" => "CentOS Linux",
"platform" => "centos",
"family" => "redhat"
},
"name" => "xxxxxxx",
"architecture" => "x86_64",
"ip" => [
[0] "10.252.10.75",
[1] "fe80::d4ee:d927:5185:8d0"
],
"hostname" => "xxxxxxx",
"mac" => [
[0] "00:15:5d:10:0b:62"
],
"id" => "e1cebd3d12bc4510bdecafd61726096c",
"containerized" => false
}
}
```````````````````````````````````````````as you can see here, the destination object is not populated with geo information and there's a lookup-failure tag. there should be an error related to geoip filter in the logstash log.
you might want to update to this:
geoip {
source => "[destination][ip]"
target = > “[destination][geo]”
}
Thank you Ptamba. I am seeing source and destiation locations on Maps. I need another option how do i get connecting lines between source and destianation on Maps?
Kibana forum will probably be a more appropriate location to ask for visualization related questions 
In Elastic Maps you have to use the Point to Point layer type to draw lines that connect points on your documents. Here a quick video using Kibana Sample Flights dataset.
Thank you Jsanz
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.