GEOIP Enable


(Maile Halatuituia) #1

Hi
Firstly i am apologise for bugging the forum but it is just that i really want this to work. I have setup the stack using this link https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-16-04
and it is work perfectly. Also they have another tutorial for Apache and Nginx geoip filter . But in my case i am clooecting logs from my dns instead of the web server.
I already have the client ip , server ip , etc etc on my kibana but what i am still strugglling with is to convert that ip address to counttry code etc etc .. i do really appreciate if anyone has some kind if working exaample . I hopwe this make sense and really appreciate


Apache logs not able to map geo IPs
(Magnus Bäck) #2

Please show an example event. Have you tried using the geoip filter?


(Maile Halatuituia) #3

No Magnus ....
Here is what i have on my filter

geoip {
source => "ip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
but nothing happen .. Logstash operate normally but coordinate no shown ... i am not sure this is correct or not.


(Maile Halatuituia) #4

No Compatible Fields: The "packetbeat-*" index pattern does not contain any of the following field types: geo_point

this is the error when try to create tile map


(Magnus Bäck) #5

but nothing happen .. Logstash operate normally but coordinate no shown ... i am not sure this is correct or not.

Again, please show an example event. Either use a stdout { codec => rubydebug } output or copy/paste a JSON snippet from the JSON tab in Kibana.

No Compatible Fields: The "packetbeat-*" index pattern does not contain any of the following field types: geo_point

And you've configured Logstash to store the events in packetbeat-* indexes? What do the mappings for such an index look like? Use Elasticsearch's get mapping API to find out.


(Maile Halatuituia) #6

transport:udp method:QUERY server: client_ip:202.134.26.36 client_port:40,251 client_proc: status:OK bytes_out:184 responsetime:0 query:class IN, type A, www.google.com count:1 ip:202.134.24.110 @timestamp:February 8th 2017, 20:26:19.184 type:dns direction:in bytes_in:32 beat.hostname:ns2.kalianet.to beat.name:ns2.kalianet.to port:53 dns.additionals:{ "class": "IN", "data": "216.239.32.10", "name": "ns1.google.com", "ttl": 121534, "type": "A" }, { "class": "IN", "data": "216.239.34.10", "name": "ns2.google.com", "ttl": 121534, "type": "A" }, { "class": "IN", "data": "216.239.36.10", "name": "ns3.google.com", "ttl": 121534, "type": "A" }, { "class": "IN", "data": "216.239.38.10", "name": "ns4.google.com", "ttl": 121534, "type": "A" } dns.additionals_count:4 dns.answers:{ "class": "IN", "data": "172.217.25.36", "name": "www.google.com", "ttl": 268, "type": "A" } dns.answers_count:1 dns.authorities:{ "class": "IN", "data": "ns3.google.com", "name": "google.com", "ttl": 118299, "type": "NS" }, { "class": "IN", "data": "ns4.google.com", "name": "google.com", "ttl": 118299, "type": "NS" }, { "class": "IN", "data": "ns1.google.com", "name": "google.com", "ttl": 118299, "type": "NS" }, { "class": "IN", "data": "ns2.google.com", "name": "google.com", "ttl": 118299, "type": "NS" } dns.authorities_count:4 dns.flags.authoritative:false dns.flags.recursion_allowed:true dns.flags.recursion_desired:true dns.flags.truncated_response:false dns.id:29,482 dns.op_code:QUERY dns.question.class:IN dns.question.name:www.google.com dns.question.type:A dns.response_code:NOERROR resource:www.google.com client_server: proc: @version:1 host:ns2.kalianet.to tags:beats_input_raw_event _id:AVocpq2K1VZH1ycwIW15 _type:dns _index:packetbeat-2017.02.08 _score:


(Maile Halatuituia) #7

is this what you meant ...


(Magnus Bäck) #8

is this what you meant ...

No, but I think it's good enough in this case. It looks the geoip filter isn't able to look up 202.134.24.110. Is there anything about this in the logs? What happens if you use the default geoip database (i.e. comment out the database option)? With the default database I'm certainly able to look up the address:

$ cat test.config 
input { stdin { codec => plain } }
output { stdout { codec => rubydebug } }
filter {
  geoip {
    source => "message"
  }
}
$ echo 202.134.24.110 | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
{
       "message" => "202.134.24.110",
      "@version" => "1",
    "@timestamp" => "2017-02-08T08:36:23.215Z",
          "host" => "lnxolofon",
         "geoip" => {
                    "ip" => "202.134.24.110",
         "country_code2" => "TO",
         "country_code3" => "TON",
          "country_name" => "Tonga",
        "continent_code" => "OC",
              "latitude" => -20.0,
             "longitude" => -175.0,
              "timezone" => "Pacific/Tongatapu",
              "location" => [
            [0] -175.0,
            [1] -20.0
        ]
    }
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

(Maile Halatuituia) #9

sorry magnus but i think i am not followed you .... i have remove the databases option. here is how it looks like now in my filter

geoip {
source => "ip"
target => "geoip"

database => "/etc/logstash/GeoLiteCity.dat"

      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }

is it what you suggest ... should i add extra code etc etc


(Magnus Bäck) #10

You should look in your Logstash log to see if there are clues about why the geoip filter is failing. Please also post all of your configuration. Format it as preformatted text using the </> toolbar button.


(Maile Halatuituia) #11

input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
geoip {
source => "ip"
target => "geoip"

database => "/etc/logstash/GeoLiteCity.dat"

      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }
date {
  match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}

}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Here is my logstash config


(Magnus Bäck) #12

I asked you to "format it as preformatted text using the </> toolbar button", yet you didn't. Why?

The problem is that you're only applying the geoip filter to events with the syslog type but your Packetbeat have another type.


(Maile Halatuituia) #13

sorry magnus but maybe i am not following you ... apologise for my misunderstand .. please could you let me know how to format as you mention here


(Magnus Bäck) #14

Don't you have a toolbar just above the text area where you're typing your text?


(Maile Halatuituia) #15

i don't think so ... i am using the older version i guess as i could not install the latest version ... i think i am using kibana version 4


(Magnus Bäck) #16

I am talking about https://discuss.elastic.co where you just typed a couple of sentences.


(Maile Halatuituia) #17

sorry for the toolbar i did not get it you meant for my cut and paste here ...
anyway ... any chance what can i do to find which type my packetbeat has ??


(Magnus Bäck) #18

Just look at the event's type field in Kibana.


(Maile Halatuituia) #19

sorry for that ... now i can see it ... how about the packetbeat evet type


(Maile Halatuituia) #20

it say dns ...