Can't get a field type IP

Elastic Stack Logstash
1

I'm trying to get an IP field so I can then geoip it. I've looked at the blog posts and other sites, and tried everything, but still no working IP fields. I looked at the internal blog posts, and some seem to minimize the need for a working solution on this. Please let me assure you, and I think you already know this, that getting IP addresses into IP fields (instead of strings) is very important for many of us! There were concerns about performance, but if you could do a mutate / convert on the field, that might minimize the performance issues for the masses, and allow IP fields for those that really need it.

In any case, here's what I'm doing, and I'm hoping someone could advise on why it's not working. I should mention I am doing this work on Windows 10.

From the logstash .conf file (yes, I have custom patterns):
filter {
grok {
match => { "message" => "%{HEADER:header} %{TIMESTAMP_ISO8601:syslog_timestamp} %{IP:h_hostname} %{REALM:h_realm} %{NUMBER:unknown1} %{IDNUM:id}" }
}
kv {}
geoip {
source => "h_hostname"
}
}

output {
elasticsearch { hosts => ["localhost:9200"] template => "C:\logstash\sa_elasticsearch-template.json" template_overwrite => true }
stdout {}
}

From the custom template .json:
"properties" : {
"@timestamp": { "type": "date", "doc_values" : true },
"@version": { "type": "string", "index": "not_analyzed", "doc_values" : true },
"ip": { "type": "ip"},
"geoip" : {
"type" : "object",
"dynamic": true,
"properties" : {
"ip": { "type": "ip", "doc_values" : true },
"location" : { "type" : "geo_point", "doc_values" : true },
"latitude" : { "type" : "float", "doc_values" : true },
"longitude" : { "type" : "float", "doc_values" : true }
}
}

As can be seen, I modified the default template to include an IP definition, but I'm still seeing my IP fields as strings, and not seeing any geoip fields. Any help would be appreciated. Thanks!

2

Does the h_hostname field actually contain an IP address? The field name indicates that it's a hostname. Secondly, your index template does indeed map the ip and geoip.ip fields as the ip type, but there's no mention of h_hostname.

If you show an example message it'll be easier to give specific help.

3

Yes, the field h_hostname contains valid IP addresses. I have the IP address in two different places in the message, one of which is a Key Value Pair (HostName=). The other is parsed from the message text. A sample message is:

<86>1 2016-03-21T18:28:06.224Z 172.16.18.52 Name0 5136 ID90040 [Name@27389 RequestDuration="00:00:00.6405099" UserAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2682.0 Safari/537.36" UserHostAddress="192.168.2.126" RequestID="579a628d-b607-417c-aa37-7d3372d14f1c" Realm="Name0" Appliance="qaportal2.goname.com" Company="Name OC1 QA" Version="8.2.0.57" PEN="27389" HostName="172.16.18.52" Category="AUDIT" Priority="3" EventID="90040"] 00:00:00.6405099

Thanks for the help!

4

Okay. Well, as I said there's nothing in the index template that maps h_hostname as an IP address and the IP addresses in your example are private RFC1918 addresses for which GeoIP lookups can't possibly work.

5

So are you saying that the line in the template json should read "h_hostname": { "type": "ip"}, or are you saying that nothing I can do will make this work?

6

The former. The index template tells ES how to map each named field, so the field names in the template must match the actual names of the fields.

7

Thanks for your reply. I changed the line in question to read "h_hostname": { "type": "ip"},
but I'm still not getting the geoip fields, and Elasticsearch / kibana are showing:

Any thoughts on how I can proceed? Thanks...

8

That screenshot doesn't say anything about how the field is mapped. Use the get mapping API for that. But you pushed a new index template and created a new index that you populated with data, yes?

Regarding the geoip fields they will, again, never be populated for IP addresses in the private RFC1918 namespace (like 172.16.18.52).

9

Hi,

Mapping didn't return anything, which seems strange:
C:\logstash>curl -XGET localhost:9200/logstash-2016.03.24/_mapping/h_hostname,HostName,t_hostname
{}
C:\logstash>

Yet in Kibana, the fields are present:

This IP maps to Caracas, VE

I have even tried forcing an IP, trying to get geoip to work, for example:
mutate {
add_field => { "h_hostname" => "179.44.23.35" }
add_field => { "HostName" => "179.44.23.35" }
}
grok {
match => { "message" => "%{HEADER:header} %{TIMESTAMP_ISO8601:syslog_timestamp} %{IP:t_hostname} %{REALM:h_realm} %{NUMBER:unknown1} %{IDNUM:id} %{GREEDYDATA:remainline}" }
}
kv {}
geoip {
source => "h_hostname"
}
Does not work. Yet when I download a logstash example database and the sample.conf, it does this:
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
... and it works fine, I see the geoip fields. When I looked into how COMBINEDAPACHELOG is built, I found:
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
HTTPD20_ERRORLOG [%{HTTPDERROR_DATE:timestamp}] [%{LOGLEVEL:loglevel}] (?:[client %{IPORHOST:clientip}] ){0,1}%{GREEDYDATA:errormsg}
HTTPD24_ERRORLOG [%{HTTPDERROR_DATE:timestamp}] [%{WORD:module}:%{LOGLEVEL:loglevel}] [pid %{POSINT:pid}:tid %{NUMBER:tid}]( (%{POSINT:proxy_errorcode})%{DATA:proxy_errormessage}:)?( [client %{IPORHOST:client}:%{POSINT:clientport}])? %{DATA:errorcode}: %{GREEDYDATA:message}

which uses IPORHOST, not IP for the clientip field. IPORHOST looks like "IPORHOST (?:%{IP}|%{HOSTNAME})", so it does boil-down to IP.

Sorry about the private IP, there are a few in my database. I'm looking at valid non-private IPs for the geoip fields.

After spending several hours trying to get this to work, I'm grasping at straws. I've seen geoip work with the sample database, so I know its not the software per se, but somehow my .conf doesn't seem to work. Can you see anything I'm missing? Is it possible that since one of the key-value pair field names is HostName, could that be interfering with the HOSTNAME portion of IPORHOST, causing geoip to fail?

Thanks again for the time you've spent on this...

10

P.S. - I just commented-out the kv filter, and it had no effect, I still cannot see the geoip fields.

11

I don't think the GeoIP database that ships with Logstash contains a mapping for 179.44.23.35. I tried it myself with this very simple configuration and it's a no-go:

$ cat test.config
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  geoip {
    source => "message"
  }
}
$ echo 179.44.23.35 | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 2
Logstash startup completed
{
       "message" => "179.44.23.35",
      "@version" => "1",
    "@timestamp" => "2016-03-24T18:50:40.698Z",
          "host" => "hallonet"
}
Logstash shutdown completed

But if I try the same configuration with another address it works fine:

$ echo 195.60.68.50 | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 2
Logstash startup completed
{
       "message" => "195.60.68.50",
      "@version" => "1",
    "@timestamp" => "2016-03-24T18:51:42.485Z",
          "host" => "hallonet",
         "geoip" => {
                      "ip" => "195.60.68.50",
           "country_code2" => "SE",
           "country_code3" => "SWE",
            "country_name" => "Sweden",
          "continent_code" => "EU",
             "region_name" => "27",
               "city_name" => "Lund",
                "latitude" => 55.69999999999999,
               "longitude" => 13.183300000000003,
                "timezone" => "Europe/Stockholm",
        "real_region_name" => "Skane Lan",
                "location" => [
            [0] 13.183300000000003,
            [1] 55.69999999999999
        ]
    }
}
Logstash shutdown completed

which uses IPORHOST, not IP for the clientip field. IPORHOST looks like "IPORHOST (?:%{IP}|%{HOSTNAME})", so it does boil-down to IP.

It doesn't matter what grok pattern (if any) that's used to populate the GeoIP source field.

12

Sure enough, I used the same IP you used, and it worked! I had previously tried my IP address at home and at the office with the same result (Huntington Beach, California, USA), but if the IP isn't in the database provided, you just aren't going to see the geoip data. All that trouble because of a set of obscure IP address... Thank you for your patience through this!

I found this post on updating the geoip database:

So I'll update the database and maybe my obscure IP address will be found.

Thanks again!