Please help to grok my TMG Logs. How to use GeoIP?

Elastic Stack Logstash
21

I missed the forest for the trees...
Working, thank you!

Now I have one last problem with geoip. I have added it to my config:

geoip {
      source => "clientip"
      target => "geoip"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
       remove_field => ["year", "month", "day", "time", "hour", "minute", "second", "bin", "rest"]
       convert => [ "[geoip][coordinates]", "float"]
    }

When I know want to create a tile map it says:

No Compatible Fields: The "filebeat-*" index pattern does not contain any of the following field types: geo_point

Do you have any idea?

This is my JSON:

{
  "_index": "filebeat-2016.10.04",
  "_type": "log",
  "_id": "AVePLEGJmaRlHgp4ACvq",
  "_score": null,
  "_source": {
    "message": "10.10.10.10\tdomain\\user\tMicrosoft Office/15.0 (Windows NT 10.0; Microsoft Outlook 15.0.4859; Pro)\t2016-10-04\t10:10:04\tTMG\t-\tautodiscover.domain.com\t192.168.100.25\t443\t47\t879\t4429\thttps\tPOST\thttp://autodiscover.domain.com/autodiscover/autodiscover.xml\ttext/xml; charset=utf-8\tInet\t200\tOutlook Anywhere\tReq ID: 0b6580f8; Compression: client=No, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=yes, logged off=no, client type=unknown, user activity=yes\tPerimeter\tLocal Host\t0x600\tAllowed\t-\tAllowed\t-\t-\t-\t-\t-\t-\t0\t-\t0\t-\t192.168.100.53\t-\tWeb Proxy\tautodiscover.domain.com\t51033\t-",
    "@version": "1",
    "@timestamp": "2016-10-04T10:10:20.867Z",
    "beat": {
      "hostname": "TMG",
      "name": "TMG"
    },
    "source": "C:\\Program Files\\Microsoft Forefront Threat Management Gateway\\Logs\\ISALOG_20161004_WEB_000.w3c",
    "offset": 1163673,
    "type": "log",
    "input_type": "log",
    "host": "TMG",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "clientip": "10.10.10.10",
    "user": "domain\\user",
    "clientagent": "Microsoft Office/15.0 (Windows NT 10.0; Microsoft Outlook 15.0.4859; Pro)",
    "month": "10",
    "day": "04",
    "servername": "TMG",
    "domain": "autodiscover.domain.com",
    "timestamp": "2016-10-04 10:10:04",
    "geoip": {
      "ip": "79.74.113.40",
      "country_code2": "GB",
      "country_code3": "GBR",
      "country_name": "United Kingdom",
      "continent_code": "EU",
      "latitude": 54,
      "longitude": -2,
      "timezone": "Europe/London",
      "location": [
        -2,
        54
      ],
      "coordinates": [
        -2,
        54
      ]
    }
  },
  "fields": {
    "@timestamp": [
      1475575820867
    ]
  },
  "sort": [
    1475575820867
  ]
}

I have already refreshed the index.

EDIT: I have found out that the filebeat-template cannot map the geoip.

I had a look again in my config and had this output configured:

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => true
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

I changed it to:

output {
  elasticsearch {
    hosts => "localhost:9200"
  }
}

Now the logs from filebeat are going into the logstash index and the mapping is fine.

Did I do this right?

22

This topic was automatically closed after 21 days. New replies are no longer allowed.