# Please help to grok my TMG Logs. How to use GeoIP? **URL:** \ **Category:** Logstash\ **Created:** [September 22, 2016, 6:38am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206 "2016-09-22T06:38:05Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [September 22, 2016, 6:38am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/1 "2016-09-22T06:38:05Z")
Hello there, I am absolutely new to the ELK-Stack, but as described in my other thread I have an ELK-Stack running since yesterday and absolutely flashed what is possible with this. Since yesterday I also send my TMG access log files (w3c) per filebeat to logstash and have them now visible in Kibana. I already heard and read about how to grok such log files, but I don't know how start. I had one try yesterday, but nothing happened. My log looks like this: ``` 10.10.10.10 domain\user Microsoft Office/15.0 (Windows NT 6.3; Microsoft Outlook 15.0.4849; Pro) 2016-09-21 23:51:45 TMG - subdomain.domain.com 172.16.0.1 443 47 883 4446 https POST http://subdomain.domain.com/autodiscover/autodiscover.xml text/xml; charset=utf-8 Inet 200 Outlook Anywhere Req ID: 0a61611c; Compression: client=No, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=yes, logged off=no, client type=unknown, user activity=yes Perimeter Local Host 0x600 Allowed - Allowed - - - - - - 0 - 0 - 172.16.100.1 - Web Proxy subdomain.domain.com 41461 - ``` Could you explain me how it works? I would also like to know how I can install the GeoIP plugin and how to use that. Thank you very much! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 22, 2016, 7:20am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/2 "2016-09-22T07:20:19Z")
To reach the Logstash folks I suggest you edit your post and move it to the Logstash category. ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [September 22, 2016, 7:30am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/3 "2016-09-22T07:30:46Z")
Thank you, your are right! ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [September 28, 2016, 9:31am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/4 "2016-09-28T09:31:43Z")
I have done my best, but there's nothing happen. This is the log I want to grok: 10.10.10.10 domain\user Mozilla/5.0 (iPad; CPU OS 9\_3\_5 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13G36 Safari/601.1 2016-09-28 06:39:10 servername - sub.domain.local And this is my logstash conf: input { eventlog { type =\> 'Win32-EventLog' logfile =\> ["Application", "Security", "System"] } syslog { type =\> "syslog" port =\> 514 } } filter { if [[beat.name](http://beat.name)] == "servername" { grok { match =\> ["message" =\> "%{IPV4:clientip} %{DATA:user} %{GREEDYDATA:clientagent} %{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY} %{TIME} %{WORD:servername} %{DATA:bin} %{HOSTNAME:domain}" ] } date { match =\> ["timestamp", "YYYY-MM-dd HH:mm:ss"] target =\> "@timestamp" } mutate { remove\_field =\> "year, monthnum, monthday, time, hour, minute, second, bin" } } } output { elasticsearch { host =\> "localhost" protocol =\> "http" } } Could you please help me? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 28, 2016, 10:59am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/5 "2016-09-28T10:59:37Z")
> ``` > if [beat.name] == "servername" { > > ``` Change to: ``` if [beat][name] == "servername" { ``` > **[Accessing event data and fields | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references)** ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [September 28, 2016, 11:38am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/6 "2016-09-28T11:38:33Z")
Thanks for your help Magnus! I have changed it and will try. The other parts of the conf and filter are ok? I have also added beats { port =\> 5044 } to the logstash.conf - is that needed? Sending logs are not possible, the firewall on the TMG says on of the peers send a reset packet. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 28, 2016, 12:17pm UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/7 "2016-09-28T12:17:52Z")
No, there are other problems. - I can't see that you're defining the `timestamp` field anywhere. Instead of capturing the individual timestamp pieces you should be able to use the TIMESTAMP\_ISO8601 pattern and capture the whole timestamp into the `timestamp` field. - The `remove_field` option accepts an array of field names, not a string with comma-separated field names. Hence, change to `remove_field => ["year", "monthnum", ...]`. - Unless you're running Logstash 1.x (and why would you do that?) the elasticsearch output needs to be modified to remove the `protocol` option and to use `hosts` instead of `host`. > I have also added > beats { > port =\> 5044 > } > to the logstash.conf - is that needed? Are you going one of the Beats programs to send stuff to port 5044 on the Logstash server? ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [September 28, 2016, 12:38pm UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/8 "2016-09-28T12:38:32Z")
You are right, TIMESTAMP\_ISO8601 is working fine. I tested it in the Debugger. I have changed the remove\_field. Logstash version is 2.4.0. Yes, I am using another server to send logs by filebeat to the ELK-Stack server. The filebeat config is: filebeat.prospectors: - input\_type: log - C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs\*.w3c output.logstash: hosts: ["10.10.10.10:5044"] logging.level: debug EDIT: At this moment I also see that my syslog is no more working. Is the logstash.conf itself the problem? EDIT2: The beats plugin and filter in the logstash.conf is the problem. As soon as this is added to the config, also the syslog is no more working. ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [September 29, 2016, 7:34am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/9 "2016-09-29T07:34:44Z")
The problem is always if I change my logstash.conf and add ``` beats { port => 5044 } ``` into the input options. Only beats as input is also not working. In the filebeat log of the sending server I got error messages that the server refused the connection. Don't know any more to test out - do you have any idea? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 29, 2016, 7:39am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/10 "2016-09-29T07:39:30Z")
When you add that configuration, is Logstash actually listening on port 5044? Can you connect from the Filebeat machine via telnet? Is there a firewall on the Logstash machine that might be blocking the access? Is there a firewall on the Filebeat machine that might be blocking the access? ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [September 29, 2016, 7:52am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/11 "2016-09-29T07:52:51Z")
No, it's not listening so I can't connect by telnet. Connection refused. I tried also to connect from another machine by telnet - the same problem. Firewall is turned off. I have the feeling that anything with my logstash.conf or beats plugin is not ok, because when I have the syslog as well as the beats plugin in, syslog is also no more working. As soon as I have deleted the beats plugin again, syslog is working again. Only beats as input will also not open port 5044. Syslog on port 514 is immediately open when I save the conf and restart services. I had a look into the Windows Event Viewer and if beats plugin is added to logstash.conf the service is killed every 20 seconds and then restarting, again and again. When I delete it and put in only syslog plugin it's working and not restarting. ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [September 29, 2016, 12:05pm UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/12 "2016-09-29T12:05:07Z")
OK, working now. I did a complete re-install of the ELK-Stack. Now it's listening on 5044 and getting logs. But the filter is not working. Nothing happens with the logs and they are shown as complete message in Kibana. I want to grok this one: `10.10.10.10 anonymous Microsoft Office/14.0 (Windows NT 5.1; Microsoft Outlook 14.0.6023; Pro) 2016-09-29 12:26:35 TMG - sub.domain.local 10.10.10.20 443 1 835 2280 https POST http://sub.domain.local/autodiscover/autodiscover.xml - - 12202 Default rule Req ID: 0b64c119; Compression: client=No, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=no, logged off=no, client type=unknown, user activity=yes Internal Local Host 0x200 Denied - Allowed - - - - - - 0 - 0 - - - Web Proxy sub.domain.local 45521 -` Interesting for me is only the data until "TMG". My filter is this at the moment: ``` filter { if [host] == "TMG" { grok { match => { "message" => ""%{IPV4:clientip}\t%{DATA:user}\t%{DATA:clientagent}\t%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}\t%{TIME}\t%{WORD:servername}\t%{DATA:bin}\t%{HOSTNAME:domain}\t%{DATA:bin}" } } date { match => ["timestamp", "YYYY-MM-dd HH:mm:ss"] } target => "@timestamp" } mutate { remove_field => ["year", "monthnum", "monthday", "time", "hour", "minute", "second", "bin"] } } } ``` It's working in the grok debugger, but nothing happens. Also TIMESTAMP\_ISO8601 was no more working. ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [October 4, 2016, 7:05am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/13 "2016-10-04T07:05:11Z")
Can anybody help me with this? How can I see if an error message is generated from logstash? I have the ELK-Stack running on Windows Server. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 4, 2016, 7:29am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/14 "2016-10-04T07:29:21Z")
> Can anybody help me with this? Show us what the events look like when they leave Logstash. Either use a `stdout { codec => rubydebug }` output or copy/paste from the event's JSON tab in Kibana. No screenshots. > How can I see if an error message is generated from logstash? Read the Logstash log. Its location is given by the `-l` (or `--log`) Logstash startup option. If absent, Logstash will log to stderr (or is it stdout?). ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [October 4, 2016, 8:26am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/15 "2016-10-04T08:26:10Z")
Hello Magnus, Thank you! Shame on me... On the way to change the config and starting logstash in the commandline instead of the installed service I have found the problem... I followed a tutorial to install the ELK-Stack which is not using the default config (logstash.conf), but another one in the bin folder: logstash.json which the guys from the tutorial delivered. The service was directly started with this config, so the consequence is that nothing was applied what I have changed. You showed me the right way... thank you! Now the filter is working, all fields what I need are separated. Only the timestamp is not replaced as it should. Do you have any idea what to correct to get this working? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 4, 2016, 9:00am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/16 "2016-10-04T09:00:18Z")
> Now the filter is working, all fields what I need are separated. Only the timestamp is not replaced as it should. > Do you have any idea what to correct to get this working? Your date filter isn't working because you have no `timestamp` field to parse. The TIMESTAMP\_ISO8601 pattern should work for you, and then you capture the timestamp into a single field that you can feed to the date filter. ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [October 4, 2016, 9:21am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/17 "2016-10-04T09:21:56Z")
My json data is this: ``` { "_index": "filebeat-2016.10.04", "_type": "log", "_id": "AVeO9XjXmaRlHgp4ACp6", "_score": null, "_source": { "message": "10.10.10.10\tdomain\\user\tMicrosoft BITS/7.8\t2016-10-04\t09:10:18\tDETMG002\t-\tsub.domain.com\t192.168.100.25\t443\t1\t290\t10516\thttps\tGET\thttp://sub.domain.com/OAB/4a29dece-f668-4cfa-9d8e-7580a5579739/oab.xml\ttext/xml\tInet\t206\tOutlook Anywhere\tReq ID: 0b657e81; Compression: client=No, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=yes, valid=yes, updated=no, logged off=no, client type=public, user activity=yes, Range=13311-23502\tPerimeter\tLocal Host\t0x580\tAllowed\t-\tAllowed\t-\t-\t-\t-\t-\t-\t0\t-\t0\t-\t-\t-\tNone\tsub.domain.com\t50128\t-", "@version": "1", "@timestamp": "2016-10-04T09:10:29.858Z", "source": "C:\\Program Files\\Microsoft Forefront Threat Management Gateway\\Logs\\ISALOG_20161004_WEB_000.w3c", "offset": 929593, "type": "log", "input_type": "log", "beat": { "name": "TMG", "hostname": "TMG" }, "host": "TMG", "tags": [ "beats_input_codec_plain_applied" ], "clientip": "10.10.10.10", "user": "domain\user", "clientagent": "Microsoft BITS/7.8", "servername": "TMG", "domain": "sub.domain.com" }, "fields": { "@timestamp": [ 1475572229858 ] }, "sort": [ 1475572229858 ] } ``` When I replace my parts of the filter `\t%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}\t%{TIME}` with `\t%{TIMESTAMP_ISO8601}` I got **\_grokparsefailure**. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 4, 2016, 9:26am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/18 "2016-10-04T09:26:21Z")
Oh, you have a tab between the date and the time. In that case you can capture each date and time component into a field of its own (`%{YEAR:year}` etc) and merge those fields into a single field with `add_field`: ```nohighlight grok { match => ... add_field => { "timestamp" => "%{year}-%{month}-%{day} %{time}" } } ``` **Or** , skip the grok filter completely and use csv instead. Usually much easier for delimited data. ---
**Author:** ![toasti](https://avatars.discourse-cdn.com/v4/letter/t/b2d939/32.png) [@toasti](https://discuss.elastic.co/u/toasti)\ **Post date:** [October 4, 2016, 9:34am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/19 "2016-10-04T09:34:39Z")
Yes 🙂 This is my JSON data, the timestamp field is not correctly filled: ``` { "_index": "filebeat-2016.10.04", "_type": "log", "_id": "AVePBsUSmaRlHgp4ACrh", "_score": null, "_source": { "message": "192.168.100.3\tanonymous\tMicrosoft Office/14.0 (Windows NT 5.1; Microsoft Outlook 14.0.6023; Pro)\t2016-10-04\t09:29:11\tTMG\t-\tautodiscover.domain.com\t192.168.100.53\t443\t1\t835\t2280\thttps\tPOST\thttp://autodiscover.domain.com/autodiscover/autodiscover.xml\t-\t-\t12202\tDefault rule\tReq ID: 0b657f4a; Compression: client=No, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=no, logged off=no, client type=unknown, user activity=yes\tInternal\tLocal Host\t0x200\tDenied\t-\tAllowed\t-\t-\t-\t-\t-\t-\t0\t-\t0\t-\t-\t-\tWeb Proxy\tautodiscover.domain.com\t57730\t-", "@version": "1", "@timestamp": "2016-10-04T09:29:23.102Z", "type": "log", "input_type": "log", "beat": { "name": "TMG", "hostname": "TMG" }, "source": "C:\\Program Files\\Microsoft Forefront Threat Management Gateway\\Logs\\ISALOG_20161004_WEB_000.w3c", "offset": 993712, "host": "TMG", "tags": [ "beats_input_codec_plain_applied" ], "clientip": "192.168.100.3", "user": "anonymous", "clientagent": "Microsoft Office/14.0 (Windows NT 5.1; Microsoft Outlook 14.0.6023; Pro)", "servername": "TMG", "domain": "autodiscover.domain.com", "timestamp": "%{year}-%{month}-%{day} %{time}" }, "fields": { "@timestamp": [ 1475573363102 ] }, "sort": [ 1475573363102 ] } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 4, 2016, 9:44am UTC](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206/20 "2016-10-04T09:44:34Z")
You didn't follow this part of my previous advice: "In that case you can capture each date and time component into a field of its own (`%{YEAR:year}` etc)". [Next page](https://discuss.elastic.co/t/please-help-to-grok-my-tmg-logs-how-to-use-geoip/61206.md?page=2)