Please help to grok my TMG Logs. How to use GeoIP?

toasti · September 22, 2016, 6:38am

Hello there,

I am absolutely new to the ELK-Stack, but as described in my other thread I have an ELK-Stack running since yesterday and absolutely flashed what is possible with this.

Since yesterday I also send my TMG access log files (w3c) per filebeat to logstash and have them now visible in Kibana.

I already heard and read about how to grok such log files, but I don't know how start. I had one try yesterday, but nothing happened.

My log looks like this:

10.10.10.10    domain\user    Microsoft Office/15.0 (Windows NT 6.3; Microsoft Outlook 15.0.4849; Pro)    2016-09-21    23:51:45    TMG    -    subdomain.domain.com    172.16.0.1    443    47    883    4446    https    POST    http://subdomain.domain.com/autodiscover/autodiscover.xml    text/xml; charset=utf-8    Inet    200    Outlook Anywhere    Req ID: 0a61611c; Compression: client=No, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=yes, logged off=no, client type=unknown, user activity=yes    Perimeter    Local Host    0x600    Allowed    -    Allowed    -    -    -    -    -    -    0    -    0    -    172.16.100.1    -    Web Proxy    subdomain.domain.com    41461    -

Could you explain me how it works?

I would also like to know how I can install the GeoIP plugin and how to use that.

Thank you very much!

magnusbaeck · September 22, 2016, 7:20am

To reach the Logstash folks I suggest you edit your post and move it to the Logstash category.

toasti · September 22, 2016, 7:30am

Thank you, your are right!

toasti · September 28, 2016, 9:31am

I have done my best, but there's nothing happen.

This is the log I want to grok:

10.10.10.10 domain\user Mozilla/5.0 (iPad; CPU OS 9_3_5 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13G36 Safari/601.1 2016-09-28 06:39:10 servername - sub.domain.local

And this is my logstash conf:

input {
eventlog {
type => 'Win32-EventLog'
logfile => ["Application", "Security", "System"]
}
syslog {
type => "syslog"
port => 514
}
}

filter {
if [beat.name] == "servername" {
grok {
match => [ "message" => "%{IPV4:clientip} %{DATA:user} %{GREEDYDATA:clientagent} %{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY} %{TIME} %{WORD:servername} %{DATA:bin} %{HOSTNAME:domain}" ]
}
date {
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss" ]
target => "@timestamp"
}
mutate {
remove_field => "year, monthnum, monthday, time, hour, minute, second, bin"
}
}
}

output {
elasticsearch {
host => "localhost"
protocol => "http"
}
}

Could you please help me?

magnusbaeck · September 28, 2016, 10:59am

if [beat.name] == "servername" {

Change to:

if [beat][name] == "servername" {

toasti · September 28, 2016, 11:38am

Thanks for your help Magnus!

I have changed it and will try. The other parts of the conf and filter are ok?

I have also added

beats {
port => 5044
}

to the logstash.conf - is that needed?

Sending logs are not possible, the firewall on the TMG says on of the peers send a reset packet.

magnusbaeck · September 28, 2016, 12:17pm

No, there are other problems.

I can't see that you're defining the timestamp field anywhere. Instead of capturing the individual timestamp pieces you should be able to use the TIMESTAMP_ISO8601 pattern and capture the whole timestamp into the timestamp field.
The remove_field option accepts an array of field names, not a string with comma-separated field names. Hence, change to remove_field => ["year", "monthnum", ...].
Unless you're running Logstash 1.x (and why would you do that?) the elasticsearch output needs to be modified to remove the protocol option and to use hosts instead of host.

I have also added

beats {
port => 5044
}

to the logstash.conf - is that needed?

Are you going one of the Beats programs to send stuff to port 5044 on the Logstash server?

toasti · September 28, 2016, 12:38pm

You are right, TIMESTAMP_ISO8601 is working fine. I tested it in the Debugger.
I have changed the remove_field.

Logstash version is 2.4.0.

Yes, I am using another server to send logs by filebeat to the ELK-Stack server.

The filebeat config is:

filebeat.prospectors:

input_type: log

paths:
C:\Program Files\Microsoft Forefront Threat Management Gateway\Logs*.w3c

output.logstash:

hosts: ["10.10.10.10:5044"]

logging.level: debug

EDIT: At this moment I also see that my syslog is no more working. Is the logstash.conf itself the problem?

EDIT2: The beats plugin and filter in the logstash.conf is the problem. As soon as this is added to the config, also the syslog is no more working.

toasti · September 29, 2016, 7:34am

The problem is always if I change my logstash.conf and add

beats {
port => 5044
}

into the input options.

Only beats as input is also not working.
In the filebeat log of the sending server I got error messages that the server refused the connection.

Don't know any more to test out - do you have any idea?

magnusbaeck · September 29, 2016, 7:39am

When you add that configuration, is Logstash actually listening on port 5044? Can you connect from the Filebeat machine via telnet? Is there a firewall on the Logstash machine that might be blocking the access? Is there a firewall on the Filebeat machine that might be blocking the access?

toasti · September 29, 2016, 7:52am

No, it's not listening so I can't connect by telnet. Connection refused.
I tried also to connect from another machine by telnet - the same problem.

Firewall is turned off.

I have the feeling that anything with my logstash.conf or beats plugin is not ok, because when I have the syslog as well as the beats plugin in, syslog is also no more working.

As soon as I have deleted the beats plugin again, syslog is working again.

Only beats as input will also not open port 5044.

Syslog on port 514 is immediately open when I save the conf and restart services.

I had a look into the Windows Event Viewer and if beats plugin is added to logstash.conf the service is killed every 20 seconds and then restarting, again and again. When I delete it and put in only syslog plugin it's working and not restarting.

toasti · September 29, 2016, 12:05pm

OK, working now. I did a complete re-install of the ELK-Stack.
Now it's listening on 5044 and getting logs.

But the filter is not working. Nothing happens with the logs and they are shown as complete message in Kibana.

I want to grok this one:

10.10.10.10 anonymous Microsoft Office/14.0 (Windows NT 5.1; Microsoft Outlook 14.0.6023; Pro) 2016-09-29 12:26:35 TMG - sub.domain.local 10.10.10.20 443 1 835 2280 https POST http://sub.domain.local/autodiscover/autodiscover.xml - - 12202 Default rule Req ID: 0b64c119; Compression: client=No, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=no, logged off=no, client type=unknown, user activity=yes Internal Local Host 0x200 Denied - Allowed - - - - - - 0 - 0 - - - Web Proxy sub.domain.local 45521 -

Interesting for me is only the data until "TMG". My filter is this at the moment:

filter {
  if [host] == "TMG" {
    grok {
       match => { "message" => ""%{IPV4:clientip}\t%{DATA:user}\t%{DATA:clientagent}\t%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}\t%{TIME}\t%{WORD:servername}\t%{DATA:bin}\t%{HOSTNAME:domain}\t%{DATA:bin}" }
    }
    date {
           match => [ "timestamp", "YYYY-MM-dd HH:mm:ss" ] }
           target => "@timestamp"
    }
    mutate {
       remove_field => ["year", "monthnum", "monthday", "time", "hour", "minute", "second", "bin"]
    }
  }
}

It's working in the grok debugger, but nothing happens. Also TIMESTAMP_ISO8601 was no more working.

toasti · October 4, 2016, 7:05am

Can anybody help me with this?

How can I see if an error message is generated from logstash?

I have the ELK-Stack running on Windows Server.

magnusbaeck · October 4, 2016, 7:29am

Can anybody help me with this?

Show us what the events look like when they leave Logstash. Either use a stdout { codec => rubydebug } output or copy/paste from the event's JSON tab in Kibana. No screenshots.

How can I see if an error message is generated from logstash?

Read the Logstash log. Its location is given by the -l (or --log) Logstash startup option. If absent, Logstash will log to stderr (or is it stdout?).

toasti · October 4, 2016, 8:26am

Hello Magnus,
Thank you!

Shame on me... On the way to change the config and starting logstash in the commandline instead of the installed service I have found the problem...

I followed a tutorial to install the ELK-Stack which is not using the default config (logstash.conf), but another one in the bin folder: logstash.json which the guys from the tutorial delivered.
The service was directly started with this config, so the consequence is that nothing was applied what I have changed.

You showed me the right way... thank you!

Now the filter is working, all fields what I need are separated. Only the timestamp is not replaced as it should.

Do you have any idea what to correct to get this working?

magnusbaeck · October 4, 2016, 9:00am

Now the filter is working, all fields what I need are separated. Only the timestamp is not replaced as it should.

Do you have any idea what to correct to get this working?

Your date filter isn't working because you have no timestamp field to parse. The TIMESTAMP_ISO8601 pattern should work for you, and then you capture the timestamp into a single field that you can feed to the date filter.

toasti · October 4, 2016, 9:21am

My json data is this:

{
  "_index": "filebeat-2016.10.04",
  "_type": "log",
  "_id": "AVeO9XjXmaRlHgp4ACp6",
  "_score": null,
  "_source": {
    "message": "10.10.10.10\tdomain\\user\tMicrosoft BITS/7.8\t2016-10-04\t09:10:18\tDETMG002\t-\tsub.domain.com\t192.168.100.25\t443\t1\t290\t10516\thttps\tGET\thttp://sub.domain.com/OAB/4a29dece-f668-4cfa-9d8e-7580a5579739/oab.xml\ttext/xml\tInet\t206\tOutlook Anywhere\tReq ID: 0b657e81; Compression: client=No, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=yes, valid=yes, updated=no, logged off=no, client type=public, user activity=yes, Range=13311-23502\tPerimeter\tLocal Host\t0x580\tAllowed\t-\tAllowed\t-\t-\t-\t-\t-\t-\t0\t-\t0\t-\t-\t-\tNone\tsub.domain.com\t50128\t-",
    "@version": "1",
    "@timestamp": "2016-10-04T09:10:29.858Z",
    "source": "C:\\Program Files\\Microsoft Forefront Threat Management Gateway\\Logs\\ISALOG_20161004_WEB_000.w3c",
    "offset": 929593,
    "type": "log",
    "input_type": "log",
    "beat": {
      "name": "TMG",
      "hostname": "TMG"
    },
    "host": "TMG",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "clientip": "10.10.10.10",
    "user": "domain\user",
    "clientagent": "Microsoft BITS/7.8",
    "servername": "TMG",
    "domain": "sub.domain.com"
  },
  "fields": {
    "@timestamp": [
      1475572229858
    ]
  },
  "sort": [
    1475572229858
  ]
}

When I replace my parts of the filter

\t%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}\t%{TIME}

with

\t%{TIMESTAMP_ISO8601}

I got _grokparsefailure.

magnusbaeck · October 4, 2016, 9:26am

Oh, you have a tab between the date and the time. In that case you can capture each date and time component into a field of its own (%{YEAR:year} etc) and merge those fields into a single field with add_field:

grok {
  match => ...
  add_field => {
    "timestamp" => "%{year}-%{month}-%{day} %{time}"
  }
}

Or, skip the grok filter completely and use csv instead. Usually much easier for delimited data.

toasti · October 4, 2016, 9:34am

Yes

This is my JSON data, the timestamp field is not correctly filled:

{
  "_index": "filebeat-2016.10.04",
  "_type": "log",
  "_id": "AVePBsUSmaRlHgp4ACrh",
  "_score": null,
  "_source": {
    "message": "192.168.100.3\tanonymous\tMicrosoft Office/14.0 (Windows NT 5.1; Microsoft Outlook 14.0.6023; Pro)\t2016-10-04\t09:29:11\tTMG\t-\tautodiscover.domain.com\t192.168.100.53\t443\t1\t835\t2280\thttps\tPOST\thttp://autodiscover.domain.com/autodiscover/autodiscover.xml\t-\t-\t12202\tDefault rule\tReq ID: 0b657f4a; Compression: client=No, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=no, logged off=no, client type=unknown, user activity=yes\tInternal\tLocal Host\t0x200\tDenied\t-\tAllowed\t-\t-\t-\t-\t-\t-\t0\t-\t0\t-\t-\t-\tWeb Proxy\tautodiscover.domain.com\t57730\t-",
    "@version": "1",
    "@timestamp": "2016-10-04T09:29:23.102Z",
    "type": "log",
    "input_type": "log",
    "beat": {
      "name": "TMG",
      "hostname": "TMG"
    },
    "source": "C:\\Program Files\\Microsoft Forefront Threat Management Gateway\\Logs\\ISALOG_20161004_WEB_000.w3c",
    "offset": 993712,
    "host": "TMG",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "clientip": "192.168.100.3",
    "user": "anonymous",
    "clientagent": "Microsoft Office/14.0 (Windows NT 5.1; Microsoft Outlook 14.0.6023; Pro)",
    "servername": "TMG",
    "domain": "autodiscover.domain.com",
    "timestamp": "%{year}-%{month}-%{day} %{time}"
  },
  "fields": {
    "@timestamp": [
      1475573363102
    ]
  },
  "sort": [
    1475573363102
  ]
}

magnusbaeck · October 4, 2016, 9:44am

You didn't follow this part of my previous advice: "In that case you can capture each date and time component into a field of its own (%{YEAR:year} etc)".