New to this: Looking for Consulting Type Folks

Elastic Stack Beats
1

Hello Everyone!

So I (through much effort and a kind soul on Reddit) finally got a full ELKStack setup up and running where by the ELKStack lives on a Windows Machine. (Microsoft house) The Logs/Data I care about is from IIS on other windows remote web servers. I have IIS pumping in to Kibana via direct-to-elastic filebeat and the built-in IIS module.

The next part I'm looking for is this:

It seems IIS logs just pump their main log-lines into the "message" field in Kibana. So I have a single field called "message" that has the entire log-line from IIS in it.

The "message" field looks something like:
2019-10-25 07:03:45 W3SVC8 10.173.5.113 GET /my/path/to/things.htm - 443 - 184.50.88.108 Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+5X+Build/MMB29P)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/41.0.2272.96+Mobile+Safari/537.36+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - - www.mydomain.com 200 0 0 118799 1428 406 66.249.75.87

That's pretty useless. I'm looking to parse those out into more useful fields (i.e. - IIS.Client.Useragent) and from what I'm reading, I need to use a "Grok Filter" that seems to go into some YML file somewhere?

So If I can find resources that help me do this myself, that's great and I'll take care of it.

Honestly though, having somebody around in the U.S. (For timezone reasons, also Company can pay a consulting fee) who knows this kind of thing (specifically the Windows Deployment) would be ideal. I have a ton of questions and trying to comb through all the Linux folks and various pieces of semi-disparate documentation is proving difficult.

So! What I'm looking for:
A) You have a little bit of time to answer questions and help a guy out for a fee. Just having somebody with experience on a windows deployment would be amazing.
B) You know where I can find documentation on WHERE/HOW to apply the grok filter and maybe some examples of IIS W3C log Grok Filters that I can tool around with.

Thanks!

2

The filebeat iis module should populate fields and provide the groks you need, maybe something isn't quite right with it. Some things to check:

GET _ingest/pipeline - Should list all pipelines, including the filebeat iis pipeline.

Share your filebeat.yml file for the iis logs, and make sure filebeat modules list shows iis is active. (Unless you configured the module in the filebeat.yml file)

I don't have any way to check windows modules at the moment, I got lazy and just have my chromebook :slight_smile:

3

In the filebeat install, there should be files ../module/iis/access/ingest/default.json and .../module/iis/error/ingest/default.json that are the source for the ingest pipelines.

What version of filebeat did you install? I don't know then the iis module was added, I first started looking at at about 6.7.

4

Filebeat is 7.4.1

I'll check out those JSON's and look for something that looks similiar to a grok filter?

5

Those "should" load to ingest pipelines at first filebeat startup. If not, they can be loaded per the filebeat setup --pipelines --modules iis. (doc)

edited for wrong url...

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.