X-Forwarded Headers with IIS Module

[rlindner (Richard L)]

Hi there,

In 6.2 I was parsing IIS logs with Filebeat to Elasticsearch using a pipeline with grok filters, and modifying the filebeat.yml file to use the setup.template.name and setup.template.pattern. I was able to configure the grok filter and the ingest to include the X-Forwarded-For header, which was important since we're using an AWS Load Balancer.

I upgraded to 6.3 and am trying out the IIS Module. It works great (especially the user agent ingestion, which I hadn't yet tried to work with on 6.2). The problem is I don't seem to have any way of getting the X-Forwarded header to play nicely with the module.

My first question is, is it possible to modify the grok filter in the default.json ingest file for the IIS Module to include additional fields? I've tried adding %{NOTSPACE:iis.access.X-Forwarded-For}" and some variations to the grok filter on the default.json, and changing the "geoip" field like this:

{ "geoip": { "field": "iis.access.X-Forwarded-For", "target_field": "iis.access.geoip"}

But that does not work at all, I don't see an X-Forwarded header nor a geoip.

I've also tried adding the X-Forwarded-For header to the Filebeat fields.yml file, with the type of "array" (since that's how it looks like it's being done with nginx).

So that brings me back to my question of whether modifying these default files is even possible, or do I need to go back to how I was doing it previous to the IIS Module?

If I do need to create the custom JSON and ingest pipeline, will I still be able to use the IIS Module along with the geoip and user agent plugins?

I can provide more configs if needed, but just looking for some general guidance at this point since I'm hitting a wall.

[andrewkroh]

@rlindner

I've tried adding %{NOTSPACE:iis.access.X-Forwarded-For}" and some variations to the grok filter on the default.json

Unless you manually deleted the Ingest Node pipeline from Elasticsearch each time you modified the default.json the updated pipeline was never installed. Filebeat will check if the pipeline is absent and install it. It will not overwrite it if it already exists.

To delete a pipeline you would use the ingest node API. Delete pipeline API | Elasticsearch Guide [master] | Elastic

There has been some discussion about making it easier to customize, but until then you'll need to delete the existing pipeline before you customizations will take effect.

[rlindner (Richard L)]

Thank you! That's exactly the kind of oversight I was hoping I was having.

This should work, but I'll make sure to post an update if/when I fully get accomplish what I'm trying to do.

[rlindner (Richard L)]

That was definitely the part I was missing. I did end up creating a new template and pipeline altogether, but it really helped knowing that I need to delete and re-add the pipeline for changes to take effect.

It took a bit of trial and error with my grok filters, but I eventually got the X-Forwarded-For header parsing just the first IP with a regex pattern in the grok. That allowed it to work with the geo IP plugin after I mapped the geo-ip to the X-Forwarded-For header. Works awesome.

Thanks again!