# Apache HTTPD logs with (comma-delimited) X-Forwarded-For IPs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 21, 2019, 3:37am UTC](https://discuss.elastic.co/t/apache-httpd-logs-with-comma-delimited-x-forwarded-for-ips/181917 "2019-05-21T03:37:30Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jamiejackson](https://avatars.discourse-cdn.com/v4/letter/j/d07c76/32.png) [@jamiejackson](https://discuss.elastic.co/u/jamiejackson)\ **Post date:** [May 21, 2019, 3:37am UTC](https://discuss.elastic.co/t/apache-httpd-logs-with-comma-delimited-x-forwarded-for-ips/181917/1 "2019-05-21T03:37:30Z")
Hi Folks, I thought I'd be able to use the filebeat apache module out of the box, but then I noticed the handling of X-Forwarded-For. It only seems to get the inner-most IP from it (which is a proxy, and not the end-user IP). HTTPD Log Format ```auto LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" ``` Sample: ```auto 2.2.2.2, 10.100.49.203 - - [21/May/2019:21:20:41 +0000] "GET /resource/reportmanagement/published/ESD_900000010053582_05112019_900000010189257_1526056201037.docx HTTP/1.1" 301 353 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 2.2.2.2, 10.100.48.62 - - [21/May/2019:21:20:41 +0000] "GET /onecpd/includes/themes/hudexchange/images/favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" 2.2.2.2, 10.100.49.203 - - [21/May/2019:21:20:41 +0000] "GET /trainings/courses HTTP/1.1" 200 7175 "-" "PiplBot (+http://www.pipl.com/bot/)" 2.2.2.2, 10.100.49.203 - - [21/May/2019:21:20:41 +0000] "GET /s3redirect/?ref=/resource/reportmanagement/published/ESD_900000010053582_05112019_900000010189257_1526056201037.docx HTTP/1.1" 301 86 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" ``` I want the `2.2.2.2` addresses, but I get the `10.100.*` addresses: ```json { "_index": "filebeat-7.1.0-2019.05.20-000001", "_type": "_doc", "_id": "1JRv2GoBZOU7qiPlQNcD", "_version": 1, "_score": null, "_source": { "agent": { "hostname": "f64aab1e2619", "id": "38dfc846-4fec-47df-a33b-cde479c27f11", "type": "filebeat", "ephemeral_id": "6fe6e7e5-36d1-4d04-a9a4-ae263c6a345b", "version": "7.1.0" }, "log": { "file": { "path": "/var/log/apache2/access_log" }, "offset": 0 }, "source": { "address": "10.100.49.203", "ip": "10.100.49.203" }, "fileset": { "name": "access" }, "url": { "original": "/resource/reportmanagement/published/ESD_900000010053582_05112019_900000010189257_1526056201037.docx" }, "input": { "type": "log" }, "apache": { "access": {} }, "@timestamp": "2019-05-21T21:20:41.000Z", "ecs": { "version": "1.0.0" }, "service": { "type": "apache" }, "host": { "name": "f64aab1e2619" }, "http": { "request": { "referrer": "-", "method": "GET" }, "response": { "status_code": 301, "body": { "bytes": 353 } }, "version": "1.1" }, "event": { "created": "2019-05-21T03:28:09.661Z", "module": "apache", "dataset": "apache.access" }, "user": { "name": "-" }, "user_agent": { "original": "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "os": { "name": "Android", "version": "6.0.1", "full": "Android 6.0.1" }, "name": "Googlebot", "device": { "name": "Spider" }, "version": "2.1" } }, "fields": { "suricata.eve.timestamp": [ "2019-05-21T21:20:41.000Z" ], "@timestamp": [ "2019-05-21T21:20:41.000Z" ], "event.created": [ "2019-05-21T03:28:09.661Z" ] }, "sort": [ 1558473641000 ] } ``` I'd like to leverage as much of the auto fanciness as possible. Is there any tweak I can do to keep this in filebeat instead of cobbling something together with logstash? I saw some similar topics from a few years back, but I wasn't sure if there had been any beat developments in the meantime. Thanks, Jamie ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [May 21, 2019, 9:15am UTC](https://discuss.elastic.co/t/apache-httpd-logs-with-comma-delimited-x-forwarded-for-ips/181917/2 "2019-05-21T09:15:20Z")
Hi @jamiejackson, Filebeat apache module only parses addresses in the common log format used by default by Apache. If you need to parse a custom log format you would need to define your own pipeline. Take into account that even if you need a custom pipeline you can still do it with an Elastic Ingest node, you don't need logstash only for that. You can read more about this here: [https://www.elastic.co/guide/en/beats/filebeat/7.0/configuring-ingest-node.html](https://www.elastic.co/guide/en/beats/filebeat/7.0/configuring-ingest-node.html) Alternatively if you don't want the `10.100.*` addresses you could consider removing this part from the log format and replace it with the `X-Forwarded-For` part, so the original ip would be captured as the source ip. ---
**Author:** ![jamiejackson](https://avatars.discourse-cdn.com/v4/letter/j/d07c76/32.png) [@jamiejackson](https://discuss.elastic.co/u/jamiejackson)\ **Post date:** [May 21, 2019, 4:50pm UTC](https://discuss.elastic.co/t/apache-httpd-logs-with-comma-delimited-x-forwarded-for-ips/181917/3 "2019-05-21T16:50:10Z")
Thanks for the response. A few questions: I hadn't looked into ingest pipelines before. Am I on the right track with this? (FYI, the `ip_trail` field is my attempt at preserving the original list of IPs from X-Forwarded-for.) ```auto POST /_ingest/pipeline/_simulate { "pipeline" : { "description": "Parse HTTP Access Logs", "processors": [ { "grok": { "field" : "message", "patterns" : [ "^%{IP_LIST:ip_trail}" ], "pattern_definitions": { "IP_LIST": "(?:-|(?:%{IPORHOST}(?:, [^\\s]+)*))" } } }, { "split": { "field": "ip_trail", "separator": "(,\\s+)" } }, { "gsub": { "field" : "message", "pattern" : "^(\\d+\\.\\d+\\.\\d+.\\d+)(, \\d+\\.\\d+\\.\\d+\\.\\d+)*", "replacement" : "$1" } } ] }, "docs" : [ { "_source": { "message": "2.2.2.2, 10.100.49.203 - - [21\/May\/2019:21:20:41 +0000] \"GET \/resource\/reportmanagement\/published\/ESD_900000010053582_05112019_900000010189257_1526056201037.docx HTTP\/1.1\" 301 353 \"-\" \"Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Build\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/41.0.2272.96 Mobile Safari\/537.36 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)\""} }, {"_source": { "message": "2.2.2.2, 10.100.49.203 - - [21\/May\/2019:21:20:41 +0000] \"GET \/resource\/reportmanagement\/published\/ESD_900000010053582_05112019_900000010189257_1526056201037.docx HTTP\/1.1\" 301 353 \"-\" \"Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Build\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/41.0.2272.96 Mobile Safari\/537.36 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)\""} }, { "_source": { "message": "12.234.21.234 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond"} }, { "_source": { "message": "- - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond"} } ] } ``` Which yields: ```auto { "docs": [ { "doc": { "_index": "_index", "_type": "_doc", "_id": "_id", "_source": { "message": "2.2.2.2 - - [21/May/2019:21:20:41 +0000] \"GET /resource/reportmanagement/published/ESD_900000010053582_05112019_900000010189257_1526056201037.docx HTTP/1.1\" 301 353 \"-\" \"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"", "ip_trail": [ "2.2.2.2", "10.100.49.203" ] }, "_ingest": { "timestamp": "2019-05-21T16:46:10.92033Z" } } }, { "doc": { "_index": "_index", "_type": "_doc", "_id": "_id", "_source": { "message": "2.2.2.2 - - [21/May/2019:21:20:41 +0000] \"GET /resource/reportmanagement/published/ESD_900000010053582_05112019_900000010189257_1526056201037.docx HTTP/1.1\" 301 353 \"-\" \"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"", "ip_trail": [ "2.2.2.2", "10.100.49.203" ] }, "_ingest": { "timestamp": "2019-05-21T16:46:10.920343Z" } } }, { "doc": { "_index": "_index", "_type": "_doc", "_id": "_id", "_source": { "message": "12.234.21.234 - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond", "ip_trail": [ "12.234.21.234" ] }, "_ingest": { "timestamp": "2019-05-21T16:46:10.920348Z" } } }, { "doc": { "_index": "_index", "_type": "_doc", "_id": "_id", "_source": { "message": "- - - [08/Jan/2018:00:00:26 -0500] RspTime= 75484 microsecond", "ip_trail": [ "-" ] }, "_ingest": { "timestamp": "2019-05-21T16:46:10.920353Z" } } } ] } ``` Also, this will need to be automated. Is there a way for the filebeat to create the pipeline on elasticsearch, or must that be a separate step? Finally, let me know if there are any refinements I should consider. (I'm new to all of this, so tips are appreciated.) > Alternatively if you don't want the `10.100.*` addresses you could consider removing this part from the log format and replace it with the `X-Forwarded-For` part, so the original ip would be captured as the source ip. Thanks, I might consider that, down the line, if those intermediate addresses don't prove useful. ---
**Author:** ![jamiejackson](https://avatars.discourse-cdn.com/v4/letter/j/d07c76/32.png) [@jamiejackson](https://discuss.elastic.co/u/jamiejackson)\ **Post date:** [May 21, 2019, 8:49pm UTC](https://discuss.elastic.co/t/apache-httpd-logs-with-comma-delimited-x-forwarded-for-ips/181917/4 "2019-05-21T20:49:07Z")
My latest problem is that the pipeline I'm specifying in filebeat.yml doesn't seem to be honored. (It seems to use the module's default.) ```auto filebeat.modules: - module: apache access: enabled: true var.paths: ["/var/log/apache2/access*"] error: enabled: true var.paths: ["/var/log/apache2/error*"] filebeat.config.modules: reload.enabled: true reload.period: 5s setup.kibana: host: "kibana-monitoring:5601" output.elasticsearch: hosts: ["elasticsearch-monitoring:9200"] pipeline: apache_with_x_forwarded_for ``` Are modules incompatible with `output.elasticsearch.pipeline` specifications? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [May 22, 2019, 5:17pm UTC](https://discuss.elastic.co/t/apache-httpd-logs-with-comma-delimited-x-forwarded-for-ips/181917/5 "2019-05-22T17:17:33Z")
> [@jamiejackson](#): > > I hadn't looked into ingest pipelines before. Am I on the right track with this? Yes, the idea would be to define your own pipeline as you are doing. You could start with the one used by the module, that you can find here: [https://github.com/elastic/beats/blob/v7.0.0/filebeat/module/apache/access/ingest/default.json](https://github.com/elastic/beats/blob/v7.0.0/filebeat/module/apache/access/ingest/default.json) > [@jamiejackson](#): > > Also, this will need to be automated. Is there a way for the filebeat to create the pipeline on elasticsearch, or must that be a separate step? You have to install the custom pipelines that you create, to automate this you can use the [API](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/put-pipeline-api.html). > [@jamiejackson](#): > > My latest problem is that the pipeline I'm specifying in filebeat.yml doesn't seem to be honored. (It seems to use the module's default.) It can be tricky to make modules work with custom pipelines, you need to override the settings of the input defined by the module, in your case something like this could work (not tested): ```auto filebeat.modules: - module: apache access: input: pipeline: apache_with_x_forwarded_for paths: ["/var/log/apache2/access*"] error: var.paths: ["/var/log/apache2/error*"] filebeat.config.modules: reload.enabled: true reload.period: 5s setup.kibana: host: "kibana-monitoring:5601" output.elasticsearch: hosts: ["elasticsearch-monitoring:9200"] ``` You can find more information in the documentation about [advanced module configuration](https://www.elastic.co/guide/en/beats/filebeat/7.0/advanced-settings.html), and the options you can use under `input` are the ones of the [log input](https://www.elastic.co/guide/en/beats/filebeat/7.0/filebeat-input-log.html). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 19, 2019, 5:17pm UTC](https://discuss.elastic.co/t/apache-httpd-logs-with-comma-delimited-x-forwarded-for-ips/181917/6 "2019-06-19T17:17:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.