# CombinedApacheLog pattern where request is without parameters **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 5, 2017, 3:12pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383 "2017-12-05T15:12:43Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Ronald\_Haring](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronald_haring/32/145144_2.png) [@Ronald\_Haring](https://discuss.elastic.co/u/Ronald_Haring)\ **Post date:** [December 5, 2017, 3:12pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383/1 "2017-12-05T15:12:43Z")
Hello i have set up a filebeat es and kibana stack but in the kibana/es i can see that the request contains parameters making it hard to collect all data on a specific url, i did find the URIPATH parameter but dont know if its possible to chain the combinedapachelog output to another grok pattern in ingest any pointers? regards Ronald ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [December 5, 2017, 5:17pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383/2 "2017-12-05T17:17:02Z")
You can add further grok patterns to the existing ones. Exmaple: `my-new-pattern` is added to existing patterns: ```auto "grok": { "field": "message", "patterns":[ "%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \\[%{HTTPDATE:apache2.access.time}\\] \"%{WORD:apache2.access.method} %{DATA:apache2.access.url} HTTP/%{NUMBER:apache2.access.http_version}\" %{NUMBER:apache2.access.response_code} (?:%{NUMBER:apache2.access.body_sent.bytes}|-)( \"%{D ATA:apache2.access.referrer}\")?( \"%{DATA:apache2.access.agent}\")?", "%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \\[%{HTTPDATE:apache2.access.time}\\] \"-\" %{NUMBER:apache2.access.response_code} -", "my-new-pattern" ], } ``` After adding it do not forget to update the template. By default FB does not update pipelines if it is changed. So you must do it manually. ---
**Author:** ![Rolf\_Anderegg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rolf_anderegg/32/45010_2.png) [@Rolf\_Anderegg](https://discuss.elastic.co/u/Rolf_Anderegg)\ **Post date:** [December 5, 2017, 11:52pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383/3 "2017-12-05T23:52:21Z")
Can you tell me how to do such an update? I'm stuck since some hours with a custom made Pattern whre i allways get an "Provided Grok expressions do not match field value" in kibana. But i tried the pattern in the grok test tool "[http://grokconstructor.appspot.com/do/match#result](http://grokconstructor.appspot.com/do/match#result)" an there the patern works perfect there. ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [December 6, 2017, 1:20pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383/4 "2017-12-06T13:20:38Z")
I usually use the Simulate API or the Grok Debugger in X-Pack Basic. Simulate API: [https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html) Grok debugger: [https://www.elastic.co/guide/en/kibana/current/xpack-grokdebugger.html](https://www.elastic.co/guide/en/kibana/current/xpack-grokdebugger.html) A possible gotcha is that in `pipeline.json` files you must put escape backslashes whereas, it is not required in other Grok testers. So you could try changing every `\` into `\\`. If these does not help you, feel free to share a sample log line and your grok pattern. So we can work out a solution. ---
**Author:** ![Rolf\_Anderegg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rolf_anderegg/32/45010_2.png) [@Rolf\_Anderegg](https://discuss.elastic.co/u/Rolf_Anderegg)\ **Post date:** [December 6, 2017, 8:52pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383/5 "2017-12-06T20:52:14Z")
Hello Noémi, thanks for your Reply, I copied the Apache Filebeat Configuration ant tried to adapt to my Logfile (for testing purposes i took an exaple from somwhere to be shure the Pattern is correct). This is my /usr/share/filebeat/module/perf/access/manifest.yml: ``` module_version: 1.0 var: - name: paths default: - /var/log/httpd/perf.log - /var/log/apache2/other_vhosts_access.log* os.darwin: - /usr/local/var/log/apache2/access_log* os.windows: - "C:/tools/Apache/httpd-2.*/Apache24/logs/access.log*" - "C:/Program Files/Apache Software Foundation/Apache2.*/logs/access.log*" ingest_pipeline: ingest/pipeline.json prospector: config/perf.yml ``` This is the /usr/share/filebeat/module/perf/access/ingest/pipeline.json (i double checked that this pipeline.json is loaded, i renamed it and i got the error the file is not found) ``` { "description": "Pipeline for parsing Apache2 performance logs. Requires the geoip and user_agent plugins.", "processors": [{ "grok": { "field": "message", "patterns":[ "%{TIMESTAMP_ISO8601:perf.access.timestamp} \\[%{IPV4:ip};%{WORD:perf.access.environment}\\] %{LOGLEVEL:perf.access.log_level} %{GREEDYDATA:perf.access.message}" ], "ignore_missing": false } },{ "remove":{ "field": "message" } }], "on_failure" : [{ "set" : { "field" : "error.message", "value" : "{{ _ingest.on_failure_message }}" } }] } ``` This is the Log File /var/log/httpd/perf.log (i filled by hand to test) ``` 2016-09-19T18:19:00 [8.8.8.8:prd] DEBUG this is an example log message 2016-09-19T18:19:00 [8.8.8.8:prd] DEBUG this is an example log message ``` This is the Output when i try: filebeat -e -modules=perf -d "\*" ``` 2017/12/06 20:21:59.321210 prospector.go:350: DBG Check file for harvesting: /var/log/httpd/perf.log 2017/12/06 20:21:59.321224 prospector.go:436: DBG Update existing file for harvesting: /var/log/httpd/perf.log, offset: 71 2017/12/06 20:21:59.321231 prospector.go:488: DBG Harvester for file is still running: /var/log/httpd/perf.log 2017/12/06 20:21:59.321239 prospector.go:157: DBG Prospector states cleaned up. Before: 1, After: 1 2017/12/06 20:22:04.322709 log.go:85: DBG End of file reached: /var/log/httpd/perf.log; Backoff now. 2017/12/06 20:22:09.321364 prospector.go:140: DBG Run prospector 2017/12/06 20:22:09.321387 prospector.go:136: DBG Start next scan 2017/12/06 20:22:09.321431 prospector.go:350: DBG Check file for harvesting: /var/log/httpd/perf.log 2017/12/06 20:22:09.321443 prospector.go:436: DBG Update existing file for harvesting: /var/log/httpd/perf.log, offset: 71 2017/12/06 20:22:09.321449 prospector.go:488: DBG Harvester for file is still running: /var/log/httpd/perf.log 2017/12/06 20:22:09.321457 prospector.go:157: DBG Prospector states cleaned up. Before: 1, After: 1 2017/12/06 20:22:14.322957 processor.go:262: DBG Publish event: { "@timestamp": "2017-12-06T20:22:14.322Z", "@metadata": { "beat": "filebeat", "type": "doc", "version": "6.0.0", "pipeline": "filebeat-6.0.0-perf-access-pipeline" }, "message": "2016-09-19T18:19:00 [8.8.8.8:prd] DEBUG this is an example log message", "source": "/var/log/httpd/perf.log", "offset": 142, "fileset": { "module": "perf", "name": "access" }, "prospector": { "type": "log" }, "beat": { "name": "enac2-dev", "hostname": "enac2-dev", "version": "6.0.0" } } 2017/12/06 20:22:14.323011 log.go:85: DBG End of file reached: /var/log/httpd/perf.log; Backoff now. 2017/12/06 20:22:15.323169 log.go:85: DBG End of file reached: /var/log/httpd/perf.log; Backoff now. 2017/12/06 20:22:15.326545 client.go:282: DBG PublishEvents: 1 events have been published to elasticsearch in 3.352298ms. 2017/12/06 20:22:15.326578 logger.go:29: DBG ackloop: receive ack [1: 0, 1] 2017/12/06 20:22:15.326589 logger.go:29: DBG broker ACK events: count=1, start-seq=2, end-seq=2 2017/12/06 20:22:15.326597 logger.go:18: DBG ackloop: return ack to broker loop:1 2017/12/06 20:22:15.326603 logger.go:18: DBG ackloop: done send ack 2017/12/06 20:22:15.326623 registrar.go:200: DBG Processing 1 events 2017/12/06 20:22:15.326633 registrar.go:195: DBG Registrar states cleaned up. Before: 2, After: 2 2017/12/06 20:22:15.326638 registrar.go:228: DBG Write registry file: /var/lib/filebeat/registry 2017/12/06 20:22:15.327260 registrar.go:253: DBG Registry file updated. 2 states written. 2017/12/06 20:22:17.323368 log.go:85: DBG End of file reached: /var/log/httpd/perf.log; Backoff now. 2017/12/06 20:22:19.310159 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4194304 beat.memstats.memory_alloc=2487336 beat.memstats.memory_total=4269592 filebeat.events.added=5 filebeat.events.done=5 filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.config.module.running=0 libbeat.config.reloads=1 libbeat.output.read.bytes=1611 libbeat.output.type=elasticsearch libbeat.output.write.bytes=1673 libbeat.pipeline.clients=3 libbeat.pipeline.events.active=0 libbeat.pipeline.events.filtered=3 libbeat.pipeline.events.published=2 libbeat.pipeline.events.retry=1 libbeat.pipeline.events.total=5 libbeat.pipeline.queue.acked=2 registrar.states.current=2 registrar.states.update=5 registrar.writes=5 2017/12/06 20:22:19.321564 prospector.go:140: DBG Run prospector ``` This is the json i see in KABANA ``` { "_index": "filebeat-6.0.0-2017.12.06", "_type": "doc", "_id": "b7WFLWAB_HTHudAEojbx", "_score": 1, "_source": { "@timestamp": "2017-12-06T20:30:20.674Z", "offset": 213, "beat": { "hostname": "enac2-dev", "name": "enac2-dev", "version": "6.0.0" }, "prospector": { "type": "log" }, "source": "/var/log/httpd/perf.log", "message": "2016-09-19T18:19:00 [8.8.8.8:prd] DEBUG this is an example log message", "fileset": { "module": "perf", "name": "access" }, "error": { "message": "Provided Grok expressions do not match field value: [2016-09-19T18:19:00 [8.8.8.8:prd] DEBUG this is an example log message]" } }, "fields": { "@timestamp": [ "2017-12-06T20:30:20.674Z" ] } } ``` For me it seems that the pattern is completly ignored but i cant figure out why. But i dont knw where to further investigate. Thankful for any hint Rolf ---
**Author:** ![Ronald\_Haring](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronald_haring/32/145144_2.png) [@Ronald\_Haring](https://discuss.elastic.co/u/Ronald_Haring)\ **Post date:** [December 8, 2017, 2:55pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383/6 "2017-12-08T14:55:40Z")
thanks for that insight i have now changed my pipeline to the following pattern: ``` { "description" : "Ingest pipeline for Apache httpd Combined Log Format", "processors" : [ { "grok": { "field": "message", "patterns": ["%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{URIPATH:requestpath}(?:%{URIPARAM:requestparameters})?(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)"] } }, { "date": { "field": "timestamp", "formats": ["dd/MMM/YYYY:HH:mm:ss Z"] } }, { "geoip": { "field": "client_ip" } }, { "user_agent": { "field": "agent" } } ] } ``` when i run this pattern in the grok debugger i see that my request is indeed split up, yet in the final elasticsearch i do not see them again, is there perhaps a setting that helps me see what message is finally digested by elasticsearch? Regards Ronald ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [December 8, 2017, 3:02pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383/7 "2017-12-08T15:02:49Z")
You should try `\\[%{HTTPDATE:timestamp}\\]` instead of `\[%{HTTPDATE:timestamp}\]`. ---
**Author:** ![Ronald\_Haring](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronald_haring/32/145144_2.png) [@Ronald\_Haring](https://discuss.elastic.co/u/Ronald_Haring)\ **Post date:** [December 13, 2017, 3:13pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383/8 "2017-12-13T15:13:51Z")
thanks updated my pattern to double escape the slashes, when i run it in the \_ingest/simulate it works but after running filebeat the timestamp is set to the moment i ran the filebeat command see my filebeat.yml[pastebin -\> filebeat.yml](https://pastebin.com/7xp0HarD) and the result from the simulate command[simulate result](https://pastebin.com/RVpjqQhx), so how can i inform es to use correct timestamp? or alternatively does someone have a good example to use filebeat to parse apache logs where the request is split up to have seperate entry for request with and without parameters thanks Ronald ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 10, 2018, 3:14pm UTC](https://discuss.elastic.co/t/combinedapachelog-pattern-where-request-is-without-parameters/110383/9 "2018-01-10T15:14:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.