# Filebeat and grok parsing errors **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 7, 2018, 4:29pm UTC](https://discuss.elastic.co/t/filebeat-and-grok-parsing-errors/143371 "2018-08-07T16:29:49Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![rdesanno](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdesanno/32/13607_2.png) [@rdesanno](https://discuss.elastic.co/u/rdesanno)\ **Post date:** [August 7, 2018, 4:29pm UTC](https://discuss.elastic.co/t/filebeat-and-grok-parsing-errors/143371/1 "2018-08-07T16:29:49Z")
This one should be pretty easy.... In the past, I was using filebeat to log to logstash directly, and putting in my own custom grok statements. It worked well but to be honest, was way more complicated than it needed to be. So I decided to reconfigure the filebeat output back to ES directly over port 9200. Everything is fine and works however, I am getting the following error while trying to ingest some of my apache ssl logs: _Provided Grok expressions do not match field value_ I had this covered with my previous configuration but now that im using FB-\>ES directly, can't seem to find where I should be setting these custom grok statements. Ive played around with _/usr/share/filebeat/module/apache2/access/ingest/default.json_ but that didn't seem to work. I've also tried to configure my FB apache modules but have been running into issues there as well _# filebeat modules enable apache_ _Error in modules manager: modules management requires 'filebeat.config.modules.path' setting_ What am I doing wrong? I can't seem to find the answers I am looking for and stumped. ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [August 7, 2018, 5:19pm UTC](https://discuss.elastic.co/t/filebeat-and-grok-parsing-errors/143371/2 "2018-08-07T17:19:50Z")
Hi @rdesanno, `filebeat` doesn't support grok expressions on itself, what we do with modules is to prepare a set of predefined pipelines that are installed in the elasticsearch ingest nodes, the json files you see contain these definitions. If you need to define your own grok patterns, you can [define and add your own pipeline](https://www.elastic.co/guide/en/beats/filebeat/6.3/configuring-ingest-node.html). Once added you can add to the filebeat configuration a [log input](https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-input-log.html) with: - The `paths` to the files you want to collect and parse - The `pipeline` option set to your created pipeline If you think that your custom patterns can be used by more people, or can be included in the apache module, we'll be happy to accept them as contributions in filebeat 🙂 Regarding the error enabling modules, your `filebeat.yml` configuration file needs to include the path to the modules configuration, something like this: ```auto filebeat.config.modules: path: ${path.config}/modules.d/*.yml ``` But these lines are already included in our distributed packages, how are you installing filebeat? Do you overwrite the configuration file? ---
**Author:** ![rdesanno](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdesanno/32/13607_2.png) [@rdesanno](https://discuss.elastic.co/u/rdesanno)\ **Post date:** [August 7, 2018, 6:35pm UTC](https://discuss.elastic.co/t/filebeat-and-grok-parsing-errors/143371/3 "2018-08-07T18:35:03Z")
> [@jsoriano](#): > > ```auto > filebeat.config.modules: > path: ${path.config}/modules.d/*.yml > > ``` With regards to the above, I've tried adding this but always run into a config issue like the following: # /usr/share/filebeat/bin/filebeat test config -c /etc/filebeat/filebeat.yml Exiting: error loading config file: yaml: line 14: did not find expected key And this is how I was testing it: #========================== Modules configuration ============================ filebeat.modules: filebeat.config.modules: path: ${path.config}/modules.d/\*.yml I'm sure its something wrong with my config but not seeing it. ---
**Author:** ![rdesanno](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdesanno/32/13607_2.png) [@rdesanno](https://discuss.elastic.co/u/rdesanno)\ **Post date:** [August 7, 2018, 6:42pm UTC](https://discuss.elastic.co/t/filebeat-and-grok-parsing-errors/143371/4 "2018-08-07T18:42:28Z")
> [@jsoriano](#): > > If you need to define your own grok patterns, you can [define and add your own pipeline](https://www.elastic.co/guide/en/beats/filebeat/6.3/configuring-ingest-node.html). Once added you can add to the filebeat configuration a [log input](https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-input-log.html) with: > > - The `paths` to the files you want to collect and parse > - The `pipeline` option set to your created pipeline No easier way of doing this? I considered redirecting my output to use a pipeline to handle these but not sure if it's worth the extra complication. No way to modify the apache/access ingester directly on the few hosts where this is an issue? If not, I guess I can look into this approach. ---
**Author:** ![rdesanno](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdesanno/32/13607_2.png) [@rdesanno](https://discuss.elastic.co/u/rdesanno)\ **Post date:** [August 7, 2018, 6:47pm UTC](https://discuss.elastic.co/t/filebeat-and-grok-parsing-errors/143371/5 "2018-08-07T18:47:00Z")
> [@jsoriano](#): > > If you think that your custom patterns can be used by more people, or can be included in the apache module, we'll be happy to accept them as contributions in filebeat 🙂 Here are the GROK statements for apache access that works in my environment if you want them. Some are copied over from _/usr/share/filebeat/module/apache2/access/ingest/default.json_, but the others are all me. ``` "%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \\[%{HTTPDATE:apache2.access.time}\\] \"%{WORD:apache2.access.method} %{DATA:apache2.access.url} HTTP/%{NUMBER:apache2.access.http_version}\" %{NUMBER:apache2.access.response_code} (?:%{NUMBER:apache2.access.body_sent.bytes}|-)( \"%{DATA:apache2.access.referrer}\")?( \"%{DATA:apache2.access.agent}\")?", "%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \\[%{HTTPDATE:apache2.access.time}\\] \"%{WORD:apache2.access.method} %{DATA:apache2.access.url} HTTP/%{NUMBER:apache2.access.http_version}\" %{NUMBER:apache2.access.response_code} (?:%{NUMBER:apache2.access.body_sent.bytes}|-)( \"%{DATA:apache2.access.referrer}\")?( \"%{DATA:apache2.access.agent}\")?", "\\[%{HTTPDATE:apache2.access.time}\\] %{IPORHOST:apache2.access.remote_ip} TLSv%{NUMBER:apache2.access.tls_version} (?:([A-z]|[0-9]|-| )*\")%{WORD:apache2.access.method} %{DATA:apache2.access.url} HTTP\/%{NUMBER:apache2.access.http_version}\" (?:%{NUMBER:apache2.access.body_sent.bytes}|-)", "%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \\[%{HTTPDATE:apache2.access.time}\\] \"-\" %{NUMBER:apache2.access.response_code} -" ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 4, 2018, 6:47pm UTC](https://discuss.elastic.co/t/filebeat-and-grok-parsing-errors/143371/6 "2018-09-04T18:47:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.