@prophoto I would recommend you use the Grok Debugger in Kibana (in Kibana's main menu go to the Management section of the menu > Dev Tools > Grok Debugger).
To find the Grok patterns in use by the Elasticsearch Ingest Node's Grok Processor which is part of an ingest pipeline set up by Filebeats, visit Kibana's Stack Management > Ingest Node Pipelines > filebeat-7.8.0-apache-access-pipeline. Clicking on it will bring up a Processors section that includes the Grok Processor definition and the patterns used by default to match Apache's log lines. As an alternative you can perform a "GET /_ingest/pipeline/filebeat-7.8.0-apache-access-pipeline`" to retrieve the JSON describing the pipeline configuration. Note that there's a separate pipeline for the error logs.
Also note that when using the Grok Debugger you will need to un-escape the Grok patterns that are included in the ingest pipeline's Grok Processor JSON definition, i.e. replace \"
with just "
and \\
with just \
Example: the first Grok pattern in filebeat-7.8.0-apache-access-pipeline
is properly escaped as part of the JSON document describing the pipeline configuration:
%{IPORHOST:destination.domain} %{IPORHOST:source.ip} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-)?\" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-)( \"%{DATA:http.request.referrer}\")?( \"%{DATA:user_agent.original}\")?
In Grok Debugger you would test the following pattern:
%{IPORHOST:destination.domain} %{IPORHOST:source.ip} - %{DATA:user.name} \[%{HTTPDATE:apache.access.time}\] "(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-)?" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-)( "%{DATA:http.request.referrer}")?( "%{DATA:user_agent.original}")?
@tactics In the parallel discussion "Filebeat versions from 7.0 - 7.8 fail to create alias field mappings for majority of modules" Elastic's Marcin Tojek pointed out the documentation where the creation of aliases is mentioned as a compatibility layer during the transition to ECS: Upgrade from 6.x to 7.x. The list of field renames can be found in Breaking changes in 7.0.