7.8 doesn't parse apache access logs

Elastic Stack Beats
1

I had an old 6.x stack setup a while ago but had to shut it down. I'm back to try another round and hopefully its permanent this time. I need to have my Apache 2.4 logs parsed so I can create custom dashboards.

All servers are CentOS 7. I installed Elasticsearch 7.8, setup Kibana 7.8, and Filebeat 7.8. ES and Kibana are installed in a local VM, the web servers are on AWS. I have Apache logs flowing properly into ES.

The problem is that the message field is not being parsed into separate fields. See the screenshot below, searching for apache2.access.url or apache2.access:referrer is showing no results. How do I configure Filebeat to parse the data or how would I diagnose my installation to make it happen?

Screen Shot 2020-07-23 at 8.57.04 AM
Screen Shot 2020-07-23 at 8.57.04 AM819Ɨ486 47.7 KB

2

I've been seeing a similar issue with the Nginx module. Inspecting the output of filebeat export template -eusing Filebeat 7.8 reveals the following (omitting the parts before and after the "apache" object):

      "apache": {
        "properties": {
          "access": {
            "properties": {
              "ssl": {
                "properties": {
                  "cipher": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "protocol": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              }
            }
          },
          "error": {
            "properties": {
              "module": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          }
        }
      },
      "apache2": {
        "properties": {
          "access": {
            "properties": {
              "geoip": {
                "properties": {}
              },
              "user_agent": {
                "properties": {}
              }
            }
          },
          "error": {
            "properties": {}
          }
        }
      },

This is a rather limited set of mappings compared to what is defined in the fields.yml:

- key: apache
  title: "Apache"
  description: >
    Apache Module
  short_config: true
  fields:
    - name: apache2
      type: group
      description: >
        Aliases for backward compatibility with old apache2 fields
      fields:
        - name: access
          type: group
          fields:
            - name: remote_ip
              type: alias
              path: source.address
              migration: true
            - name: ssl.protocol
              type: alias
              path: apache.access.ssl.protocol
              migration: true
            - name: ssl.cipher
              type: alias
              path: apache.access.ssl.cipher
              migration: true
            - name: body_sent.bytes
              type: alias
              path: http.response.body.bytes
              migration: true
            - name: user_name
              type: alias
              path: user.name
              migration: true
            - name: method
              type: alias
              path: http.request.method
              migration: true
            - name: url
              type: alias
              path: url.original
              migration: true
            - name: http_version
              type: alias
              path: http.version
              migration: true
            - name: response_code
              type: alias
              path: http.response.status_code
              migration: true
            - name: referrer
              type: alias
              path: http.request.referrer
              migration: true
            - name: agent
              type: alias
              path: user_agent.original
              migration: true

            - name: user_agent
              type: group
              fields:
                - name: device
                  type: alias
                  path: user_agent.device.name
                  migration: true
                - name: name
                  type: alias
                  path: user_agent.name
                  migration: true
                - name: os
                  type: alias
                  path: user_agent.os.full_name
                  migration: true
                - name: os_name
                  type: alias
                  path: user_agent.os.name
                  migration: true
                - name: original
                  type: alias
                  path: user_agent.original
                  migration: true
            - name: geoip
              type: group
              fields:
                - name: continent_name
                  type: alias
                  path: source.geo.continent_name
                  migration: true
                - name: country_iso_code
                  type: alias
                  path: source.geo.country_iso_code
                  migration: true
                - name: location
                  type: alias
                  path: source.geo.location
                  migration: true
                - name: region_name
                  type: alias
                  path: source.geo.region_name
                  migration: true
                - name: city_name
                  type: alias
                  path: source.geo.city_name
                  migration: true
                - name: region_iso_code
                  type: alias
                  path: source.geo.region_iso_code
                  migration: true
        - name: error
          type: group
          fields:
            - name: level
              type: alias
              path: log.level
              migration: true
            - name: message
              type: alias
              path: message
              migration: true
            - name: pid
              type: alias
              path: process.pid
              migration: true
            - name: tid
              type: alias
              path: process.thread.id
              migration: true
            - name: module
              type: alias
              path: apache.error.module
              migration: true


    - name: apache
      type: group
      description: >
        Apache fields.
      fields:
        - name: access
          type: group
          description: >
            Contains fields for the Apache HTTP Server access logs.
          fields:
            - name: ssl.protocol
              type: keyword
              description: >
                SSL protocol version.
        
            - name: ssl.cipher
              type: keyword
              description: >
                SSL cipher name.
        - name: error
          type: group
          description: >
            Fields from the Apache error logs.
          fields:
            - name: module
              type: keyword
              description: >
                The module producing the logged message.

Here's the nginx object in the output of filebeat setup -e which is also missing numerous mappings that are supposed to be exported according to Filebeat Reference 7.8, Exported fields, Nginx fields:

      "nginx": {
        "properties": {
          "access": {
            "properties": {
              "geoip": {
                "properties": {}
              },
              "user_agent": {
                "properties": {}
              }
            }
          },
          "error": {
            "properties": {
              "connection_id": {
                "type": "long"
              }
            }
          },
          "ingress_controller": {
            "properties": {
              "geoip": {
                "properties": {}
              },
              "http": {
                "properties": {
                  "request": {
                    "properties": {
                      "id": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "length": {
                        "type": "long"
                      },
                      "time": {
                        "type": "double"
                      }
                    }
                  }
                }
              },
              "upstream": {
                "properties": {
                  "alternative_name": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "ip": {
                    "type": "ip"
                  },
                  "name": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "port": {
                    "type": "long"
                  },
                  "response": {
                    "properties": {
                      "length": {
                        "type": "long"
                      },
                      "status_code": {
                        "type": "long"
                      },
                      "time": {
                        "type": "double"
                      }
                    }
                  }
                }
              },
              "user_agent": {
                "properties": {}
              }
            }
          }
        }
      },

By the looks of it none of the alias data type are getting created. There are some aliases beneath the suricata.* hierarchy and also some others but there's an awful few of them.

This was reproduced on Ubuntu 18.04 LTS with the filebeat package version 7.8.0 installed from the official repository at https://artifacts.elastic.co/packages/7.x/apt.

This might be some bug during the index template generation. Anybody else experiencing this problem?

1 Like
3

.Bump

4

@prophoto - I think that field is deprecated. The field is now "url.original". If you go into the "Discover" screen and then expand the arrow at the side of any entry it will show all the fields in there. I think some of the names have changed from 6.x to 7.

5

@tactics I see I read that doc page wrong! Thanks.

While poking around I did find this error. Here is my apache LogFormat from httpd.conf. Thoughts?

 LogFormat "%h %l %u %t %{Host}i \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{SSL_PROTOCOL}x\" %D %T" combined

"error": {
      "message": "Provided Grok expressions do not match field value:
6

@prophoto just checked my LogFormat, been a while since I looked up where and how they are generated but I am just using the standard common log format so:

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

So looks like you have a few different ones in there to me.

First difference is that you have this:

%{Host}i - the contents of the host header of the request.

Next, you have %b and I have %O it's pretty much the same thing.

Then you have:

"%{SSL_PROTOCOL}x" %D %T

%D - The time taken to serve the request, in microseconds.

%T - The time taken to serve the request, in seconds.

"%{SSL_PROTOCOL}x" - the SSL protocol:
https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#logformats

What are you using to parse the logs? Are you using the filebeat Apache module? I think SSL Protocol is in there from what I have seen. It could well be the %D and %T entries that are bugging out. Is there anything in the error message that would help identify it? Just trying to work out grok patterns for myself here so not much help!

7

I tried removing "%{SSL_PROTOCOL}x" %D %T no change.

8

Did you check to see if the error contained any further useful info? I was getting one that was just to an internal request for an XML feed from the site that didn't pass a user agent, plus just found some hack attempts with no user agent string, just "-" - this is throwing the same error you are getting.

If you load some data and then look in the Discover page of Kibana, are all entries showing the error or just some of them?

9

After this it just shows the entire log line. I also looked in system logs but didn't find anything relevant.

10

@tactics Can you provide some link that mentions that aliases are getting deprecated and the reasons behind this movie? I have just created a new discussion thread regarding this as the missing field mapping aliases are confusing for first-time users, see my full report Filebeat versions from 7.0 - 7.8 fail to create alias field mappings for majority of modules .

11

@b0le - deprecated as in not used as the field value for new imports of data. So if you are used to a particular layout or field mapping from 6.X and you set up from new and import again then things might not be where they were before.

I can see what you mean about the aliases though. By the looks of the fields.yml and docs the alias mappings to define the old fields to the new versions should be there.

https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-apache.html

This suggests an alias for apache2.access.url should be present and should alias to url.original. Maybe it's there as part of the ingest pipeline? Need to look into this further here as well as have some oddities I need to work out too. Distracted by some other work at present but will be back on it shortly.

Definitely agree that some clarity on how this works (or is supposed to work) would be of great help.

12

Thank you for looking into this @b0le! I believe my issue may be related to this error from the default grok filter? I am not sure how that works but seems that way. Still very new to Elasticsearch. Any suggestions?

      "message": "Provided Grok expressions do not match field value:

You mentioned in the other post about viewing additional logs from Filebeat. I ran this from the cli (CentOS 7) and get consistent errors about missing config and missing modules directory. I think it also prevented me from properly enabling the modules.

[centos@server9 filebeat]$ sudo bin/filebeat --path.config /etc/filebeat/ -e -d "publish"
2020-07-28T13:57:09.375Z	INFO	instance/beat.go:647	Home path: [/usr/share/filebeat/bin] Config path: [/etc/filebeat/] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]
2020-07-28T13:57:09.377Z	INFO	instance/beat.go:655	Beat ID: f72ad2d7-baa6-46e9-9a03-8252683b1db5
2020-07-28T13:57:09.377Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-07-28T13:57:09.377Z	INFO	[beat]	instance/beat.go:983	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/usr/share/filebeat/bin/data", "home": "/usr/share/filebeat/bin", "logs": "/usr/share/filebeat/bin/logs"}, "type": "filebeat", "uuid": "f72ad2d7-baa6-46e9-9a03-8252683b1db5"}}}
2020-07-28T13:57:09.377Z	INFO	[beat]	instance/beat.go:992	Build info	{"system_info": {"build": {"commit": "f79387d32717d79f689d94fda1ec80b2cf285d30", "libbeat": "7.8.0", "time": "2020-06-14T18:15:37.000Z", "version": "7.8.0"}}}
2020-07-28T13:57:09.377Z	INFO	[beat]	instance/beat.go:995	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.13.10"}}}
............
...........
2020-07-28T13:57:09.379Z	INFO	instance/beat.go:310	Setup Beat: filebeat; Version: 7.8.0
2020-07-28T13:57:09.379Z	INFO	[index-management]	idxmgmt/std.go:183	Set output.elasticsearch.index to 'filebeat-7.8.0' as ILM is enabled.
2020-07-28T13:57:09.379Z	INFO	eslegclient/connection.go:97	elasticsearch url: http://mydomain.org:9200
2020-07-28T13:57:09.379Z	INFO	[publisher]	pipeline/module.go:113	Beat name: server9
2020-07-28T13:57:09.379Z	ERROR	fileset/modules.go:127	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-07-28T13:57:09.380Z	INFO	instance/beat.go:463	filebeat start running.
2020-07-28T13:57:09.380Z	INFO	registrar/migrate.go:104	No registry home found. Create: /usr/share/filebeat/bin/data/registry/filebeat
2020-07-28T13:57:09.380Z	INFO	registrar/migrate.go:112	Initialize registry meta file
2020-07-28T13:57:09.380Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-07-28T13:57:09.382Z	INFO	registrar/registrar.go:108	No registry file found under: /usr/share/filebeat/bin/data/registry/filebeat/data.json. Creating a new registry file.
2020-07-28T13:57:09.384Z	INFO	registrar/registrar.go:145	Loading registrar data from /usr/share/filebeat/bin/data/registry/filebeat/data.json
2020-07-28T13:57:09.384Z	INFO	registrar/registrar.go:152	States Loaded from registrar: 0
2020-07-28T13:57:09.384Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 0
2020-07-28T13:57:09.397Z	ERROR	fileset/modules.go:127	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-07-28T13:57:09.397Z	ERROR	fileset/modules.go:127	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-07-28T13:57:09.398Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2020-07-28T13:57:09.398Z	INFO	cfgfile/reload.go:164	Config reloader started
2020-07-28T13:57:09.398Z	ERROR	fileset/modules.go:127	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-07-28T13:57:09.398Z	INFO	eslegclient/connection.go:97	elasticsearch url: http://studio.cazillo.org:9200
2020-07-28T13:57:09.488Z	INFO	[esclientleg]	eslegclient/connection.go:306	Attempting to connect to Elasticsearch version 7.8.0
2020-07-28T13:57:09.488Z	ERROR	fileset/modules.go:127	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-07-28T13:57:09.488Z	INFO	eslegclient/connection.go:97	elasticsearch url: http://mydomain.org:9200
2020-07-28T13:57:09.543Z	INFO	[esclientleg]	eslegclient/connection.go:306	Attempting to connect to Elasticsearch version 7.8.0
2020-07-28T13:57:09.543Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
13

Moved logging to its own file.

[centos@server9 filebeat]$ sudo systemctl start filebeat
[centos@server9 filebeat]$ sudo systemctl status filebeat
ā— filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-07-28 14:05:29 UTC; 4s ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 26230 (filebeat)
   CGroup: /system.slice/filebeat.service
           ā””ā”€26230 /usr/share/filebeat/bin/filebeat -environment systemd -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/fileb...
Jul 28 14:05:29 server9 systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
[centos@server9 filebeat]$ sudo ls /var/log/filebeat
filebeat  filebeat.1  filebeat.2  filebeat.3  filebeat.4  filebeat.5  filebeat.6  filebeat.7

sudo cat /var/log/filebeat/filebeat
2020-07-28T14:05:29.342Z	INFO	instance/beat.go:647	Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-07-28T14:05:29.346Z	INFO	instance/beat.go:655	Beat ID: 7e03be58-5896-4100-b203-d47c82a9dd9e
2020-07-28T14:05:29.346Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-07-28T14:05:29.346Z	INFO	[beat]	instance/beat.go:983	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "7e03be58-5896-4100-b203-d47c82a9dd9e"}}}
2020-07-28T14:05:29.347Z	INFO	[beat]	instance/beat.go:992	Build info	{"system_info": {"build": {"commit": "f79387d32717d79f689d94fda1ec80b2cf285d30", "libbeat": "7.8.0", "time": "2020-06-14T18:15:37.000Z", "version": "7.8.0"}}}
2020-07-28T14:05:29.347Z	INFO	[beat]	instance/beat.go:995	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.13.10"}}}
2020-07-28T14:05:29.347Z	INFO	[beat]	instance/beat.go:999	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-04T22:19:24Z","containerized":false,"name":"server9.cazillo.net","ip":["127.0.0.1/8","::1/128","10.0.0.109/24","fe80::c9f:2eff:fee9:265f/64"],"kernel_version":"3.10.0-957.27.2.el7.x86_64","mac":["0e:9f:2e:e9:26:5f"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"f9afeb75a5a382dce8269887a67fbf58"}}}
2020-07-28T14:05:29.348Z	INFO	[beat]	instance/beat.go:1028	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 26230, "ppid": 1, "seccomp": {"mode":"filter"}, "start_time": "2020-07-28T14:05:28.280Z"}}}
2020-07-28T14:05:29.348Z	INFO	instance/beat.go:310	Setup Beat: filebeat; Version: 7.8.0
2020-07-28T14:05:29.348Z	INFO	[index-management]	idxmgmt/std.go:183	Set output.elasticsearch.index to 'filebeat-7.8.0' as ILM is enabled.
2020-07-28T14:05:29.348Z	INFO	eslegclient/connection.go:97	elasticsearch url: http://mydomain.org:9200
2020-07-28T14:05:29.348Z	INFO	[publisher]	pipeline/module.go:113	Beat name: server9.cazillo.net
2020-07-28T14:05:29.392Z	INFO	instance/beat.go:463	filebeat start running.
2020-07-28T14:05:29.393Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-07-28T14:05:29.393Z	INFO	registrar/registrar.go:145	Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json
2020-07-28T14:05:29.393Z	INFO	registrar/registrar.go:152	States Loaded from registrar: 27
2020-07-28T14:05:29.393Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 0
2020-07-28T14:05:29.424Z	INFO	log/input.go:152	Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log* /var/log/httpd/access_log*]
2020-07-28T14:05:29.424Z	INFO	log/input.go:152	Configured paths: [/var/log/apache2/error.log* /var/log/httpd/error_log*]
2020-07-28T14:05:29.454Z	INFO	log/input.go:152	Configured paths: [/var/log/auth.log* /var/log/secure*]
2020-07-28T14:05:29.454Z	INFO	log/input.go:152	Configured paths: [/var/log/messages /var/log/audit/audit.log /var/log/firewalld]
2020-07-28T14:05:29.455Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2020-07-28T14:05:29.455Z	INFO	cfgfile/reload.go:164	Config reloader started
2020-07-28T14:05:29.464Z	INFO	log/input.go:152	Configured paths: [/var/log/apache2/access.log* /var/log/apache2/other_vhosts_access.log* /var/log/httpd/access_log*]
2020-07-28T14:05:29.479Z	INFO	log/input.go:152	Configured paths: [/var/log/apache2/error.log* /var/log/httpd/error_log*]
2020-07-28T14:05:29.479Z	INFO	eslegclient/connection.go:97	elasticsearch url: http://mydomain.org:9200
2020-07-28T14:05:29.706Z	INFO	[esclientleg]	eslegclient/connection.go:306	Attempting to connect to Elasticsearch version 7.8.0
2020-07-28T14:05:29.800Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/httpd/access_log
2020-07-28T14:05:29.800Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/httpd/error_log
2020-07-28T14:05:30.443Z	INFO	log/input.go:152	Configured paths: [/var/log/auth.log* /var/log/secure*]
2020-07-28T14:05:30.451Z	INFO	log/input.go:152	Configured paths: [/var/log/messages /var/log/audit/audit.log /var/log/firewalld]
2020-07-28T14:05:30.451Z	INFO	eslegclient/connection.go:97	elasticsearch url: http://mydomain.org:9200
2020-07-28T14:05:30.508Z	INFO	[esclientleg]	eslegclient/connection.go:306	Attempting to connect to Elasticsearch version 7.8.0
2020-07-28T14:05:30.559Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
2020-07-28T14:05:30.561Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/secure
2020-07-28T14:05:30.562Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/messages
2020-07-28T14:05:30.562Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/audit/audit.log
2020-07-28T14:05:30.801Z	INFO	[publisher_pipeline_output]	pipeline/output.go:144	Connecting to backoff(elasticsearch(http://mydomain.org:9200))
2020-07-28T14:05:30.802Z	INFO	[publisher]	pipeline/retry.go:221	retryer: send unwait signal to consumer
2020-07-28T14:05:30.802Z	INFO	[publisher]	pipeline/retry.go:225	  done
2020-07-28T14:05:30.860Z	INFO	[esclientleg]	eslegclient/connection.go:306	Attempting to connect to Elasticsearch version 7.8.0
2020-07-28T14:05:30.930Z	INFO	[license]	licenser/es_callback.go:51	Elasticsearch license: Basic
2020-07-28T14:05:30.953Z	INFO	[esclientleg]	eslegclient/connection.go:306	Attempting to connect to Elasticsearch version 7.8.0
2020-07-28T14:05:31.017Z	INFO	[index-management]	idxmgmt/std.go:259	Auto ILM enable success.
2020-07-28T14:05:31.042Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2020-07-28T14:05:31.042Z	INFO	[index-management]	idxmgmt/std.go:272	ILM policy successfully loaded.
2020-07-28T14:05:31.042Z	INFO	[index-management]	idxmgmt/std.go:405	Set setup.template.name to '{filebeat-7.8.0 {now/d}-000001}' as ILM is enabled.
2020-07-28T14:05:31.042Z	INFO	[index-management]	idxmgmt/std.go:410	Set setup.template.pattern to 'filebeat-7.8.0-*' as ILM is enabled.
2020-07-28T14:05:31.042Z	INFO	[index-management]	idxmgmt/std.go:444	Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.8.0 {now/d}-000001} as ILM is enabled.
2020-07-28T14:05:31.042Z	INFO	[index-management]	idxmgmt/std.go:448	Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-07-28T14:05:31.068Z	INFO	template/load.go:89	Template filebeat-7.8.0 already exists and will not be overwritten.
2020-07-28T14:05:31.068Z	INFO	[index-management]	idxmgmt/std.go:296	Loaded index template.
2020-07-28T14:05:31.095Z	INFO	[index-management]	idxmgmt/std.go:307	Write alias successfully generated.
2020-07-28T14:05:31.225Z	INFO	[publisher_pipeline_output]	pipeline/output.go:152	Connection to backoff(elasticsearch(http://mydomain.org:9200)) established
14

@prophoto I would recommend you use the Grok Debugger in Kibana (in Kibana's main menu go to the Management section of the menu > Dev Tools > Grok Debugger).

To find the Grok patterns in use by the Elasticsearch Ingest Node's Grok Processor which is part of an ingest pipeline set up by Filebeats, visit Kibana's Stack Management > Ingest Node Pipelines > filebeat-7.8.0-apache-access-pipeline. Clicking on it will bring up a Processors section that includes the Grok Processor definition and the patterns used by default to match Apache's log lines. As an alternative you can perform a "GET /_ingest/pipeline/filebeat-7.8.0-apache-access-pipeline`" to retrieve the JSON describing the pipeline configuration. Note that there's a separate pipeline for the error logs.

Also note that when using the Grok Debugger you will need to un-escape the Grok patterns that are included in the ingest pipeline's Grok Processor JSON definition, i.e. replace \" with just " and \\ with just \

Example: the first Grok pattern in filebeat-7.8.0-apache-access-pipeline is properly escaped as part of the JSON document describing the pipeline configuration:

%{IPORHOST:destination.domain} %{IPORHOST:source.ip} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-)?\" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-)( \"%{DATA:http.request.referrer}\")?( \"%{DATA:user_agent.original}\")?

In Grok Debugger you would test the following pattern:

%{IPORHOST:destination.domain} %{IPORHOST:source.ip} - %{DATA:user.name} \[%{HTTPDATE:apache.access.time}\] "(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-)?" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-)( "%{DATA:http.request.referrer}")?( "%{DATA:user_agent.original}")?

@tactics In the parallel discussion "Filebeat versions from 7.0 - 7.8 fail to create alias field mappings for majority of modules" Elastic's Marcin Tojek pointed out the documentation where the creation of aliases is mentioned as a compatibility layer during the transition to ECS: Upgrade from 6.x to 7.x. The list of field renames can be found in Breaking changes in 7.0.

15

Thanks for your help on this @b0le. I was able to pull the pipeline and started working with the Grok pattern debugger. First question, do you happen to know the proper term for %D and %T in? I don't need %T since its in seconds, I can probably just use microseconds %D and send notifications based on that. Here is my full Logformat in Apache.

LogFormat "%{Host}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{SSL_PROTOCOL}x\" %D %T" combined

Next question is once I get the proper Grok pattern, how do I apply it?

16

Bump

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.