Need help with setting up Apache2 to monitor the session time on every website

Elastic Stack Beats
1

Hi,

I'm new to this Kibana/logstash stuff, and due to the current situation, so I'm seeking some help here.

My objective:

Setting up Apache2 on Kibana/Logstash server so that I can get the following information from the web client that runs Apache/httpd, and able to send that information to Kibana (sys log), as well as generating a report or something.

  • User login and logout time on website

I've done some research but still unable to get it configured correctly, I managed to get the ingest-user-agent and ingest-geoip.html installed.

On the Kibana's interface, I'm able to see Filebeat -> apache2.access.agent and other apache modules on the list from "Add a filter" but when i tried to use visualize, the information is not showing on the web.

What I'd like to know is:
1). Do I need to setup anything on agent? I can see agents running on Logstash.
2). Can someone give me the config that I need to use for filebeat, logstash etc?

I'd appreciate if someone can help me. Please let me know if you have any questions or concerns.

Best Regards,
Lwengsheng

I've done checking the config i don't see anything wrong:
[root@m filebeat]# /usr/bin/filebeat.sh -configtest -e
2018/08/27 02:32:07.371161 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018/08/27 02:32:07.371203 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.2
2018/08/27 02:32:07.371229 publish.go:228: WARN Support for loading more than one output is deprecated and will not be supported in version 6.0.
2018/08/27 02:32:07.371260 metrics.go:23: INFO Metrics logging every 30s
2018/08/27 02:32:07.371372 output.go:258: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2018/08/27 02:32:07.372128 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/filebeat/filebeat.template-es2x.json
2018/08/27 02:32:07.372871 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /etc/filebeat/filebeat.template-es6x.json
2018/08/27 02:32:07.373560 client.go:128: INFO Elasticsearch url: http://localhost:9200
2018/08/27 02:32:07.373595 outputs.go:108: INFO Activated elasticsearch as output plugin.
2018/08/27 02:32:07.373655 logstash.go:90: INFO Max Retries set to: 3
2018/08/27 02:32:07.373709 outputs.go:108: INFO Activated logstash as output plugin.
2018/08/27 02:32:07.373804 publish.go:300: INFO Publisher name: m
2018/08/27 02:32:07.374080 async.go:63: INFO Flush Interval set to: 1s
2018/08/27 02:32:07.374099 async.go:64: INFO Max Bulk Size set to: 50
2018/08/27 02:32:07.374128 async.go:63: INFO Flush Interval set to: 1s
2018/08/27 02:32:07.374144 async.go:64: INFO Max Bulk Size set to: 2048
Config OK

2

If anyone can share what is the easiest way to setup

1). What file do i have to edit
2). What is the parameter I need to use
3). Do i need to add anything on the client? or just the server?
4). What service do i need to restart
5). How to debug the error for Apache
6). How to run a test

Thanks!

3

@tylerjl

4

(following up from a personal DM)

You mention that you've used the Filebeat Apache2 module to attempt to collect logs, but that no data is showing up in the Visualize or Discover tabs. The first item to debug here would be to ensure that logs/documents are being correctly indexed.

  • Does your elasticsearch cluster have any indices created? Check a simple API endpoint like _cat/indices to see what indices are in your cluster.
  • Are there any errors or connectivity problems in Filebeat's logs?
5

Can you show me how to check the simple API endpoint? also I'm seeing errors like File is inactive: /data/device_logs/....-sys.log. Closing because close_inactive of 5m0s reached, is this what you're talking about?

I honestly speaking not sure what file I need to configure, in order to achieve the objective, I did add the following on filebeat configuration file filebeat.yml:

  • /var/log/httpd/access.log
    input_type: log
    document_type: apache-access
    fields_under_root: true

I'm currently on the Kibana server, I can see output on Logstash, but not on Filebeat, however I can see Apache modules on Filebeat, not Logstash, I believe the connection is working because I'm seeing result on Logstash, just not on Filebeat.

Thanks Tyler for all your help!

6

Can you show me how to check the simple API endpoint?

I mention this just so that you can confirm whether Filebeat is actually indexing any logs. This can be queried through a simple curl command against your Elasticsearch cluster:

$ curl http://<elasticsearch hostname or IP>:9200/_cat/indices

Or you can use GET _cat/indices from the Kibana Dev Tools page in the sidebar.

Closing because close_inactive of 5m0s reached, is this what you're talking about?

I mention Filebeat logs because if Filebeat is unable to index documents into Elasticsearch, you may see errors about connectivity come up in Filebeat's logs - they may be buried somewhere in the log file, though.

I'm currently on the Kibana server, I can see output on Logstash, but not on Filebeat, however I can see Apache modules on Filebeat, not Logstash, I believe the connection is working because I'm seeing result on Logstash, just not on Filebeat.

So is there another, separate instance of Logstash running somewhere in your infrastructure indexing documents into Elasticsearch, or are you sending Filebeat logs to Logstash first? Getting an idea of your flow of logs can help clear up which services are talking and maybe help find what's happening.

I did add the following on filebeat configuration file filebeat.yml

If you find yourself manually specifying log file paths in Filebeat's config, you may not be using the Filebeat Apache2 module correctly - the module will already understand Apache log paths, format, etc. so many of these configuration lines aren't 100% necessary if you're using the Apache2 Filebeat module.

7

Here's the output:
[root@m ~]# curl http://:9200/_cat/indices
-bash: elasticsearch: No such file or directory
[root@m ~]#

If you find yourself manually specifying log file paths in Filebeat's config, you may not be using the Filebeat Apache2 module correctly - the module will already understand Apache log paths, format, etc. so many of these configuration lines aren't 100% necessary if you're using the Apache2 Filebeat module. -> how can i tell what mistake i made and correct it? Can you show me the correct config file?

8

The example curl command wasn't meant to be used literally with <elasticsearch hostname or IP> - you need to know where Elasticsearch is running in order to interact with Elasticsearch's REST API.

Giving you an exact config file won't work, since your environment will almost certainly have differences, such as what the Elasticsearch REST API endpoint will be.

I think your best bet would be to familiarize yourself with your current environment a bit more - for example, where Elasticsearch is being hosted, whether you have API access to Elasticsearch's rest endpoint, et cetera - and start from zero in the Filebeat documentation to begin with a clean environment and follow the steps to configure Filebeat from the beginning. The only steps necessary to get Filebeat up and running with the Apache2 module should basically be to a) install Filebeat, b) enable the Apache2 module, c) point Filebeat at Elasticsearch in the Filebeat configuration file, and d) start Filebeat. It sounds like you may need to get a little more familiar with how you're interacting with Elasticsearch in your environment if you're not sure how to reach Elasticsearch in the first place.

9

My bad, it's installed on the local server, I should be paying more attention to your command, here's the output:

[footk@m1 ~]$ curl http://127.0.0.1:9200/_cat/indices
yellow open logstash-2018.08.18 uSvduMs7RdCM07tb_6fSqA 5 1 172653704 0 109.1gb 109.1gb
yellow open .kibana so9lsWQ_Qfmh21Skt82bOg 1 1 69 31 151.9kb 151.9kb
yellow open logstash-2018.08.30 Aqs-k_LjR4iaE85czz44bQ 5 1 46363856 0 12.4gb 12.4gb
yellow open logstash-2018.09.01 XOWMI0-YSySkRI41WnNR8A 5 1 61337070 0 15.8gb 15.8gb
yellow open logstash-2018.08.10 IbQ8cJjvSRmmDTtEt8TxmQ 5 1 170091564 0 109.4gb 109.4gb
yellow open logstash-2018.08.22 hohTjfXnS4S_jPgts1d70g 5 1 176667916 0 113.9gb 113.9gb
yellow open logstash-2018.09.05 xXVYWowAQ2S4y1NsT4qx5A 5 1 364033358 0 230.9gb 230.9gb
yellow open logstash-2018.08.27 AlsMBfC1Qf62AhnbzeEepw 5 1 8171803 0 3gb 3gb
yellow open logstash-2018.08.11 seG80D4zTc-qmYsDrncGew 5 1 167980096 0 108.3gb 108.3gb
yellow open logstash-2018.08.26 xjS25Hj_RKW99d2wXVc0pw 5 1 168308785 0 106.3gb 106.3gb
yellow open logstash-2018.08.23 sDXnKk4XRf-JnpQXlRMRaw 5 1 172967264 0 111.3gb 111.3gb
yellow open logstash-2018.08.31 uX0RAO68TyqaOwNMisA-UA 5 1 60309482 0 15.7gb 15.7gb
yellow open logstash-2018.08.29 RX5gDGYRRIOVdUY9QCzD8Q 5 1 48849518 0 12.8gb 12.8gb
yellow open logstash-2018.08.28 CsLhYZf9R7KH7nU_g25z9w 5 1 12609438 0 3.5gb 3.5gb
yellow open logstash-2018.08.21 CX2PtRHTQQWN8gs56KPFqg 5 1 200433371 0 123.5gb 123.5gb
yellow open logstash-2018.08.14 _2lSSi6_Qom48qOeFCQhvQ 5 1 167109138 0 108.7gb 108.7gb
yellow open logstash-2018.08.25 KDj7QeemRwih9UPYqeJgYQ 5 1 169707402 0 108.5gb 108.5gb
yellow open logstash-2018.08.24 ldyafLhWQPGxCx1bp4Fv5g 5 1 171012594 0 109.9gb 109.9gb
yellow open logstash-2018.08.08 H75bHi5nSZqdu7HMACeBMA 5 1 168203721 0 108.6gb 108.6gb
yellow open logstash-2018.08.12 gDQoiK59RVuN6u7_xXkFEg 5 1 166978051 0 107.3gb 107.3gb
yellow open filebeat-2018.08.27 BYrJ6i4xSMS4eOTFBLpsjQ 5 1 3119207 0 872mb 872mb
yellow open logstash-2018.08.13 ivMW7GdvT3Cue5_xhGa3Xw 5 1 168752516 0 109.5gb 109.5gb
yellow open logstash-2018.09.06 Rv5xteO3Sd6SRXvgzACZ2Q 5 1 438259709 0 272.3gb 272.3gb
yellow open logstash-2018.09.02 mjEJG1d5RuSmQ6spY7iJYw 5 1 118297916 0 63.7gb 63.7gb
yellow open logstash-2018.08.15 a9kRzSuTQgyLMstEXbcMUA 5 1 164489000 0 106.7gb 106.7gb
yellow open logstash-2018.09.04 Tjdw039qT8-Ux_ir-i0r6A 5 1 326567234 0 214gb 214gb
yellow open logstash-2018.09.07 I7HE2jvmQJWuXln6WMlTGA 5 1 204646 0 245.4mb 245.4mb
yellow open logstash-2018.08.20 Vp9D6BXZRTWpXGj4mpnWkA 5 1 176992747 0 113.9gb 113.9gb
yellow open logstash-2018.08.16 WI0PoOxaTnmCoVoPMuBJ9A 5 1 193719323 0 114.8gb 114.8gb
yellow open logstash-2018.08.09 BiMLjJ81R3iWfSTKyc3_ug 5 1 166917096 0 108.2gb 108.2gb
yellow open logstash-2018.09.03 dkx2re7kQxKkwwVyT3zD1w 5 1 327102106 0 151.3gb 151.3gb
yellow open logstash-2018.08.17 cvAivaupQ-CekpuUoPLsRw 5 1 185266610 0 113.6gb 113.6gb
yellow open logstash-2018.08.19 ywPumxAfTwC6F9jeA6jzLA 5 1 172353772 0 108.9gb 108.9gb

Can you tell me what's the next step?

10
  1. Filebeat is already installed
  2. Apache2 module should be enabled as below:
    /usr/share/filebeat/module/apache2
    /usr/share/filebeat/module/apache2/access
    /usr/share/filebeat/module/apache2/access/config
    /usr/share/filebeat/module/apache2/access/config/access.yml
    /usr/share/filebeat/module/apache2/access/ingest
    /usr/share/filebeat/module/apache2/access/ingest/default.json
    /usr/share/filebeat/module/apache2/access/manifest.yml

3). Point filebeat at Elasticsearch in filebeat configuration - I believe i haven't done this step correctly, can you walk me though?

4). Filebeat is already started

Tyler, I'm not familiar how the Elastic search is being setup, that's why i'm asking in this forum, none of the person in the team knows Kibana, if someone knew how to set it up i probably won't seek for help here. I'd appreciate if you can point to me what i can do better or wrong to get it setup correctly.

11

First, I'd check to ensure that the Apache2 module is enabled:

$ filebeat modules list

And make sure that the output indicates that the apache2 module is enabled.

Pointing Filebeat at Elasticsearch is a matter of setting the right value in filebeat.yml, which would be something like the following:

output.elasticsearch:
  hosts: ["localhost:9200"]

Or, if your Elasticsearch instance is not at localhost:9200, the correct URI.

If filebeat instances are still not being created in Elasticsearch, there will be messages indicating why that isn't the case in Filebeat's logs, which are typically under /var/log/ if you've installed Filebeat using the Linux distribution packages (which I assume is the case).

13

Thanks for getting back to me Tyler, for some reason I'm not able to find the filebeat command, looks like the .sh works, just fyi filebeat version is 5.6+ not sure if this is the reason:

[root@ filebeat]# /usr/bin/filebeat.sh -modules list
Exiting: Error getting filesets for module list: open /usr/share/filebeat/module/list: no such file or directory

Tried to add the following on filebeat.yml but nothing happen:
filebeat.modules:

  • module: nginx
  • module: mysql
  • module: system
  • module: apache2

filebeat.config.modules:
enabled: true
path.home: /usr/share/filebeat/module/*/*yml

It will just keep giving me the error:

As far as i know, i can see some modules on the following path:
[root@m filebeat]# find / | grep apache2
find: ‘/proc/31960/task/27475’: No such file or directory
/root/apache2.conf.bak
/usr/share/filebeat/module/apache2
/usr/share/filebeat/module/apache2/access
/usr/share/filebeat/module/apache2/access/config
/usr/share/filebeat/module/apache2/access/config/access.yml
/usr/share/filebeat/module/apache2/access/ingest
/usr/share/filebeat/module/apache2/access/ingest/default.json
/usr/share/filebeat/module/apache2/access/manifest.yml
/usr/share/filebeat/module/apache2/error
/usr/share/filebeat/module/apache2/error/config
/usr/share/filebeat/module/apache2/error/config/error.yml
/usr/share/filebeat/module/apache2/error/ingest
/usr/share/filebeat/module/apache2/error/ingest/pipeline.json
/usr/share/filebeat/module/apache2/error/manifest.yml

Do you know what parameter should I be using to specify the module path? I also not able to find /usr/share/filebeat/module/list on filebeat.yml not sure where is pointing to.

Thanks for all your time and effort Tyler, you are awesome! Sorry if i ask some stupid questions.

14

I've added
output.elasticsearch:
hosts: ["localhost:9200"]

as well as path.config: /etc/filebeat/filebeat.yml on /etc/filebeat/filebeat.yml. It looks like filebeat is showing result now. However I'm still not able to see anything on Apache2 Dashboard.

I've also added the following on /etc/filebeat/filebeat.yml, is this the right thing to do?

  • module: apache2
    access:
    enabled: true
    var.paths: ["/var/log/httpd/ssl_access.log*"]
    error:
    enabled: true
    var.paths: ["/var/log/httpd/ssl_access.log*"]

Question, Is the parameters correct? If not can you provide the correct parameter? What else do i have to do in order to be able to capture user login time, logout time, password invalid etc from the Apache log?

Thanks!
Tuck

15

The way that Filebeat modules is by using some pre-configured files in the modules.d directory. If you take a look in that directory you'll see many files with .disabled at the end of their filenames. When you run filebeat modules enable apache2, essentially Filebeat is removing that disabled at the end of the filename so that it just ends in .yml, which the default Filebeat config will pull in by default - a very small filebeat.yml that would do this very simply might look like:

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
output.elasticsearch:
  hosts: ["localhost:9200"]

This means that Filebeat will pull in any files ending with .yml in your modules.d directory, and send everything to localhost:9200.

You mention some log fields like login time, logout time, password invalid, etc. that you're trying to capture. Please be aware that the Apache2 module collects default Apache logs in the format that Apache logs by default, which doesn't include granular events like the ones you're describing (i.e., whether a password is invalid). Although the pipeline the module uses to parse log lines will happily parse things like response codes and URL paths (for example, the module will parse whether /login returns 500), it will not parse custom application logs by default.

16

When I run filebeat module list, it hangs and don't give me any error in the output, is there anything i can do to resolve this?

[root@m ~]# /usr/bin/filebeat.sh modules enable apache2
------ > waiting here,.

Please bear with me, I can't run filebeat modules enable apache it will tell me the command is not found
^C[root@m~]# filebeat modules enable apache
-bash: filebeat: command not found

Can you advise where should i place the default.json at?

17

I'm actually fine with the default format of the Apache log, or use the custom parse modules, just need to know where i can place those. For now my issue is everytime i run filebeat modules enable apache2, it seems to be hang, and is there anyway to debug this?

18

@Lee_Weng_Sheng it's interesting you can't run the filebeat command. How did you install the filebeat on the host you're working on? If you installed it via the Debian package, can you show the results of dpkg -L filebeat | grep bin, and if you installed the package via the rpm package, the output of rpm -ql filebeat | grep bin?

19

Thanks for the prompt response, here's the output:

[root@ ~]# rpm -ql filebeat | grep bin
/usr/bin/filebeat.sh
/usr/share/filebeat/bin/filebeat
/usr/share/filebeat/bin/filebeat-god

I have no idea why i can't run the filebeat, i did research on how to debug the config but i can't find the problem, as far as i know, filebeat.sh is the one I'd use to run to check the config,

[root@m ~]# /usr/share/filebeat/bin/filebeat -modules list
filebeat2018/09/14 01:35:47.865184 beat.go:346: CRIT Exiting: error loading config file: stat filebeat.yml: no such file or directory
Exiting: error loading config file: stat filebeat.yml: no such file or directory - how can i resolve this error? I found filebeat.yml on /etc/filebeat, how can i know where the default filebeat.yml is located?

[root@m ~]# /usr/share/filebeat/bin/filebeat-god -modules list
/usr/share/filebeat/bin/filebeat-god: invalid option -- 'm'
Use: god [options] [--] program [arguments]
Options:
-h --help show this help and exit
-v --version show version and exit
-f --foreground run in foreground
-n --nohup make the program immune to SIGHUP
-l --logfile FILE write the program's stdout and stderr to FILE
-p --pidfile FILE write pid to FILE
-r --rundir DIR switch to DIR before executing the program
-u --user USER switch to USER before executing the program
-g --group GROUP switch to GROUP before executing the program

The program's output go to a blackhole if no logfile is set.
Log files are recycled on SIGHUP.

[root@m ~]# /usr/bin/filebeat.sh -configtest -e
2018/09/14 01:39:11.413644 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018/09/14 01:39:11.413684 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.2
2018/09/14 01:39:11.413751 logstash.go:90: INFO Max Retries set to: 3
2018/09/14 01:39:11.413794 metrics.go:23: INFO Metrics logging every 30s
2018/09/14 01:39:11.413826 outputs.go:108: INFO Activated logstash as output plugin.
2018/09/14 01:39:11.413971 publish.go:300: INFO Publisher name: mh
2018/09/14 01:39:11.414275 async.go:63: INFO Flush Interval set to: 1s
2018/09/14 01:39:11.414294 async.go:64: INFO Max Bulk Size set to: 2048
Config OK

20

Can someone please help?

21

I really need some help here, can someone please tell me what else should I look at, what type of setup I must have in order to achieve the objective?