Metricbeat Apache Info

[elJoeyJojo]

I've setup the metricbeat config on 3 servers (1 of them is the kibana/elasticsearch host) and i am able to see the logs/dashboards for system overall, cpu, memory etc.. I enabled the apache and mysql module on all 3 servers and when i go to the Discover menu in Kibana and search with index metric* I see every detail. Now the MySQL doesn't show info 'cause I didn't specify the root/secret for the the DB and that's fair but the Apache module is enabled yet I don't see any info the Apache Dashboards. If I go into the Discover area I see the #_score values for apache.* like so:

But then when I go to Metricbeat -Apache I see nothing:

I guess what i am trying to find out is why is the [Metricbeat Apache] tied into the # _score sets for apache? for example i see in Discover apache.status.* but then when I look into the dashboards it's empty 'cause it's looking for apache.status.load.5 apache.status.cpu.load etc... and when I look for those in the Discover section I can't find them. So it seems like some data is being sent but other data is not being sent. I enabled the module on that one particular server so why is it getting some stuff but not indexing others?

As you can see the by the next snapshot, the Dashboards are looking for apache.status.total* and the Discovery is nothing showing anything under that.

Any help on this would be greatly appreciated. THANK YOU

[exekias (Carlos Pérez Aradros)]

Hi @elJoeyJojo,

First of all, we have to determine if metrics are being correctly retrieved, could you paste the logs you get out of Metricbeat? If there is an error while getting Apache status it should be present there.

[elJoeyJojo]

ok so i went through it, but i'm confused on a few things with elasticsearch/kibana
i installed the filebeat on both the elasticsearch/kibana server and the remote host i wanna pull those apache logs from, i enabled the modules on both the elasticsearch/kibana and the remote host. When I check the output of the filebeat log on the remote host I see this:

2018-01-29T10:36:28-08:00 INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-01-29T10:36:28-08:00 INFO Beat UUID: 2fe7e6d2-2ee3-4138-b875-cd89f12078f9
2018-01-29T10:36:28-08:00 INFO Metrics logging every 30s
2018-01-29T10:36:28-08:00 INFO Setup Beat: filebeat; Version: 6.1.2
2018-01-29T10:36:28-08:00 INFO Elasticsearch url: http://cdyvrlelk001:9200
2018-01-29T10:36:28-08:00 INFO Beat name: dev
2018-01-29T10:36:28-08:00 INFO filebeat start running.
2018-01-29T10:36:28-08:00 INFO Registry file set to: /var/lib/filebeat/registry
2018-01-29T10:36:28-08:00 INFO Loading registrar data from /var/lib/filebeat/registry
2018-01-29T10:36:28-08:00 INFO States Loaded from registrar: 14
2018-01-29T10:36:28-08:00 INFO Loading Prospectors: 1
2018-01-29T10:36:28-08:00 INFO Starting Registrar
2018-01-29T10:36:28-08:00 INFO Starting prospector of type: log; ID: 11378805102105908354
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/name.cd.local/access_ssl.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/name.cd.local/access_ssl.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/name.cd.local/error.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/name/access.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/api.name.cd.local/access.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/name.cd.local/access.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/name.cd.local/error_ssl.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/name.cd.local/access.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/name.cd.local/error_ssl.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/name.cd.local/error.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2name/error.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/apache2/api.name.cd.local/error.log
2018-01-29T10:36:28-08:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018-01-29T10:36:28-08:00 INFO Config reloader started
2018-01-29T10:36:28-08:00 INFO Connected to Elasticsearch version 6.1.2
2018-01-29T10:36:28-08:00 INFO Template already exists and will not be overwritten.
2018-01-29T10:36:28-08:00 INFO Starting 2 runners ...
2018-01-29T10:36:28-08:00 INFO Elasticsearch url: http://cdyvrlelk001:9200
2018-01-29T10:36:28-08:00 INFO Connected to Elasticsearch version 6.1.2
2018-01-29T10:36:28-08:00 INFO Starting prospector of type: log; ID: 17216501560277081620
2018-01-29T10:36:28-08:00 INFO Starting prospector of type: log; ID: 4128373258093383538
2018-01-29T10:36:28-08:00 INFO Elasticsearch url: http://cdyvrlelk001:9200
2018-01-29T10:36:28-08:00 INFO Connected to Elasticsearch version 6.1.2
2018-01-29T10:36:28-08:00 INFO Starting prospector of type: log; ID: 18044925928469627648
2018-01-29T10:36:28-08:00 INFO Starting prospector of type: log; ID: 8734454681295277600
2018-01-29T10:36:28-08:00 INFO Loading of config files completed.
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/auth.log
2018-01-29T10:36:28-08:00 INFO Harvester started for file: /var/log/syslog

I can see data in the Discover from that specific server hitting those specific "name.cd.local/error.log access.log etc... but when I go to the Dashboard of Filebeat Apache2 (which I enabled on the Elasticsearch/Kibana) and all I see is this:

which doesn't make sense 'cause the index-pattern is configured:

I already restarted all services so I'm confused

[elJoeyJojo]

anybody out there that can help me?

[elJoeyJojo]

i'll ask one last time, is there anybody out there who can help me?

[tylersmalley]

It looks like the index pattern was re-created at some point. Kibana objects have unique id's and the imported dashboards expect a certain ID to exist.

If there is nothing already using that index pattern, I would recommend deleting it and re-creating using the link on the dashboard. Alternatively, you could use the setup task again which will re-create the index pattern.

[elJoeyJojo]

hello Tyler,

I felt like I removed all the indexes a few dozen times and same thing, do I have to remove these from /var/lib/elasticsearch/nodes/0/indices ??

Also I'm confused on the tutorial https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-apache2.html which never bothers to mention if these commands are to be ran in the actual host or on the ELK server? for example: $ ./filebeat setup -e is this suppose to be run in the host or in the ELK server? I ran the $ ./filebeat -e and I see his output:

2018/02/08 16:07:46.789389 beat.go:436: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018/02/08 16:07:46.789470 metrics.go:23: INFO Metrics logging every 30s
2018/02/08 16:07:46.789507 beat.go:443: INFO Beat UUID: 2fe7e6d2-2ee3-4138-b875-cd89f12078f9
2018/02/08 16:07:46.789536 beat.go:203: INFO Setup Beat: filebeat; Version: 6.1.2
2018/02/08 16:07:46.789824 client.go:123: INFO Elasticsearch url: http://elk.domain.local:9200
2018/02/08 16:07:46.790144 module.go:76: INFO Beat name: dev
2018/02/08 16:07:46.790433 beat.go:276: INFO filebeat start running.
2018/02/08 16:07:46.790494 registrar.go:88: INFO Registry file set to: /var/lib/filebeat/registry
2018/02/08 16:07:46.790518 registrar.go:108: INFO Loading registrar data from /var/lib/filebeat/registry
2018/02/08 16:07:46.790859 registrar.go:119: INFO States Loaded from registrar: 26
2018/02/08 16:07:46.790880 crawler.go:48: INFO Loading Prospectors: 1
2018/02/08 16:07:46.791022 registrar.go:150: INFO Starting Registrar
2018/02/08 16:07:46.813931 prospector.go:87: INFO Starting prospector of type: log; ID: 11378805102105908354
2018/02/08 16:07:46.817588 harvester.go:215: INFO Harvester started for file: /var/log/apache2/acd.domain.local/access.log
2018/02/08 16:07:46.837134 crawler.go:82: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/02/08 16:07:46.837175 reload.go:127: INFO Config reloader started
2018/02/08 16:07:46.859835 reload.go:258: INFO Starting 1 runners ...
2018/02/08 16:07:46.859934 client.go:123: INFO Elasticsearch url: http://elk.domain.local:9200
2018/02/08 16:07:46.862608 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018/02/08 16:07:46.865754 prospector.go:87: INFO Starting prospector of type: log; ID: 4128373258093383538
2018/02/08 16:07:46.865779 prospector.go:87: INFO Starting prospector of type: log; ID: 17216501560277081620
2018/02/08 16:07:46.865788 reload.go:219: INFO Loading of config files completed.
2018/02/08 16:07:47.819833 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018/02/08 16:07:47.821273 load.go:73: INFO Template already exists and will not be overwritten.

I find the documentation although very well put together sometimes very unclear on to where and when to do it. I could setup the metricbeat just fine and that worked but the filebeat although it shows in the "Discover" section the Dashboard just won't work

[tylersmalley]

When I mentioned removing the index pattern, I was referring to the index pattern in the Kibana UI. Goto Management > Index Patterns. Choose the filebeat-* pattern and delete it.

Also, per the linked tutorial, you want to run ./filebeat setup -e. It will setup the index pattern.

[elJoeyJojo]

hey, I already removed it from Kibana various times, I create it which gives me a long list of indices like so:

clearly it sees the filebeat logs for each day, so I create it typing in "filebeat-*"
it then asks me for a Time Filter Field Name: and I use @timestamp

after this I can see clearly that my index pattern is created:

I then confirm in Discover that the logs are coming through because I pointed it to pickup logs from the /var/log/apache2/acd.cd.local/access.log

Go to my Dashboards and choose the [Filebeat Apache2] Access and Error Logs

and voila!

I'm confused as to why this keeps telling me to re-create it also it saids "click here" but nothing clicks or is clickable. I did setup X-Pack, would that have anything to do with it?

[elJoeyJojo]

it's not just that one, looking at [Filebeat System] Syslog dashboard shows nothing as well but the logs clearly say otherwise:

2018-02-08T09:11:44-08:00 INFO Harvester started for file: /var/log/auth.log
2018-02-08T09:11:44-08:00 INFO Harvester started for file: /var/log/syslog

I don't know this ELK thing is great but it's way too confusing to setup when it's so simple in other open source projects so that I can view logs from wherever i want. Those Apache2 ones for GeoLocation looks awesome but it's too difficult to setup. I can't even get a simple Syslog Dashboard to show information 'cause everything saids there is no index-pattern

[elJoeyJojo]

I went into the Visualize section and looked for all apache/nginx visualizations available, and when I clicked on some this shows up (look below) should I delete the visualization? seems like the index: filebeat-* is the correct one I made. If I delete it will it re-create it if I reload dashboards?