Apache2 module | Filebeat - No results found

[alexus]

I'm trying to Configure the module and even though I see events inside of my %{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd} whenever I open Filebeat Apache2 Dashboard via Kibana all I see is:

No results found

# cat /etc/filebeat/modules.d/apache2.yml 
- module: apache2
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/*access.log*"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/*error.log*"]
# 
# ls -ld /var/log/apache2/*access.log* | head -1
-rw-r----- 1 root adm  764580 Feb 21 16:37 /var/log/apache2/access.log
# ls -ld /var/log/apache2/*error.log* | head -1
-rw-r----- 1 root adm 725 Feb 21 06:25 /var/log/apache2/error.log
# 

recap: events makes into elasticsearch, however I'm unable to see any via Kibana's dashboard...

Please advise.

[tudor (Tudor Golubenco)]

Which time range you have set. Note that the Filebeat module parses the timestamps from the logs, so they can be quite a bit old.

If increasing the time rage to 1y or so doesn't help, try:

• checking the data in the Discover tab.
• If nothing is in the Discover tab as well, try querying the data with Kibana Console

[alexus]

• I do see events (in filebeat-*) via Discover even within Last 15 minutes.

one of the doc:

{
  "_index": "filebeat-6.2.1-2018.02.24",
  "_type": "doc",
  "_id": "4bpxxmEB9U9uoDsEohOO",
  "_version": 1,
  "_score": null,
  "_source": {
    "host": "web",
    "@timestamp": "2018-02-24T06:13:10.303Z",
    "@version": "1",
    "prospector": {
      "type": "log"
    },
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "fileset": {
      "name": "access",
      "module": "apache2"
    },
    "offset": 1771291,
    "beat": {
      "name": "web",
      "version": "6.2.1",
      "hostname": "web0"
    },
    "source": "/var/log/apache2/access.log",
    "message": "127.0.0.1 - - [24/Feb/2018:06:13:09 +0000] \"GET / HTTP/1.1\" 200 3315 \"-\" \"Go-http-client/1.1\""
  },
  "fields": {
    "@timestamp": [
      "2018-02-24T06:13:10.303Z"
    ]
  },
  "sort": [
    1519452790303
  ]
}

Please advise.

[tudor (Tudor Golubenco)]

The events are there but the messages are not parsed. It looks like you are sending these events through Logstash? Unfortunately it's currently not too easy to run Filebeat modules with Logstash, see: https://www.elastic.co/guide/en/logstash/current/filebeat-modules.html

[alexus]

@tudor I followed that when I set this up filebeat to work with logstash.

# filebeat test output
logstash: app11:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.142.0.6
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
# 

as you saw earlier events make all the way into elasticsearch, it's not shipping where it fails it's parsing...

any help with how address that would be appreciate it)

[tudor (Tudor Golubenco)]

If you followed that guide, then the parsing needs to be done by Logstash. Can you paste the Logstash config here?

[alexus]

ok, my missing part: Configuration Examples | Logstash Reference

Thanks!