Does auditbeat require logstash?


(Guillaume Bettayeb) #1

Hi everyone,

Not sure what I am doing wrong here, but it seems to me that auditbeat is correctly configured although it does not send anything to ElasticSearch.

Here's my auditbeat logs:
2018/08/03 16:12:45.328929 beat.go:260: INFO auditbeat start running.
2018/08/03 16:12:45.329013 wrapper.go:98: DBG Starting Wrapper[name=audit, len(metricSetWrappers)=1]
2018/08/03 16:12:45.329082 wrapper.go:98: DBG Starting Wrapper[name=audit, len(metricSetWrappers)=1]
2018/08/03 16:12:45.329139 wrapper.go:164: DBG audit/kernel will start after 1.600942072s
2018/08/03 16:12:45.329166 wrapper.go:164: DBG audit/file will start after 4.738901871s
2018/08/03 16:12:46.930387 wrapper.go:172: DBG Starting metricSetWrapper[module=audit, name=kernel, host=]
2018/08/03 16:12:46.949965 audit_linux.go:163: INFO [audit.kernel] Deleted 184 pre-existing audit rules.
2018/08/03 16:12:47.116885 audit_linux.go:177: INFO [audit.kernel] Successfully added 184 of 184 kernel audit rules.
2018/08/03 16:12:47.117081 audit_linux.go:197: DBG [audit.kernel] audit status from kernel at start: status=&{Mask:0 Enabled:1 Failure:0 PID:708 RateLimit:0 BacklogLimit:8192 Lost:0 Backlog:0 FeatureBitmap:61 BacklogWaitTime:0}
2018/08/03 16:12:50.068380 wrapper.go:172: DBG Starting metricSetWrapper[module=audit, name=file, host=]
2018/08/03 16:13:15.307816 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4521792 beat.memstats.memory_alloc=2427960 beat.memstats.memory_total=11299032 libbeat.config.module.running=0 libbeat.output.type=elasticsearch libbeat.pipeline.clients=2 libbeat.pipeline.events.active=0
2018/08/03 16:13:45.307811 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4521792 beat.memstats.memory_alloc=2438136 beat.memstats.memory_total=11309208 libbeat.config.module.running=0 libbeat.pipeline.clients=2 libbeat.pipeline.events.active=0
2018/08/03 16:14:15.307892 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4521792 beat.memstats.memory_alloc=2448856 beat.memstats.memory_total=11319928 libbeat.config.module.running=0 libbeat.pipeline.clients=2 libbeat.pipeline.ev

that last line just goes on in a loop....

When I test ElasticSearch, I see that I don't get any logs going through:

[root@ELK auditbeat]# curl -XGET 'http://0.0.0.0:9200/auditbeat-*/_search?pretty'
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : 0.0,
"hits" : [ ]
}
}

However, the YAML file for auditbeat specifically says:

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["0.0.0.0:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

And as you can see, logstash is disabled. I've not bothered installing at as it is apparently not necessary(I don't have a large volume of servers right now, just testing)

So, is that possible to send all auditd logs straight to ElasticSearch or maybe I must install logstash if I want to use beats ?

Thanks,


(Andrew Kroh) #2

Logstash is not required by Auditbeat. It will work fine sending directly to Elasticsearch.

It does look like you are using an old (probably beta) version of Auditbeat. I highly recommend updating to the latest version

Change 0.0.0.0 inhosts: ["0.0.0.0:9200"] to the actual IP address or hostname of the Elasticsearch server (like localhost or 10.0.0.2).


(Guillaume Bettayeb) #3

Hey @andrewkroh, thank you for your reply ! :slight_smile: I tried using the proper IP too instead of the loopback interface, and it indeed helped getting rid of those messages. However, I still see no index in Kibana.

I believe there is something missing with my Filebeat configuration, because ES status seems fine but doesn't seem to be receiving anything from Filebeat:

> [root@ELK conf.d]# curl -X GET "10.1.5.20:9200/_cat/indices?v"
> health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
> green  open   .monitoring-kibana-6-2018.08.07 uWIyQPsDR02zd0ekWnBmrQ   1   0       3101            0    729.8kb        729.8kb
> green  open   .monitoring-kibana-6-2018.08.06 r8m6eU5nSJ6fJmLBxgJL_Q   1   0       2794            0    679.6kb        679.6kb
> green  open   .monitoring-es-6-2018.08.07     gARZQnUPQG21_W7J-QSAYg   1   0      28451           20     20.4mb         20.4mb
> green  open   .kibana                         1FBktM85SQOgnioT6_7EeQ   1   0          2            0     13.5kb         13.5kb
> green  open   .monitoring-es-6-2018.08.06     1ynD67aVRleYleWC_F5ORg   1   0      19565           15      7.4mb          7.4mb

And when I run Filebeat on its own, this is what I get:

[root@ELK filebeat]# filebeat -e -d "publish"
[..]
{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":5,"patch":1804,"codename":"Core"},"timezone":"BST","timezone_offset_sec":3600,"id":"e3a9e7e0b15e492287f42f616b614918"}}}
2018-08-07T10:05:12.650+0100 INFO [beat] instance/beat.go:761 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 36381, "ppid": 12877, "seccomp": {"mode":"disabled"}, "start_time": "2018-08-07T10:05:12.570+0100"}}}
2018-08-07T10:05:12.650+0100 INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.2
2018-08-07T10:05:12.650+0100 INFO elasticsearch/client.go:145 Elasticsearch url: http://10.1.5.20:9200
2018-08-07T10:05:12.651+0100 DEBUG [publish] pipeline/consumer.go:120 start pipeline event consumer
2018-08-07T10:05:12.652+0100 INFO pipeline/module.go:81 Beat name: ELK
2018-08-07T10:05:12.653+0100 INFO instance/beat.go:315 filebeat start running.
2018-08-07T10:05:12.653+0100 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-08-07T10:05:12.653+0100 INFO registrar/registrar.go:117 Loading registrar data from /var/lib/filebeat/registry
2018-08-07T10:05:12.653+0100 INFO registrar/registrar.go:124 States Loaded from registrar: 3
2018-08-07T10:05:12.653+0100 INFO crawler/crawler.go:48 Loading Inputs: 1
2018-08-07T10:05:12.653+0100 INFO log/input.go:118 Configured paths: [/var/log/*.log]
2018-08-07T10:05:12.653+0100 INFO input/input.go:88 Starting input of type: log; ID: 11204088409762598069
2018-08-07T10:05:12.654+0100 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 1
2018-08-07T10:05:12.654+0100 INFO cfgfile/reload.go:122 Config reloader started
2018-08-07T10:05:12.654+0100 INFO cfgfile/reload.go:214 Loading of config files completed.
2018-08-07T10:05:26.616+0100 INFO beater/filebeat.go:420 Stopping filebeat
2018-08-07T10:05:26.617+0100 INFO crawler/crawler.go:109 Stopping Crawler
2018-08-07T10:05:26.617+0100 INFO crawler/crawler.go:119 Stopping 1 inputs
2018-08-07T10:05:26.617+0100 INFO cfgfile/reload.go:217 Dynamic config reloader stopped
2018-08-07T10:05:26.617+0100 INFO input/input.go:122 input ticker stopped
2018-08-07T10:05:26.617+0100 INFO input/input.go:139 Stopping Input: 11204088409762598069
2018-08-07T10:05:26.617+0100 DEBUG [publish] pipeline/client.go:131 client: closing acker
2018-08-07T10:05:26.617+0100 DEBUG [publish] pipeline/client.go:133 client: done closing acker
2018-08-07T10:05:26.617+0100 DEBUG [publish] pipeline/client.go:137 client: cancelled 0 events
2018-08-07T10:05:26.617+0100 INFO crawler/crawler.go:135 Crawler stopped
2018-08-07T10:05:26.617+0100 INFO registrar/registrar.go:339 Stopping Registrar
2018-08-07T10:05:26.617+0100 INFO registrar/registrar.go:265 Ending Registrar
2018-08-07T10:05:26.622+0100 INFO [monitoring] log/log.go:132 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":30}},"total":{"ticks":50,"time":{"ms":56},"value":50},"user":{"ticks":20,"time":{"ms":26}}},"info":{"ephemeral_id":"0b4b9088-9c0f-4410-8c30-af493256fab2","uptime":{"ms":13985}},"memstats":
[...]
2018-08-07T10:05:26.622+0100 INFO [monitoring] log/log.go:110 Stopping metrics logging.
2018-08-07T10:05:26.622+0100 INFO instance/beat.go:321 filebeat stopped.

I am suspecting an issue with the prospector configuration? If so, I am not sure how/where to set it?

Thanks for your help again,

Guillaume