I am trying to create a home lab. I have setup everything in ubuntu machine in vmware. The auditbeat is also in this machine. But I cannot see any data from auditbeat or even the host itself in security overview. Then I installed and configured winlogbeat in a windows machine but the result is same. It is worth mentioning that I have configurred x-pack also. But still unable. I have added some screenshots for your reference.
Hi,
Welcome to community
Can you check if your beats are able to send data to Elasticsearch not , Any error logs?
Hi,
I guess there is issue with both the implementations.
As you see, both your indices health are in yellow state. I guess, you are running a single instance of ES.
Hence, by default a primary and replica shards are created for an index, and your replica shard is unassigned. So, fix this issue as well by setting the no. of replicas : 0
Go to Dev Tools, and apply the below.
Note: Do only if your node is a single node.
PUT /*/_settings
{
"index" : {
"number_of_replicas":0
}
}
This applies to all the indexes available in your cluster.
Also, check the audit event and winlogbeat logs if they are throwing any errors.
This is the output I am getting.
#! this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .fleet-artifacts-7, .fleet-enrollment-api-keys-7, .fleet-policies-7, .kibana_7.17.7_001, .kibana_task_manager_7.17.7_001, .security-7, .tasks, .transform-internal-007], but in a future major version, direct access to system indices will be prevented by default
#! Overriding settings on system indices: [.transform-internal-*] -> [index.number_of_replicas], [.fleet-artifacts*] -> [index.number_of_replicas], [.tasks*] -> [index.number_of_replicas], [.fleet-policies-[0-9]+*] -> [index.number_of_replicas], [.fleet-enrollment-api-keys*] -> [index.number_of_replicas]. This will not work in the next major version
{
"acknowledged" : true
}
Great. Now if you see, I guess your indices will be showing in Green as healthy.
If yes, one issue is fixed currently which I've seen in your screenshots.
Now, have you seen the logs of your beats if they're throwing any errors?
Yes they are green now. How to check the logs? I got one link that says cat api but I cannot find the api. I know I might be bothering you, but I am very new and a noob. Can you tell me how I can check the logs.
How have you installed your beats?
Have you enabled logging in your beats configuration?
Check the below links
I have setup the auditbeat, but no folder created in the /var/log path.
It depends on how you installed it.
Is it an RPM package or a .tar.gz file.
You can change the path to any path you have access to. Configure it according to your implementation.
You can check set-up section of those beats on how to implement everything.
thats the standard error log from auditbeat. I used the command auditbeat -e
{"log.level":"info","@timestamp":"2022-12-13T08:35:55.458+1100","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/auditbeat] Config path: [/etc/auditbeat] Data path: [/var/lib/auditbeat] Logs path: [/var/log/auditbeat]","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:35:55.489+1100","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: c2f143d3-0b3d-4c5f-93bc-3582013c46a7","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:35:58.495+1100","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.662+1100","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.689+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"auditbeat","system_info":{"beat":{"path":{"config":"/etc/auditbeat","data":"/var/lib/auditbeat","home":"/usr/share/auditbeat","logs":"/var/log/auditbeat"},"type":"auditbeat","uuid":"c2f143d3-0b3d-4c5f-93bc-3582013c46a7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.690+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"auditbeat","system_info":{"build":{"commit":"6d03209df870c63ef9d59d609268c11dfdc835dd","libbeat":"8.5.3","time":"2022-12-04T04:51:51.000Z","version":"8.5.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.690+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"auditbeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.18.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.690+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"auditbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-12-13T08:09:21+11:00","containerized":false,"name":"ubuntuMoh","ip":["127.0.0.1/8","::1/128","192.168.111.135/24","fe80::b8fd:f0bc:5755:f7b/64"],"kernel_version":"5.15.0-56-generic","mac":["00:0c:29:0a:f5:34"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.1 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":1,"codename":"jammy"},"timezone":"AEDT","timezone_offset_sec":39600,"id":"34c76a64533b48b583d63889415c4d2d"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.690+1100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"auditbeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/var/log","exe":"/usr/share/auditbeat/bin/auditbeat","name":"auditbeat","pid":11104,"ppid":4663,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-12-13T08:35:52.540+1100"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.691+1100","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: auditbeat; Version: 8.5.3","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.695+1100","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: http://192.168.111.135:9200","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.697+1100","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: ubuntuMoh","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.699+1100","log.logger":"auditd","log.origin":{"file.name":"auditd/audit_linux.go","file.line":107},"message":"auditd module is running as euid=0 on kernel=5.15.0-56-generic","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.702+1100","log.logger":"auditd","log.origin":{"file.name":"auditd/audit_linux.go","file.line":134},"message":"socket_type=multicast will be used.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:35:58.704+1100","log.logger":"cfgwarn","log.origin":{"file.name":"package/package.go","file.line":201},"message":"BETA: The system/package dataset is beta","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:35:58.910+1100","log.logger":"cfgwarn","log.origin":{"file.name":"host/host.go","file.line":201},"message":"BETA: The system/host dataset is beta","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:35:58.911+1100","log.logger":"cfgwarn","log.origin":{"file.name":"login/login.go","file.line":94},"message":"BETA: The system/login dataset is beta","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:35:58.911+1100","log.logger":"cfgwarn","log.origin":{"file.name":"process/process.go","file.line":146},"message":"BETA: The system/process dataset is beta","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:35:58.912+1100","log.logger":"cfgwarn","log.origin":{"file.name":"socket/socket_linux.go","file.line":127},"message":"BETA: The system/socket dataset is beta.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:35:58.931+1100","log.logger":"socket","log.origin":{"file.name":"socket/socket_linux.go","file.line":284},"message":"Setting up system/socket for kernel 5.15.0-56-generic","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:35:59.114+1100","log.logger":"socket","log.origin":{"file.name":"guess/guess.go","file.line":258},"message":"Running 17 guesses ...","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:01.643+1100","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:02.542+1100","log.logger":"socket","log.origin":{"file.name":"guess/guess.go","file.line":305},"message":"Guessing done after 3.427928547s","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:36:03.017+1100","log.logger":"cfgwarn","log.origin":{"file.name":"user/user.go","file.line":232},"message":"BETA: The system/user dataset is beta","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:03.023+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:03.023+1100","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"auditbeat start running.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:03.080+1100","log.logger":"auditd","log.origin":{"file.name":"auditd/audit_linux.go","file.line":272},"message":"No audit_rules were specified.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:07.658+1100","log.logger":"socket","log.origin":{"file.name":"afpacket/afpacket.go","file.line":152},"message":"Starting DNS capture.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:07.794+1100","log.logger":"socket","log.origin":{"file.name":"socket/socket_linux.go","file.line":228},"message":"Bootstrapped process table using /proc","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:07.794+1100","log.logger":"socket","log.origin":{"file.name":"socket/socket_linux.go","file.line":231},"message":"system/socket dataset is running.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:08.812+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(elasticsearch(http://192.168.111.135:9200))","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:09.023+1100","log.logger":"file_integrity","log.origin":{"file.name":"file_integrity/eventreader_fsnotify.go","file.line":99},"message":"Started fsnotify watcher","service.name":"auditbeat","file_path":["/etc","/usr/bin","/usr/sbin"],"recursive":false,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-13T08:36:10.887+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(http://192.168.111.135:9200)): 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:10.888+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(http://192.168.111.135:9200)) with 1 reconnect attempt(s)","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:36:13.259+1100","log.logger":"process","log.origin":{"file.name":"process/process.go","file.line":249},"message":"failed to hash executable /usr/share/filebeat/bin/filebeat for PID 2452: failed to hash file /usr/share/filebeat/bin/filebeat: hasher: file size 128979488 exceeds max file size","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:36:13.709+1100","log.logger":"process","log.origin":{"file.name":"process/process.go","file.line":249},"message":"failed to hash executable /usr/share/auditbeat/bin/auditbeat for PID 11104: failed to hash file /usr/share/auditbeat/bin/auditbeat: hasher: file size 110410816 exceeds max file size","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-13T08:36:14.553+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(http://192.168.111.135:9200)): 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:14.553+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(http://192.168.111.135:9200)) with 2 reconnect attempt(s)","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:17.167+1100","log.logger":"file_integrity","log.origin":{"file.name":"file_integrity/scanner.go","file.line":116},"message":"File system scan completed","service.name":"auditbeat","scanner_id":1,"took":8127488955,"file_count":2019,"total_bytes":210073233,"bytes_per_sec":25847249.275037616,"files_per_sec":248.41620962867248,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-13T08:36:18.606+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(http://192.168.111.135:9200)): 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:18.606+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(http://192.168.111.135:9200)) with 3 reconnect attempt(s)","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:36:24.519+1100","log.logger":"process","log.origin":{"file.name":"process/process.go","file.line":249},"message":"failed to hash executable /usr/share/filebeat/bin/filebeat for PID 2452: failed to hash file /usr/share/filebeat/bin/filebeat: hasher: file size 128979488 exceeds max file size","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T08:36:25.372+1100","log.logger":"process","log.origin":{"file.name":"process/process.go","file.line":249},"message":"failed to hash executable /usr/share/auditbeat/bin/auditbeat for PID 11104: failed to hash file /usr/share/auditbeat/bin/auditbeat: hasher: file size 110410816 exceeds max file size","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-13T08:36:33.023+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(http://192.168.111.135:9200)): 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:33.023+1100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":141},"message":"Attempting to reconnect to backoff(elasticsearch(http://192.168.111.135:9200)) with 4 reconnect attempt(s)","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T08:36:33.045+1100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"auditbeat","monitoring":{"metrics":{"auditd":{"received_msgs":8},"beat":{"cgroup":{"cpu":{"id":"vte-spawn-be2c2b49-db28-45dc-8def-8eeb47106021.scope"},"memory":{"id":"vte-spawn-be2c2b49-db28-45dc-8def-8eeb47106021.scope","mem":{"usage":{"bytes":455925760}}}},"cpu":{"system":{"ticks":4880,"time":{"ms":4880}},"total":{"ticks":6500,"time":{"ms":6500},"value":6500},"user":{"ticks":1620,"time":{"ms":1620}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":117},"info":{"ephemeral_id":"285d0285-872e-4ce9-aa1a-5428a5b58e9b","name":"auditbeat","uptime":{"ms":38185},"version":"8.5.3"},"memstats":{"gc_next":40661000,"memory_alloc":18322968,"memory_sys":59089098,"memory_total":196092928,"rss":144084992},"runtime":{"goroutines":67}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"read":{"bytes":2025},"type":"elasticsearch","write":{"bytes":1265}},"pipeline":{"clients":8,"events":{"active":2100,"published":2100,"retry":7016,"total":2100},"queue":{"max_events":4096}}},"metricbeat":{"auditd":{"auditd":{"events":8,"success":8}},"file_integrity":{"file":{"events":1,"success":1}},"system":{"host":{"events":1,"success":1},"login":{"events":4,"success":4},"package":{"events":1754,"success":1754},"process":{"events":280,"success":280},"socket":{"events":2,"success":2},"user":{"events":50,"success":50}}},"system":{"cpu":{"cores":4},"load":{"1":4.52,"15":3.47,"5":4.03,"norm":{"1":1.13,"15":0.8675,"5":1.0075}}}},"ecs.version":"1.6.0"}}
As you can see there are errors in your logs.
Your connection from beats to elasticsearch is not established properly.
I guess you've not given the right credentials to authenticate to elasticsearch.
Check your configuration and retry.
This is my auditbeat.yml file. I have used the beats system user and its password under output.elasticsearch. Previously I also tried with the elastic username and credentials
###################### Auditbeat Configuration Example #########################
# This is an example configuration file highlighting only the most common
# options. The auditbeat.reference.yml file from the same directory contains all
# the supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/auditbeat/index.html
# =========================== Modules configuration ============================
auditbeat.modules:
- module: auditd
# Load audit rules from separate files. Same format as audit.rules(7).
audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
audit_rules: |
## Define audit rules here.
## Create file watches (-w) or syscall audits (-a or -A). Uncomment these
## examples or add your own rules.
## If you are on a 64 bit platform, everything should be running
## in 64 bit mode. This rule will detect any use of the 32 bit syscalls
## because this might be a sign of someone exploiting a hole in the 32
## bit API.
#-a always,exit -F arch=b32 -S all -F key=32bit-abi
## Executions.
#-a always,exit -F arch=b64 -S execve,execveat -k exec
## External access (warning: these can be expensive to audit).
#-a always,exit -F arch=b64 -S accept,bind,connect -F key=external-access
## Identity changes.
#-w /etc/group -p wa -k identity
#-w /etc/passwd -p wa -k identity
#-w /etc/gshadow -p wa -k identity
## Unauthorized access attempts.
#-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
#-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
- module: file_integrity
paths:
- /bin
- /usr/bin
- /sbin
- /usr/sbin
- /etc
- module: system
datasets:
- package # Installed, updated, and removed packages
period: 2m # The frequency at which the datasets check for changes
- module: system
datasets:
- host # General host information, e.g. uptime, IPs
- login # User logins, logouts, and system boots.
## - package #Installed,updated, and removed packages
- process # Started and stopped processes
- socket # Opened and closed sockets
- user # User information
# How often datasets send state updates with the
# current state of the system (e.g. all currently
# running processes, all open sockets).
state.period: 12h
# Enabled by default. Auditbeat will read password fields in
# /etc/passwd and /etc/shadow and store a hash locally to
# detect any changes.
user.detect_password_changes: true
# File patterns of the login record files.
login.wtmp_file_pattern: /var/log/wtmp*
login.btmp_file_pattern: /var/log/btmp*
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "192:168.111.135:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Auditbeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["192.168.111.135:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "beats_system"
#password: "VADepz1ZwdoQVdFZDkl7"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/auditbeat
name: auditbeat
keepfiles: 7
permissions: 0640
# ============================= X-Pack Monitoring ==============================
# Auditbeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Auditbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
# ============================== Instrumentation ===============================
# Instrumentation support for the auditbeat.
#instrumentation:
# Set to true to enable instrumentation of auditbeat.
#enabled: false
# Environment in which auditbeat is running on (eg: staging, production, etc.)
#environment: ""
# APM Server hosts to report instrumentation results to.
#hosts:
# - http://localhost:8200
# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:
# Secret token for the APM Server(s).
#secret_token:
# ================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
This is the log If I use elastic username and its appropiate credentials
{"log.level":"info","@timestamp":"2022-12-13T22:02:06.498+1100","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/auditbeat] Config path: [/etc/auditbeat] Data path: [/var/lib/auditbeat] Logs path: [/var/log/auditbeat]","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T22:02:06.499+1100","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: c2f143d3-0b3d-4c5f-93bc-3582013c46a7","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-13T22:02:09.507+1100","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": dial tcp 169.254.169.254:80: i/o timeout (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-13T22:02:09.514+1100","log.origin":{"file.name":"instance/beat.go","file.line":392},"message":"auditbeat stopped.","service.name":"auditbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-13T22:02:09.515+1100","log.origin":{"file.name":"instance/beat.go","file.line":1057},"message":"Exiting: cannot obtain lockfile: connot start, data directory belongs to process with pid 10570","service.name":"auditbeat","ecs.version":"1.6.0"}
Exiting: cannot obtain lockfile: connot start, data directory belongs to process with pid 10570
I still cannot see any host, or any data from the auditbeat.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.