I'm trying to make filebeat send log to logstash on another machine and I just can't get it to work. This is the filebeat.yml configuration:
https://pastebin.com/8a2RtGBa (Using pastebin because of character limit)
This is the configuration on the machine that has logstash:
[root@ELK ~]$ cat /etc/logstash/conf.d/testing.conf
input{
file{
path => "/var/log/commands.log"
}
beats{
port => 5044
}
}
filter {
if [path] == "/var/log/commands.log" {
grok{
match => { "message" => "\[(%{TIMESTAMP_ISO8601:sys_timestamp})\]\s(?<field1>[0-9a-zA-Z_-]+)\s(?<field2>[0-9a-zA-Z_-]+)\:USER=(?<field3>[0-9a-zA-Z_-]+)\sPWD=(?<field4>[0-9a-zA-Z_/-]+)\sPID=\[(?<field5>[0-9]+)\]\sCMD=\"(?<field6>.*)\"\sExit=\[(?<field7>[0-9]+)\]\sCONNECTION=(?<field8>.*)"
}
}
}
}
output{
elasticsearch { hosts => ["localhost:9200"]}
}
This is what I get if I try setting filebeat with "filebeat setup -e":
[root@server150 ~]# filebeat setup -e
2020-07-17T08:12:36.199Z INFO instance/beat.go:647 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-07-17T08:12:36.200Z INFO instance/beat.go:655 Beat ID: aa84fd5b-d016-4688-a4a1-172dbcf2054a
2020-07-17T08:12:36.202Z INFO [beat] instance/beat.go:983 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "aa84fd5b-d016-4688-a4a1-172dbcf2054a"}}}
2020-07-17T08:12:36.203Z INFO [beat] instance/beat.go:992 Build info {"system_info": {"build": {"commit": "f79387d32717d79f689d94fda1ec80b2cf285d30", "libbeat": "7.8.0", "time": "2020-06-14T18:15:37.000Z", "version": "7.8.0"}}}
2020-07-17T08:12:36.203Z INFO [beat] instance/beat.go:995 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.10"}}}
2020-07-17T08:12:36.204Z INFO [beat] instance/beat.go:999 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-09T11:44:16Z","containerized":false,"name":"server150","ip":["127.0.0.1/8","*ip-of-thismachine*/25"],"kernel_version":"3.10.0-1062.18.1.el7.x86_64","mac":["00:0c:29:e5:a1:fa"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"3eec437c66d444a59ef5f075a429441d"}}}
2020-07-17T08:12:36.204Z INFO [beat] instance/beat.go:1028 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 12559, "ppid": 11800, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-07-17T08:12:35.340Z"}}}
2020-07-17T08:12:36.204Z INFO instance/beat.go:310 Setup Beat: filebeat; Version: 7.8.0
2020-07-17T08:12:36.205Z INFO [publisher] pipeline/module.go:113 Beat name: server150
2020-07-17T08:12:36.206Z WARN beater/filebeat.go:156 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-07-17T08:12:36.206Z ERROR instance/beat.go:958 Exiting: Index management requested but the Elasticsearch output is not configured/enabled
Exiting: Index management requested but the Elasticsearch output is not configured/enabled
This is what I get when I try to run filebeat with "filebeat -e":
[root@server150 ~]# filebeat -e
2020-07-17T08:16:47.104Z INFO instance/beat.go:647 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-07-17T08:16:47.104Z INFO instance/beat.go:655 Beat ID: aa84fd5b-d016-4688-a4a1-172dbcf2054a
2020-07-17T08:16:47.107Z INFO instance/beat.go:404 filebeat stopped.
2020-07-17T08:16:47.107Z ERROR instance/beat.go:958 Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
Does anyone knows what's going on? I just can't get it to work. Huge thanks ahead!
