hello,
i want to integrate snort3 with elk stack. when i use this command :
sudo filebeat setup -E output.logstash.enabled=false -E output.Elasticsearch.hosts=['192.168.200.100:9200'] -E setup.kibana.host=192.168.200.100:5601
i get this error :
Overwriting ILM policy is disabled. Set setup.ilm.overwrite: true for enabling.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: Machine Learning in the Elastic Stack [8.2] | Elastic
It is not possble to load ML jobs into an Elasticsearch 8.0.0 or newer using the Beat.
Exiting: 1 error: Error setting up ML for apache_ecs: 10 errors: ; ; ; ; ; ; ; ; ;
this is my filebeat.yml :
============================== Filebeat inputs ===============================
filebeat.inputs:
Each - is an input. Most options can be set at the input level, so
you can use different inputs for various configurations.
Below are the input specific configurations.
filestream is an input for collecting log messages from files.
-
type: log
Change to true to enable this input configuration.
enabled: true
Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
- /var/log/snort/alert_json.txt
#- c:\programdata\Elasticsearch\logs*
setup.kibana:
host: "http://192.168.200.100:5601"
---------------------------- Elasticsearch Output ----------------------------
output.Elasticsearch:
Array of hosts to connect to.
hosts: ["https://192.168.200.100:9200"]
sudo filebeat -e setup :
2022-05-15T12:56:58.646+0100 INFO instance/beat.go:685 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-05-15T12:56:58.657+0100 INFO instance/beat.go:693 Beat ID: e805a936-a384-47e9-a062-dd3b6bcde065
2022-05-15T12:57:01.726+0100 WARN [add_cloud_metadata] add_cloud_metadata/provider_aws_ec2.go:79 read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.
2022-05-15T12:57:01.730+0100 INFO [beat] instance/beat.go:1039 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "e805a936-a384-47e9-a062-dd3b6bcde065"}}}
2022-05-15T12:57:01.732+0100 INFO [beat] instance/beat.go:1048 Build info {"system_info": {"build": {"commit": "1993ee88a11cb34f61a1fb45c7c3cf50533682cb", "libbeat": "7.17.3", "time": "2022-04-19T09:27:20.000Z", "version": "7.17.3"}}}
2022-05-15T12:57:01.734+0100 INFO [beat] instance/beat.go:1051 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.17.8"}}}
2022-05-15T12:57:01.736+0100 INFO [beat] instance/beat.go:1055 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-05-15T08:47:04+01:00","containerized":false,"name":"ids-VirtualBox","ip":["127.0.0.1/8","::1/128","192.168.200.50/24","fe80::748c:4f53:7b00:9a66/64","192.168.1.10/24","fe80::fb65:6477:4d28:31e1/64"],"kernel_version":"5.13.0-41-generic","mac":["08:00:27:3c:6b:bb","08:00:27:0a:b4:f4"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"CET","timezone_offset_sec":3600,"id":"883cf7ca5e9a40af9ef50059f8204fe7"}}}
2022-05-15T12:57:01.738+0100 INFO [beat] instance/beat.go:1084 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/home/ids", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 10782, "ppid": 10781, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2022-05-15T12:56:56.690+0100"}}}
2022-05-15T12:57:01.741+0100 INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.17.3
2022-05-15T12:57:01.742+0100 INFO [index-management] idxmgmt/std.go:184 Set output.Elasticsearch.index to 'filebeat-7.17.3' as ILM is enabled.
2022-05-15T12:57:01.746+0100 INFO [esclientleg] eslegclient/connection.go:105 Elasticsearch url: https://192.168.200.100:9200
2022-05-15T12:57:01.760+0100 INFO [publisher] pipeline/module.go:113 Beat name: ids-VirtualBox
2022-05-15T12:57:01.780+0100 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2022-05-15T12:57:01.869+0100 INFO [esclientleg] eslegclient/connection.go:105 Elasticsearch url: https://192.168.200.100:9200
2022-05-15T12:57:01.880+0100 ERROR [esclientleg] transport/logging.go:37Error dialing tls: first record does not look like a TLS handshake {"network": "tcp", "address": "192.168.200.100:9200"}
2022-05-15T12:57:01.887+0100 ERROR [esclientleg] eslegclient/connection.go:231 error connecting to Elasticsearch at https://192.168.200.100:9200: Get "https://192.168.200.100:9200": http: server gave HTTP response to HTTPS client
2022-05-15T12:57:01.887+0100 ERROR instance/beat.go:1014 Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://192.168.200.100:9200: Get "https://192.168.200.100:9200": http: server gave HTTP response to HTTPS client]
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://192.168.200.100:9200: Get "https://192.168.200.100:9200": http: server gave HTTP response to HTTPS client]
curl -XGET 'http://192.168.200.100:9200/filebeat-*/_search?pretty'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" :
}
}
curl -XGET 'http://192.168.200.100:9200/?pretty'
{
"name" : "node-1",
"cluster_name" : "Elasticsearch",
"cluster_uuid" : "jUKN1EpIQmOBD7eCl2skPw",
"version" : {
"number" : "7.17.2",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "de7261de50d90919ae53b0eff9413fd7e5307301",
"build_date" : "2022-03-28T15:12:21.446567561Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
