Hello, i'm a newbie trying to send my second dataset, sitting inside my downloads folder (extracted) to kibana.
Filebeat, elasticsearch, kibana are all running
locahost:5601 is running
filebeat.yml config as follow: (Note: the first log set has been received by kibana and is running, however my second attempt to add more logs is giving me an error message - look below)
`#filebeat.inputs:
#- type: stdin
filebeat.inputs:
- type: log
paths:
- /home/ubuntu/Downloads/SAT-*
- /home/ubuntu/Downloads/03-11/LDAP.csv
- /home/ubuntu/Downloads/03-11/MSSQL.csv
- /home/ubuntu/Downloads/03-11/NetBIOS.csv
- /home/ubuntu/Downloads/03-11/Portmap.csv
- /home/ubuntu/Downloads/03-11/Syn.csv
- /home/ubuntu/Downloads/03-11/UDP.csv
- /home/ubuntu/Downloads/03-11/UDPLag.csv
output.elasticsearch:
hosts: ["http://35.230.53.171:9200"]
#output.console:
#pretty: true`
~ ``
Getting the following error:
[sudo] password for ubuntu:
2021-05-22T10:17:50.751-0700 INFO instance/beat.go:660 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-05-22T10:17:50.751-0700 DEBUG [beat] instance/beat.go:712 Beat metadata path: /var/lib/filebeat/meta.json
2021-05-22T10:17:50.751-0700 INFO instance/beat.go:668 Beat ID: f0f06cda-f19f-4d42-a7e6-3625e03076be
2021-05-22T10:17:50.756-0700 DEBUG [conditions] conditions/conditions.go:98 New condition contains: map[]
2021-05-22T10:17:50.756-0700 DEBUG [conditions] conditions/conditions.go:98 New condition !contains: map[]
2021-05-22T10:17:50.756-0700 DEBUG [docker] docker/client.go:48 Docker client will negotiate the API version on the first request.
2021-05-22T10:17:50.757-0700 DEBUG [add_docker_metadata] add_docker_metadata/add_docker_metadata.go:87 add_docker_metadata: docker environment not detected: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2021-05-22T10:17:50.757-0700 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:128 add_cloud_metadata: starting to fetch metadata, timeout=3s
2021-05-22T10:17:50.758-0700 DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:138 Could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable {"libbeat.processor": "add_kubernetes_metadata"}
2021-05-22T10:17:53.758-0700 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:165 add_cloud_metadata: received disposition for azure after 3.001199671s. result=[provider:azure, error=failed requesting azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-05-22T10:17:53.759-0700 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:172 add_cloud_metadata: timed-out waiting for all responses
2021-05-22T10:17:53.759-0700 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:131 add_cloud_metadata: fetchMetadata ran for 3.001650193s
2021-05-22T10:17:53.759-0700 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2021-05-22T10:17:53.759-0700 DEBUG [processors] processors/processor.go:120 Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], condition=!contains: map[], add_cloud_metadata={}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.ppid]], add_kubernetes_metadata
2021-05-22T10:17:53.762-0700 INFO [api] api/server.go:62 Starting stats endpoint
2021-05-22T10:17:53.762-0700 DEBUG [seccomp] seccomp/seccomp.go:117 Loading syscall filter {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2021-05-22T10:17:53.763-0700 INFO [api] api/server.go:64 Metrics endpoint listening on: 127.0.0.1:5067 (configured: localhost)
2021-05-22T10:17:53.763-0700 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2021-05-22T10:17:53.763-0700 INFO [beat] instance/beat.go:996 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "f0f06cda-f19f-4d42-a7e6-3625e03076be"}}}
2021-05-22T10:17:53.764-0700 INFO [beat] instance/beat.go:1005 Build info {"system_info": {"build": {"commit": "651a2ad1225f3d4420a22eba847de385b71f711d", "libbeat": "7.12.1", "time": "2021-04-20T20:58:32.000Z", "version": "7.12.1"}}}
2021-05-22T10:17:53.764-0700 INFO [beat] instance/beat.go:1008 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.15.9"}}}
2021-05-22T10:17:53.766-0700 INFO [beat] instance/beat.go:1012 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-05-21T15:09:32-07:00","containerized":false,"name":"ubuntu-virtual-machine","ip":["127.0.0.1/8","::1/128","192.168.152.133/24","fe80::1cc6:ff10:5d8f:3b0c/64"],"kernel_version":"5.8.0-53-generic","mac":["00:0c:29:97:15:82"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.2 LTS (Focal Fossa)","major":20,"minor":4,"patch":2,"codename":"focal"},"timezone":"PDT","timezone_offset_sec":-25200,"id":"c211ca7fa1ec457ea6f2337b85f7f216"}}}
2021-05-22T10:17:53.767-0700 INFO [beat] instance/beat.go:1041 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39"],"ambient":null}, "cwd": "/home/ubuntu", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 165726, "ppid": 165716, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-05-22T10:17:47.910-0700"}}}
2021-05-22T10:17:53.767-0700 INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.12.1
2021-05-22T10:17:53.768-0700 DEBUG [beat] instance/beat.go:330 Initializing output plugins
2021-05-22T10:17:53.768-0700 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.12.1' as ILM is enabled.
2021-05-22T10:17:53.768-0700 INFO eslegclient/connection.go:99 elasticsearch url: http://localhost:9200
2021-05-22T10:17:53.769-0700 DEBUG [publisher] pipeline/consumer.go:148 start pipeline event consumer
2021-05-22T10:17:53.769-0700 INFO [publisher] pipeline/module.go:113 Beat name: filebeat
2021-05-22T10:17:53.772-0700 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2021-05-22T10:17:53.772-0700 INFO kibana/client.go:119 Kibana url: http://localhost:5601
2021-05-22T10:17:53.822-0700 INFO [monitoring] log/log.go:152 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"id":"user@1000.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":2194132992}}}},"cpu":{"system":{"ticks":2440,"time":{"ms":2445}},"total":{"ticks":3410,"time":{"ms":3422},"value":3410},"user":{"ticks":970,"time":{"ms":977}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":10},"info":{"ephemeral_id":"abb6eb31-29a4-47ff-962d-89bcb0650685","uptime":{"ms":5846}},"memstats":{"gc_next":18579616,"memory_alloc":10018064,"memory_sys":75056128,"memory_total":45680976,"rss":77275136},"runtime":{"goroutines":13}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":2.6,"15":0.87,"5":1.4,"norm":{"1":1.3,"15":0.435,"5":0.7}}}}}}
2021-05-22T10:17:53.827-0700 INFO [monitoring] log/log.go:153 Uptime: 5.852221368s
2021-05-22T10:17:53.827-0700 INFO [monitoring] log/log.go:130 Stopping metrics logging.
2021-05-22T10:17:53.829-0700 INFO [api] api/server.go:66 Stats endpoint (127.0.0.1:5067) finished: accept tcp 127.0.0.1:5067: use of closed network connection
2021-05-22T10:17:53.830-0700 INFO instance/beat.go:465 filebeat stopped.
2021-05-22T10:17:53.830-0700 ERROR instance/beat.go:971 Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get "http://localhost:5601/api/status": dial tcp 127.0.0.1:5601: connect: connection refused. Response: .
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get "http://localhost:5601/api/status": dial tcp 127.0.0.1:5601: connect: connection refused. Response: .
ubuntu@ubuntu-virtual-machine:~$ ```