Hi everyone.
I am using the Filebeat service, and I have a problem. I can't see the logs in Kibana.
I am on a Windows machine.
I tried to config Filebeat to get data from Elasticsearch.
This is my config file:
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
enabled: tru
paths:
# - /var/log/*.log
- C://dev/app//trunk//Src//Backend//.servers//elasticsearch-7.6.2//logs//.*.log
#============================= Filebeat modules ===============================
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "localhost:5601"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
ssl.verification_mode: "none"
verification_mode: "none"
protocol: "https"
username: "elastic"
password: "password"
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
#================================ Logging =====================================
logging.level: info
logging.selectors: ["*"]
#================================= Migration ==================================
migration.6_to_7.enabled: true
Also, this is my result from " .\filebeat -c filebeat.yml -e -d "*" "
2020-06-12T10:33:13.514+0300 INFO instance/beat.go:621 Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\Program Files\Filebeat\data] Logs path: [C:\Program Files\Filebeat\logs]
2020-06-12T10:33:13.514+0300 DEBUG [beat] instance/beat.go:673 Beat metadata path: C:\Program Files\Filebeat\data\meta.json
2020-06-12T10:33:13.515+0300 INFO instance/beat.go:629 Beat ID: b7a0c7bf-6799-4093-bc50-cb1cb27f5715
2020-06-12T10:33:13.687+0300 DEBUG [docker] docker/client.go:48 Docker client will negotiate the API version on the first request.
2020-06-12T10:33:13.687+0300 DEBUG [add_docker_metadata] add_docker_metadata/add_docker_metadata.go:87 add_docker_metadata: docker environment not detected: protocol not available
2020-06-12T10:33:13.687+0300 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:126 add_cloud_metadata: starting to fetch metadata, timeout=3s
2020-06-12T10:33:13.689+0300 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for gcp after 945.8µs. result=[provider:gcp, error=failed requesting gcp metadata: Get http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network., metadata={}]
2020-06-12T10:33:13.689+0300 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for aws after 1.9376ms. result=[provider:aws, error=failed requesting aws metadata: Get http://169.254.169.254/2014-02-25/dynamic/instance-identity/document: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network., metadata={}]
2020-06-12T10:33:13.689+0300 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for az after 1.9376ms. result=[provider:az, error=failed requesting az metadata: Get http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network., metadata={}]
2020-06-12T10:33:13.689+0300 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for openstack after 1.9376ms. result=[provider:openstack, error=failed requesting openstack metadata: Get http://169.254.169.254/2009-04-04/meta-data/instance-id: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network., metadata={}]
2020-06-12T10:33:13.689+0300 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for digitalocean after 1.9376ms. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get http://169.254.169.254/metadata/v1.json: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network., metadata={}]
2020-06-12T10:33:13.689+0300 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:129 add_cloud_metadata: fetchMetadata ran for 1.9376ms
2020-06-12T10:33:13.689+0300 DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:129 Could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided {"libbeat.processor": "add_kubernetes_metadata"}
2020-06-12T10:33:13.690+0300 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-06-12T10:33:13.690+0300 DEBUG [processors] processors/processor.go:101 Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], add_cloud_metadata={}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.ppid]], add_kubernetes_metadata
2020-06-12T10:33:13.690+0300 DEBUG [seccomp] seccomp/seccomp.go:96 Syscall filtering is only supported on Linux
2020-06-12T10:33:13.690+0300 INFO [beat] instance/beat.go:957 Beat info {"system_info": {"beat": {"path": {"config": "C:\\Program Files\\Filebeat", "data": "C:\\Program Files\\Filebeat\\data", "home": "C:\\Program Files\\Filebeat", "logs": "C:\\Program Files\\Filebeat\\logs"}, "type": "filebeat", "uuid": "b7a0c7bf-6799-4093-bc50-cb1cb27f5715"}}}
2020-06-12T10:33:13.690+0300 INFO [beat] instance/beat.go:966 Build info {"system_info": {"build": {"commit": "932b273e8940575e15f10390882be205bad29e1f", "libbeat": "7.7.1", "time": "2020-05-28T15:24:10.000Z", "version": "7.7.1"}}}
2020-06-12T10:33:13.691+0300 INFO [beat] instance/beat.go:969 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":12,"version":"go1.13.9"}}}
2020-06-12T10:33:13.778+0300 INFO [beat] instance/beat.go:973 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-06-12T09:13:56.39+03:00","name":"DESKTOP-49MF3DL","ip":["fe80::d94b:faf6:a84a:a06f/64","169.254.160.111/16","fe80::25e8:9bf9:a8d5:bab6/64","169.254.186.182/16","fe80::d476:901e:905c:3459/64","169.254.52.89/16","fe80::b1a6:7b8c:a26b:d18e/64","169.254.209.142/16","fe80::dd4:d315:9ee2:44f7/64","169.254.68.247/16","2a02:2f0e:10c:a300:8232:53ff:fead:c3ed/128","fe80::5034:a372:2450:183b/64","192.168.1.9/24","fe80::54f5:7548:68a1:4e23/64","169.254.78.35/16","::1/128","127.0.0.1/8","fe80::39e2:a0c:862b:9ecd/64","172.18.84.49/28"],"kernel_version":"10.0.18362.836 (WinBuild.160101.0800)","mac":["00:09:0f:fe:00:01","f8:75:a4:8d:24:0c","00:09:0f:aa:00:01","80:32:53:ad:c3:ee","82:32:53:ad:c3:ed","80:32:53:ad:c3:ed","80:32:53:ad:c3:f1","00:15:5d:57:db:79"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Pro N","version":"10.0","major":10,"minor":0,"patch":0,"build":"18362.836"},"timezone":"EEST","timezone_offset_sec":10800,"id":"9043fb9e-6638-483a-b674-78d77bf08f8f"}}}
2020-06-12T10:33:13.783+0300 INFO [beat] instance/beat.go:1002 Process info {"system_info": {"process": {"cwd": "C:\\Program Files\\Filebeat", "exe": "C:\\Program Files\\Filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 13760, "ppid": 18956, "start_time": "2020-06-12T10:33:12.278+0300"}}}
2020-06-12T10:33:13.783+0300 INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.7.1
2020-06-12T10:33:13.783+0300 DEBUG [beat] instance/beat.go:323 Initializing output plugins
2020-06-12T10:33:13.783+0300 INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'filebeat-7.7.1' as ILM is enabled.
2020-06-12T10:33:13.783+0300 INFO eslegclient/connection.go:84 elasticsearch url: https://localhost:9200
2020-06-12T10:33:13.784+0300 WARN [tls] tlscommon/tls_config.go:83 SSL/TLS verifications disabled.
2020-06-12T10:33:13.784+0300 DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer
2020-06-12T10:33:13.784+0300 INFO [publisher] pipeline/module.go:110 Beat name: DESKTOP-49MF3DL
2020-06-12T10:33:13.786+0300 INFO instance/beat.go:438 filebeat start running.
2020-06-12T10:33:13.786+0300 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-06-12T10:33:13.786+0300 DEBUG [service] service/service_windows.go:72 Windows is interactive: true
2020-06-12T10:33:13.786+0300 DEBUG [test] registrar/migrate.go:159 isFile(C:\Program Files\Filebeat\data\registry) -> false
2020-06-12T10:33:13.787+0300 DEBUG [test] registrar/migrate.go:159 isFile() -> false
2020-06-12T10:33:13.787+0300 DEBUG [test] registrar/migrate.go:152 isDir(C:\Program Files\Filebeat\data\registry\filebeat) -> true
2020-06-12T10:33:13.787+0300 DEBUG [test] registrar/migrate.go:159 isFile(C:\Program Files\Filebeat\data\registry\filebeat\meta.json) -> true
2020-06-12T10:33:13.787+0300 DEBUG [registrar] registrar/migrate.go:51 Registry type '0' found
2020-06-12T10:33:13.788+0300 DEBUG [registrar] registrar/registrar.go:125 Registry file set to: C:\Program Files\Filebeat\data\registry\filebeat\data.json
2020-06-12T10:33:13.789+0300 INFO registrar/registrar.go:145 Loading registrar data from C:\Program Files\Filebeat\data\registry\filebeat\data.json
2020-06-12T10:33:13.790+0300 INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-06-12T10:33:13.790+0300 INFO beater/crawler.go:73 Loading Inputs: 1
2020-06-12T10:33:13.790+0300 DEBUG [registrar] registrar/registrar.go:278 Starting Registrar
2020-06-12T10:33:13.796+0300 DEBUG [input] log/config.go:204 recursive glob enabled
2020-06-12T10:33:13.796+0300 DEBUG [input] log/input.go:164 exclude_files: []. Number of states: 0
2020-06-12T10:33:13.797+0300 DEBUG [input] log/input.go:185 input with previous states loaded: 0
2020-06-12T10:33:13.799+0300 INFO log/input.go:152 Configured paths: [C:\dev\app\trunk\Src\Backend\.servers\elasticsearch-7.6.2\logs\.*.log]
2020-06-12T10:33:13.803+0300 INFO input/input.go:114 Starting input of type: log; ID: 7499844627259818449
2020-06-12T10:33:13.807+0300 DEBUG [input] log/input.go:191 Start next scan
2020-06-12T10:33:13.807+0300 INFO beater/crawler.go:105 Loading and starting Inputs completed. Enabled inputs: 1
2020-06-12T10:33:13.807+0300 INFO cfgfile/reload.go:175 Config reloader started
2020-06-12T10:33:13.811+0300 DEBUG [input] log/input.go:212 input states cleaned up. Before: 0, After: 0, Pending: 0
Please help me!
Thank you