Here is my set up,
I am running ELK on a remote VM, and I am trying to sync all the logfiles from the local machine to the remote server without restarting the filebeat service (note the filebeat service is running on my local machine).
Configuration Details:
- ELK is running on a remote VM of Ubuntu Type
- Filebeat service is running on my local machine of Windows type.
Problem statement:
- My app service is generating logs on my local machine, and I am trying to sync it with the ELK without restarting the filebeat service every time.
- I have two log file types: info log and error log, and I want to send both to the ELK stack.
What am I solving this problem now?
I am restating the file beat service on my Windows every time. But I am only seeing one type of log.
Details:
- My file beat YML file configuration:
---
filebeat.inputs:
- type: log
id: i4eappinfo
enabled: true
paths:
- C:\\appServerLog\\output\\i4e-mf-node-api-logs.log
multiline.pattern: ^[0-9]{2}-[0-9]{2}-[0-9]{4}
multiline.negate: true
multiline.match: after
refresh_frequency: 10s
scan_frequency: 5s
- type: log
id: i4eapperror
enabled: true
paths:
- C:\\appServerLog\\output\\i4e-mf-node-api-errors.log
multiline.pattern: ^[0-9]{2}-[0-9]{2}-[0-9]{4}
multiline.negate: true
multiline.match: after
refresh_frequency: 10s
scan_frequency: 5s
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
host: <remote_server>:5601
output.elasticsearch:
hosts:
- <remote_server>:9200
username: <user_name>
password: <password>
preset: balanced
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
And here is a sample of the logs files.
Error Log:
Info Log:
Service Details:
Configuration Details on ELK:
Could someone assist what I have missed here? That would be helpful.