We are running the filebeat as a service on our ec2 machine. Below is the configuration file.
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: log
enabled: true
paths:
- /hybris/hybris/log/hybris.log
ignore_older: 1h
exclude_lines: ['.*ELB-HealthChecker', '.*healthcheck.jsp']
exclude_files: ['.gz$']
fields:
env: ******
application: ********
topic: commerce-logs
multiline.type: pattern
multiline.pattern: '(\d{4}\-\d{2}\-\d{2})'
multiline.negate: true
multiline.match: after
- type: log
enabled: true
paths:
- /hybris/hybris/log/tomcat/access*.log
ignore_older: 1h
exclude_lines: ['.*ELB-HealthChecker', '.*healthcheck.jsp']
exclude_files: ['.gz$']
fields:
env: *****
application: ******
topic: commerceaccess-logs
- type: log
enabled: true
paths:
- /hybris/hybris/log/tomcat/console*.log
ignore_older: 1h
exclude_lines: ['.*ELB-HealthChecker', '.*healthcheck.jsp']
exclude_files: ['.gz$']
fields:
env: ****
application: *****
topic: commerceconsole-logs
path.data: /var/lib/filebeat-new
path.config: /etc/filebeat
path.logs: /var/log/filebeat-new
#------------------------------Kafka Output-------------------------------------
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["*****:9092"]
# message topic selection + partitioning
topic: '%{[fields.topic]}'
codec.json:
pretty: true
partition.round_robin:
reachable_only: true
required_acks: 1
compression: gzip
max_message_bytes: 1000000
We are mentioning the data path and the log path in the configuration file but when we are starting it, it is picking up a different data and log path. Below is the attached screenshot.
