I setup filebeat to send logs to elasticsearch but for now I have two problems.
First, I receive my message header and content in two separate records. In Kibana interface they are shown like this:
But in source file this is one record:
The second problem is, that I want to parse message content and get its separate parts in different columns in Kibana. I tried to do it with dissect processor, but no fields are present in message details. My filebeat config yml file:
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.inputs:
- type: log
paths:
- /usr/share/filebeat/logs/app.log
pattern: '^[[:space:]]'
negate: false
match: after
processors:
- add_host_metadata: ~
- dissect:
tokenizer: '%{@timestamp} [%{service.pid}] %{log.logger} [%{service.thread_id}] [%{log.level}] %{message}'
field: "message"
target_prefix: ""
- timestamp:
field: "@timestamp"
layouts:
- '2006-01-02T15:04:05,999'
test:
- '2024-05-23T14:57:14,818'
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:127.0.0.1:9200}'
username: '${ELASTICSEARCH_USERNAME:elastic}'
password: '${ELASTICSEARCH_PASSWORD:elastic}'
And the message details are:
Can I parse my message with filebeat itself or I need to use Grok? Can I also show my message header and content as a single record?