My app stores the log files on azure, and in the following format.
mf-node-api 10-10-2024 06:07:36.760 AM info: Found existing fund core auth token. mf-node-api 10-10-2024 06:07:36.810 AM info: Existing fund core auth token expired. Generating new auth token. mf-node-api 10-10-2024 06:07:37.512 AM info: Fund core endpoint : /account/login, returned response with status : 200 mf-node-api 10-10-2024 06:07:37.513 AM debug: Sucessfully fetched auth token from fundcore : true mf-node-api 10-10-2024 06:07:37.549 AM debug: Fundcore auth token details saved sucessfully. mf-node-api 10-10-2024 06:07:37.830 AM info: Fund core endpoint : /master-data/country-list, returned response with status : 200 mf-node-api 10-10-2024 06:07:46.208 AM info: Found existing fund core auth token. mf-node-api 10-10-2024 06:07:46.360 AM info: Fund core endpoint : /lookup/key-items, returned response with status : 200 mf-node-api 10-10-2024 06:07:51.247 AM info: Found existing fund core auth token. mf-node-api 10-10-2024 06:07:51.397 AM info: Fund core endpoint : /master-data/tax-status-list, returned response with status : 200 mf-node-api 10-10-2024 06:07:59.202 AM info: Found existing fund core auth token. mf-node-api 10-10-2024 06:07:59.291 AM info: Fund core endpoint : /lookup/key-items, returned response with status : 200 mf-node-api 10-10-2024 06:08:04.631 AM info: Found existing fund core auth token. mf-node-api 10-10-2024 06:08:04.717 AM info: Fund core endpoint : /lookup/key-items, returned response with status : 200 mf-node-api 10-10-2024 06:08:24.416 AM info: Found existing fund core auth token. mf-node-api 10-10-2024 06:08:24.547 AM info: Fund core endpoint : /master-data/occupation-list, returned response with status : 200 mf-node-api 10-10-2024 06:08:33.982 AM info: Found existing fund core auth token. mf-node-api 10-10-2024 06:08:34.108 AM info: Fund core endpoint : /lookup/keys, returned response with status : 200 mf-node-api 10-10-2024 06:11:43.428 AM info: Found existing fund core auth token. mf-node-api 10-10-2024 06:11:43.440 AM info: Found existing fund core auth token. mf-node-api 10-10-2024 06:11:43.575 AM info: Fund core endpoint : /lookup/key-items, returned response with status : 200 mf-node-api 10-10-2024 06:11:43.613 AM info: Fund core endpoint : /lookup/key-items, returned response with status : 200
This log is not formatted, although while writing I have given proper formatting. There is only space between the log, which is expected to be like below
mf-node-api 10-10-2024 06:07:36.760 AM info: Found existing fund core auth token.
mf-node-api 10-10-2024 06:07:36.810 AM info: Existing fund core auth token expired. Generating new auth token.
mf-node-api 10-10-2024 06:07:37.512 AM info: Fund core endpoint : /account/login, returned response with status : 200
mf-node-api 10-10-2024 06:07:37.513 AM debug: Sucessfully fetched auth token from fundcore : true
mf-node-api 10-10-2024 06:07:37.549 AM debug: Fundcore auth token details saved sucessfully.
mf-node-api 10-10-2024 06:07:37.830 AM info: Fund core endpoint : /master-data/country-list, returned response with status : 200
mf-node-api 10-10-2024 06:07:46.208 AM info: Found existing fund core auth token.
mf-node-api 10-10-2024 06:07:46.360 AM info: Fund core endpoint : /lookup/key-items, returned response with status : 200
mf-node-api 10-10-2024 06:07:51.247 AM info: Found existing fund core auth token.
mf-node-api 10-10-2024 06:07:51.397 AM info: Fund core endpoint : /master-data/tax-status-list, returned response with status : 200
mf-node-api 10-10-2024 06:07:59.202 AM info: Found existing fund core auth token.
mf-node-api 10-10-2024 06:07:59.291 AM info: Fund core endpoint : /lookup/key-items, returned response with status : 200
So I want to design a pipeline , I tried with regex but it is not working . I set up filebeat as follows,
filebeat.inputs:
- type: log
id: i4e-mf-logs
enabled: true
paths:
- https://app_service/wwwroot/logs/i4e-mf-node-api-09-10-2024-logs.log
multiline.pattern: "^[0-9]{2}-[0-9]{2}-[0-9]{4}
[0-9]{2}:[0-9]{2}:[0-9]{2}\\.[0-9]{3} [A-Z]{2}:"
multiline.negate: true
multiline.match: after
processors:
- dissect:
tokenizer: "%{service} %{+timestamp} %{+time} %{+level}: %{message}"
field: message
target_prefix: ""
- drop_fields:
fields:
- message
output.elasticsearch:
hosts:
- localhost:9200
username: elasticuser
password: <password>
index: mf-node-api-logs-%{+yyyy.MM.dd}
setup.kibana:
host: localhost:5601
setup.template.settings:
index.number_of_shards: 1
logging.level: info
Somehow this is not working; could someone help me to configure it properly? what I am missing here.
Also, I tried with creating a pipeline
In the processor, i have given the following configuration,
Processor : GSUB
Field : Message
Pattern: \smf-node-api
Replacement: \nmf-node-api
Still not working