Hi ,
I've used dissect in order to process the log outputs of our application. And then I've renamed the message by dropping it and renaming other field. The log seems to be processing fine and I've verified it using output.console. But these processed logs are not getting displayed in kibana. Here is the file beat configuration. One thing I've noticed is the logs that were throwing dissect_parsing_error and are unprocessed gets displayed.
We are running in Kubernetes and here is the filebeat configuration.
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
data:
filebeat.yaml: |-
filebeat.inputs:
- type: log
enabled: true
paths:
- /home/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat"
setup.ilm.pattern: "{now/d}-000001"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
- dissect:
tokenizer: '%{longdate}|%{eventid}|%{level}|%{logger}|%{data}| %{?_url}: %{&_url}| %{?_action}: %{&_action}'
field: "message"
target_prefix: ""
- drop_fields:
fields: ["message"]
when:
has_fields: ["data"]
- rename:
fields:
- from: "data"
to: "message"
when:
has_fields: ["data"]
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
setup.kibana:
host: '${KIBANA_URL}'
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl.enabled: true
ssl.certificate: '${SSL_CERT_PATH}'
ssl.key: '${SSL_KEY_PATH}'
ssl.verification_mode: none
Here is a sample raw log:
2020-08-02 00:00:05.3881||INFO|Microsoft.AspNetCore.Hosting.Internal.WebHost|Request starting HTTP/1.1 GET http://restapi/ | url: http://restapi/| action:
2020-08-02 00:00:05.3931||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Route matched with {action = "Index", controller = "Tenant"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Index() on controller Platform.Web.Controllers.TenantController (Platform.Web). | url: | action: Index
2020-08-02 00:00:05.6870||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Executing action method Platform.Web.Controllers.TenantController.Index (Platform.Web) - Validation state: Valid | url: https://restapi/| action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Executed action method Platform.Web.Controllers.TenantController.Index (Platform.Web), returned result Microsoft.AspNetCore.Mvc.ViewResult in 73.3134ms. | url: https://restapi/| action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor|Executing ViewResult, running view Index. | url: | action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor|Executed ViewResult - view Index executed in 0.6892ms. | url: https://restapi/| action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Executed action Platform.Web.Controllers.TenantController.Index (Platform.Web) in 368.5915ms | url: https://restapi/| action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Hosting.Internal.WebHost|Request finished in 374.065ms 200 text/html; charset=utf-8 | url: https://restapi/| action: Index
2020-08-02 00:00:09.2884||INFO|Microsoft.AspNetCore.Hosting.Internal.WebHost|Request starting HTTP/1.1 GET http://10.32.0.24:5000/ | url: http://10.32.0.24/| action:
2020-08-02 00:00:09.2884||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Route matched with {action = "Index", controller = "Tenant"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Index() on controller Platform.Web.Controllers.TenantController (Platform.Web). | url: https://10.32.0.24/| action: Index
Here is a sample formatted the JSON Output (produced when output.elasticsearch is commented and used output.console):
{
"@timestamp": "2020-08-13T01:16:27.981Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.5.2"
},
"log": {
"offset": 0,
"file": {
"path": "/home/nlog.log"
}
},
"input": {
"type": "log"
},
"url": "http://restapi/",
"longdate": "2020-08-02 00:00:05.3881",
"ecs": {
"version": "1.1.0"
},
"agent": {
"ephemeral_id": "29e028fc-bee9-44f0-bdb6-341844c9b3d2",
"hostname": "filebeat-test",
"id": "b34390d1-4b72-4073-aa5f-c967ffb02f4b",
"version": "7.5.2",
"type": "filebeat"
},
"eventid": "",
"level": "INFO",
"cloud": {
"availability_zone": "",
"instance": {
"id": "572d498f50c62ccffb68ad90dcac2e95",
"name": "ubuntu18-logs"
},
"machine": {
"type": ""
},
"provider": "openstack"
},
"message": "Request starting HTTP/1.1 GET http://restapi/ ",
"host": {
"name": "filebeat-test",
"hostname": "filebeat-test",
"architecture": "x86_64",
"os": {
"version": "7 (Core)",
"family": "redhat",
"name": "CentOS Linux",
"kernel": "4.15.0-106-generic",
"codename": "Core",
"platform": "centos"
},
"containerized": false
},
"action": "",
"logger": "Microsoft.AspNetCore.Hosting.Internal.WebHost"
}
Not sure why it is not getting rendered in kibana.
Is it because We are modifying the message field? (i.e: dissect → drop message (if data filed exist) → rename data field as message (if data field exist)).
or is it because of ` appearing in between? Like in 2nd and 10th line?
Am I missing something in configuration? Unable to figure out why it is not getting displayed in kibana.
I've checked the logs in both Kibana → Logs(stream) and Kibana → Discover*.