# Filebeat dissect processed logs not appearing in Kibana

**URL:** <https://discuss.elastic.co/t/filebeat-dissect-processed-logs-not-appearing-in-kibana/245065>\
**Category:** Kibana\
**Created:** [August 15, 2020, 12:31am UTC](https://discuss.elastic.co/t/filebeat-dissect-processed-logs-not-appearing-in-kibana/245065 "2020-08-15T00:31:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![MANI\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mani_m/32/73784_2.png) [@MANI\_M](https://discuss.elastic.co/u/MANI_M)\
**Post date:** [August 15, 2020, 12:31am UTC](https://discuss.elastic.co/t/filebeat-dissect-processed-logs-not-appearing-in-kibana/245065/1 "2020-08-15T00:31:43Z")

</div>

Hi ,

I've used dissect in order to process the log outputs of our application. And then I've renamed the message by dropping it and renaming other field. The log seems to be processing fine and I've verified it using `output.console`. But these processed logs are not getting displayed in `kibana`. Here is the file beat configuration. One thing I've noticed is the logs that were throwing `dissect_parsing_error` and are unprocessed gets displayed.

We are running in Kubernetes and here is the filebeat configuration.

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
data:
  filebeat.yaml: |-
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /home/*.log
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    setup.template.settings:
      index.number_of_shards: 1
    setup.ilm.enabled: auto
    setup.ilm.rollover_alias: "filebeat"
    setup.ilm.pattern: "{now/d}-000001"
    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~
      - add_docker_metadata: ~
      - add_kubernetes_metadata: ~
      - dissect:
          tokenizer: '%{longdate}|%{eventid}|%{level}|%{logger}|%{data}| %{?_url}: %{&_url}| %{?_action}: %{&_action}'
          field: "message"
          target_prefix: ""
      - drop_fields:
          fields: ["message"]
          when:
            has_fields: ["data"]
      - rename:
          fields:
          - from: "data"
            to: "message"
          when:
            has_fields: ["data"]
    
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
    setup.kibana:
      host: '${KIBANA_URL}'
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl.enabled: true
      ssl.certificate: '${SSL_CERT_PATH}'
      ssl.key: '${SSL_KEY_PATH}'
      ssl.verification_mode: none

```

Here is a sample raw log:

```auto
2020-08-02 00:00:05.3881||INFO|Microsoft.AspNetCore.Hosting.Internal.WebHost|Request starting HTTP/1.1 GET http://restapi/ | url: http://restapi/| action: 
2020-08-02 00:00:05.3931||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Route matched with {action = "Index", controller = "Tenant"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Index() on controller Platform.Web.Controllers.TenantController (Platform.Web). | url: | action: Index
2020-08-02 00:00:05.6870||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Executing action method Platform.Web.Controllers.TenantController.Index (Platform.Web) - Validation state: Valid | url: https://restapi/| action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Executed action method Platform.Web.Controllers.TenantController.Index (Platform.Web), returned result Microsoft.AspNetCore.Mvc.ViewResult in 73.3134ms. | url: https://restapi/| action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor|Executing ViewResult, running view Index. | url: | action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor|Executed ViewResult - view Index executed in 0.6892ms. | url: https://restapi/| action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Executed action Platform.Web.Controllers.TenantController.Index (Platform.Web) in 368.5915ms | url: https://restapi/| action: Index
2020-08-02 00:00:05.7606||INFO|Microsoft.AspNetCore.Hosting.Internal.WebHost|Request finished in 374.065ms 200 text/html; charset=utf-8 | url: https://restapi/| action: Index
2020-08-02 00:00:09.2884||INFO|Microsoft.AspNetCore.Hosting.Internal.WebHost|Request starting HTTP/1.1 GET http://10.32.0.24:5000/ | url: http://10.32.0.24/| action: 
2020-08-02 00:00:09.2884||INFO|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|Route matched with {action = "Index", controller = "Tenant"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Index() on controller Platform.Web.Controllers.TenantController (Platform.Web). | url: https://10.32.0.24/| action: Index

```

Here is a sample formatted the JSON Output (produced when output.elasticsearch is commented and used output.console):

```auto
{
  "@timestamp": "2020-08-13T01:16:27.981Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.2"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/home/nlog.log"
    }
  },
  "input": {
    "type": "log"
  },
  "url": "http://restapi/",
  "longdate": "2020-08-02 00:00:05.3881",
  "ecs": {
    "version": "1.1.0"
  },
  "agent": {
    "ephemeral_id": "29e028fc-bee9-44f0-bdb6-341844c9b3d2",
    "hostname": "filebeat-test",
    "id": "b34390d1-4b72-4073-aa5f-c967ffb02f4b",
    "version": "7.5.2",
    "type": "filebeat"
  },
  "eventid": "",
  "level": "INFO",
  "cloud": {
    "availability_zone": "",
    "instance": {
      "id": "572d498f50c62ccffb68ad90dcac2e95",
      "name": "ubuntu18-logs"
    },
    "machine": {
      "type": ""
    },
    "provider": "openstack"
  },
  "message": "Request starting HTTP/1.1 GET http://restapi/ ",
  "host": {
    "name": "filebeat-test",
    "hostname": "filebeat-test",
    "architecture": "x86_64",
    "os": {
      "version": "7 (Core)",
      "family": "redhat",
      "name": "CentOS Linux",
      "kernel": "4.15.0-106-generic",
      "codename": "Core",
      "platform": "centos"
    },
    "containerized": false
  },
  "action": "",
  "logger": "Microsoft.AspNetCore.Hosting.Internal.WebHost"
}

```

Not sure why it is not getting rendered in `kibana`.  
Is it because We are modifying the message field? (i.e: **dissect → drop message** (if data filed exist) **→ rename data field as message** (if data field exist)).  
or is it because of _`_ appearing in between? Like in 2nd and 10th line?  
Am I missing something in configuration? Unable to figure out why it is not getting displayed in `kibana`.  
I've checked the logs in both **Kibana → Logs(stream)** and **Kibana → Discover** \*.

---

<div class="post-metadata">

**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\
**Post date:** [August 18, 2020, 8:19am UTC](https://discuss.elastic.co/t/filebeat-dissect-processed-logs-not-appearing-in-kibana/245065/2 "2020-08-18T08:19:53Z")

</div>

Can you check whether the documents are successfully ingested in Elasticsearch by searching the corresponding index via Dev Tools or curl? Just to rule out this problem occurs outside of Kibana

---

<div class="post-metadata">

**Author:** ![MANI\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mani_m/32/73784_2.png) [@MANI\_M](https://discuss.elastic.co/u/MANI_M)\
**Post date:** [August 18, 2020, 1:27pm UTC](https://discuss.elastic.co/t/filebeat-dissect-processed-logs-not-appearing-in-kibana/245065/3 "2020-08-18T13:27:38Z")

</div>

Issue solved. It was due to `url` which is being used. Elasticsearch was throwing `"stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: object mapping for [url] tried to parse field [url] as object, but found a concrete value",` when I renamed url to someother key the issue got sorted out.

---

<div class="post-metadata">

**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\
**Post date:** [August 18, 2020, 1:37pm UTC](https://discuss.elastic.co/t/filebeat-dissect-processed-logs-not-appearing-in-kibana/245065/4 "2020-08-18T13:37:19Z")

</div>

Glad you figured it out. When data is not showing up in Kibana at all, most of the time it's something deeper in the stack and Elasticsearch/Logstash/Beats logs can help.

---

<div class="post-metadata">

**Author:** ![MANI\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mani_m/32/73784_2.png) [@MANI\_M](https://discuss.elastic.co/u/MANI_M)\
**Post date:** [August 18, 2020, 2:34pm UTC](https://discuss.elastic.co/t/filebeat-dissect-processed-logs-not-appearing-in-kibana/245065/5 "2020-08-18T14:34:54Z")

</div>

Got it. Thanks a lot.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 15, 2020, 2:35pm UTC](https://discuss.elastic.co/t/filebeat-dissect-processed-logs-not-appearing-in-kibana/245065/6 "2020-09-15T14:35:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.