# Multiline is not displayed in the "Visualize" or "Dashboard" screen

**URL:** <https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427>\
**Category:** Logstash\
**Created:** [January 24, 2020, 1:42pm UTC](https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427 "2020-01-24T13:42:12Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![totopo\_loco](https://avatars.discourse-cdn.com/v4/letter/t/71e660/32.png) [@totopo\_loco](https://discuss.elastic.co/u/totopo_loco)\
**Post date:** [January 24, 2020, 1:42pm UTC](https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427/1 "2020-01-24T13:42:13Z")

</div>

Hello Dear Community,

I am having problems to display multi line messages after they were parsed in Kibana 7.5.2.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/3/13038d541e53cb35ee72ad8ac72e127e1ba8c406.png)

In the visualization and dashboard in Kibana the "message.keyword" is not shown when multi line messages are in the logs.  
Let me explain my configuration and my issue with multi line, I hope I can give the needed information to troubleshoot it, so here we go:

I want some sample logs to be displayed, examples of these logs:

```
2020-01-15 08:01:02,575 [DefaultMessageListenerContainer-1] INFO - a
2020-01-15 08:01:02,575 [DefaultMessageListenerContainer-1] ERROR - b
2020-01-15 08:01:02,575 [DefaultMessageListenerContainer-1] WARN - c
2020-01-15 08:01:02,575 [DefaultMessageListenerContainer-1] DEBUG - d

```

Something more realistic:

```
2020-01-21 08:01:02,575 [DefaultMessageListenerContainer-1] DEBUG - <<<<<The values of DDMEAS are>>>>>
2020-01-21 08:01:02,575 [DefaultMessageListenerContainer-1] DEBUG - Measurement type udc id is: <XXXXX>
2020-01-21 08:01:02,575 [DefaultMessageListenerContainer-1] DEBUG - External measurement code: <YYYYY>
2020-01-21 08:01:02,575 [DefaultMessageListenerContainer-1] DEBUG - DtsContext SvcPtId: <7415>

```

When the messages are in a single line all message are correctly displayed after I created the visualization and the dashboard:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/7/07d8de69988c36ad4e8b8561d7ce88e2fc540c59.png)

So far so good, now let's drop the following log file:

```
2020-01-22 08:01:02,576 [DefaultMessageListenerContainer-1] DEBUG - <<<<<The values of XXXXX are>>>>>
2020-01-22 08:01:02,577 [DefaultMessageListenerContainer-1] DEBUG - Measurement type udc id is: <9>
2020-01-22 08:01:02,578 [DefaultMessageListenerContainer-1] DEBUG - Returning BO : TimezoneBO
id[2307]
lastUpdDt[Tue Nov 06 07:34:13 UTC 2018]
insertDt[Tue Nov 06 07:34:13 UTC 2018]
insertBy[203]
lastUpdBy[203]
orgId[102]
recVersionNum[1]
dataSrc[SOURCE1]
name[Europe/Vienna]
javaName[Europe/Vienna]
displayStdOffset[GMT+01:00]
hasDst[Y] from the boBag
2020-01-22 08:01:02,579 [DefaultMessageListenerContainer-1] DEBUG - External measurement code: <10>
2020-01-22 08:01:02,580 [DefaultMessageListenerContainer-1] DEBUG - DtsContext SvcPtId: <7415>

```

As you can see the multi-line message is the one that begins with "Returning..."

When I open the "Discover" view I can see the line was parsed correctly:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/3/43a4acc622b47e26e28687a1265fb75cca87496b.png)

I open it in the same view and says:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/6/36e9ef374d0628ce457db906abd5c874c42fd4b8.png)

"No cached mapping for this field. Refresh field list from the Management \> Index Patterns page"

And that row does not appear in the Dashboard:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/3/d37709b9d927a7f7037e6c6e1af934cc31a8dac7.png)

So as suggested I refresh the list:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/2/327dc40bf2df91f7cb2e486e1147b856aa3a1981.png)

I get back to the "Discover" screen and I can see the admiration mark is gone:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/0/e0968747ea12391915e4ec78ba37df5c496551e1.png)

But in the visualization is not appearing:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/b/ab64bec88f439f5108293f51aca0e7d64beb6222.png)

Any file I drop with multi lines is not shown in the dashboard, could someone please advice what needs to be done to have displayed those logs:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/e/cea14da63361c4a78fd51fa8fd70646db4587e26.png)

**Here is my configuration**

- filebeat.yml (only relevant) (As an image as the formatting is breaking all)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/d/9dc58783bd6dc08ef3c48190f2030fac23c95a2d.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/7/0789ff0ff0ada17a621a16c277fb4bcccc79b0c7.png)

logstash configuration:

```
input {
  beats {
    port => 5044
    ssl => false
  }
}

# filter extracting from log4j log fields: timestamp, thread, servity
filter {
fingerprint {
    method => "SHA1"
    key => "KEY"
}

  grok {
    #usefull is http://grokdebug.herokuapp.com/
    match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp}\s{0,4}\[%{DATA:thread}\]\s{0,4}%{LOGLEVEL:loglevel}\s{0,4}-\s{0,4}%{GREEDYDATA:message}" ]
    overwrite => ["message"]
  }
  date {
    match => ["timestamp" , "yyyy-MM-dd HH:mm:ss,SSS"]
  }
}

output {
         elasticsearch {
            hosts => "localhost:9200"
            ilm_rollover_alias => "index-vagrant"
            ilm_pattern => "000001"
            ilm_policy => "vagrant-policy"
            document_id => "%{fingerprint}"
         }
         stdout {
            codec => rubydebug
         }
}

```

This is the script used to create the index definition and the policy:

```
#!/bin/bash

echo "--- creating policy ---"
curl -0 -v -X PUT http://localhost:9200/_ilm/policy/vagrant-policy \
-H "Expect:" \
-H 'Content-Type: application/json; charset=utf-8' \
-d @- << EOF

{
  "policy": {
"phases": {
  "hot": {
    "actions": {
      "rollover": {
        "max_age": "5d",
        "max_size": "10G"
      },
      "set_priority": {
        "priority": 100
      }
    }
  },
  "warm": {
    "min_age": "5d",
    "actions": {
      "forcemerge": {
        "max_num_segments": 1
      },
      "allocate": {
        "number_of_replicas": 0
      },
      "set_priority": {
        "priority": 50
      }
    }
  },
  "cold": {
    "min_age": "14d",
    "actions": {
      "freeze": {}
    }
  },
  "delete": {
    "min_age": "14d",
    "actions": {
      "delete": {}
    }
  }
}
  }
}
EOF

echo "--- creating template ---"
curl -0 -v -X PUT http://localhost:9200/_template/vagrant-policy \
-H "Expect:" \
-H 'Content-Type: application/json; charset=utf-8' \
-d @- << EOF
{
  "index_patterns": [
"index-vagrant-*"
  ],
  "settings": {
"index": {
  "number_of_shards": 1,
  "number_of_replicas": 1,
  "lifecycle.name": "vagrant-policy",
  "lifecycle.rollover_alias": "index-vagrant"
}
  },
  "mappings" : {
"properties" : {
  "HTTP_RESULT_CODE" : { "type" : "short" },
  "PROCESSING_TIME" : { "type" : "integer" },
	  "propertyValue": { "type" : "integer" }	    	  
}
  }
}
EOF

echo "--- creating index alias ---"
curl -0 -v -X PUT http://localhost:9200/index-vagrant-000001 \
-H "Expect:" \
-H 'Content-Type: application/json; charset=utf-8' \
-d @- << EOF
{
  "aliases": {
"index-vagrant": {
  "is_write_index": true
}
  }
}
EOF

```

Not sure this is relevant but this elasticsearch log (a part):

```
2020-01-24T13:44:04,217][INFO][o.e.c.r.a.AllocationService] [ubuntu1804.localdomain] Cluster health status changed from [RED] to [YELLOW] (reason: [shards s
[2020-01-24T13:44:10,448][INFO][o.e.c.m.MetaDataIndexTemplateService] [ubuntu1804.localdomain] adding template [.management-beats] for index patterns [.manag
[2020-01-24T13:44:41,333][INFO][o.e.c.m.MetaDataDeleteIndexService] [ubuntu1804.localdomain] [index-vagrant-000001/mKZJEITVQUaoAj9NU5uBag] deleting index
[2020-01-24T13:45:01,231][INFO][o.e.c.m.MetaDataCreateIndexService] [ubuntu1804.localdomain] [index-vagrant] creating index, cause [auto(bulk api)], template
[2020-01-24T13:45:01,459][INFO][o.e.c.m.MetaDataMappingService] [ubuntu1804.localdomain] [index-vagrant/BPsCIvDnQuWY7t7Aae1oBA] create_mapping [_doc]
[2020-01-24T13:49:18,427][INFO][o.e.c.m.MetaDataDeleteIndexService] [ubuntu1804.localdomain] [index-vagrant/BPsCIvDnQuWY7t7Aae1oBA] deleting index
[2020-01-24T13:49:29,842][INFO][o.e.c.m.MetaDataCreateIndexService] [ubuntu1804.localdomain] [index-vagrant] creating index, cause [auto(bulk api)], template
[2020-01-24T13:49:30,020][INFO][o.e.c.m.MetaDataMappingService] [ubuntu1804.localdomain] [index-vagrant/vpUJlWvVTmaH2eoQyFilqg] create_mapping [_doc]
[2020-01-24T13:52:10,597][INFO][o.e.c.m.MetaDataMappingService] [ubuntu1804.localdomain] [.kibana_1/gyRLAhm8TGaB1o7qsn5WAQ] update_mapping [_doc]
[2020-01-24T14:03:21,438][INFO][o.e.c.m.MetaDataMappingService] [ubuntu1804.localdomain] [index-vagrant/vpUJlWvVTmaH2eoQyFilqg] update_mapping [_doc]

```

I can see also the message received in logstash:

```
{
     "@timestamp" => 2020-01-11T07:01:02.578Z,
    "fingerprint" => "7f4b845356d25eaf7e507accd319382632522f72",
        "message" => "Returning BO : TimezoneBO\nid[2307]\nlastUpdDt[Tue Nov 06 07:34:13 UTC 2018]\ninsertDt[Tue Nov 06 07:34:13 UTC 2018]\ninsertBy[203]\nlastUpdBy[203]\norgId[102]\nrecVersionNum[1]\ndataSrc[SOURCE1]\nname[Europe/Vienna]\njavaName[Europe/Vienna]\ndisplayStdOffset[GMT+01:00]\nhasDst[Y] from the boBag",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
       "@version" => "1",
          "input" => {
        "type" => "log"
    },
          "agent" => {
        "ephemeral_id" => "e7fce2eb-1edd-4ce4-bff8-2ac30deefcb4",
                  "id" => "0353296c-cc91-4943-9278-1cb4716bea08",
             "version" => "7.5.2",
            "hostname" => "ubuntu1804.localdomain",
                "type" => "filebeat"
    },
            "log" => {
          "file" => {
            "path" => "/vagrant/uu.log"
        },
        "offset" => 203,
         "flags" => [
            [0] "multiline"
        ]
    },
           "host" => {
        "name" => "ubuntu1804.localdomain"
    },
         "thread" => "DefaultMessageListenerContainer-1",
      "timestamp" => "2020-01-11 08:01:02,578",
            "ecs" => {
        "version" => "1.1.0"
    },
       "loglevel" => "DEBUG"
}

```

I hope all information is ok. Thanks and regards

~Marco T.

---

<div class="post-metadata">

**Author:** ![totopo\_loco](https://avatars.discourse-cdn.com/v4/letter/t/71e660/32.png) [@totopo\_loco](https://discuss.elastic.co/u/totopo_loco)\
**Post date:** [January 24, 2020, 5:00pm UTC](https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427/2 "2020-01-24T17:00:52Z")

</div>

I have tried this, but no luck yet ☹ (truncate:maxHeight)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/3/53a1de52d4a01b96aa086716969b9279417db6a8.png)

---

<div class="post-metadata">

**Author:** ![totopo\_loco](https://avatars.discourse-cdn.com/v4/letter/t/71e660/32.png) [@totopo\_loco](https://discuss.elastic.co/u/totopo_loco)\
**Post date:** [January 24, 2020, 5:27pm UTC](https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427/3 "2020-01-24T17:27:07Z")

</div>

Even the numbers match:

I created this visualization testing the following logs:

```
2020-01-15 08:01:02,575 [DefaultMessageListenerContainer-1] INFO - a
2020-01-15 08:01:02,575 [DefaultMessageListenerContainer-1] ERROR - b
2020-01-15 08:01:02,575 [DefaultMessageListenerContainer-1] WARN - c
2020-01-15 08:01:02,575 [DefaultMessageListenerContainer-1] DEBUG - d

```

And:

```
2020-01-01 08:01:02,576 [DefaultMessageListenerContainer-1] DEBUG - <<<<<The values of XXXXX are!!!!>>>>>
2020-01-01 08:01:02,577 [DefaultMessageListenerContainer-1] DEBUG - Measurement type udc id is: <1234>
2020-01-01 08:01:02,578 [DefaultMessageListenerContainer-1] DEBUG - Returning BO : LocoBO
id[2307]
lastUpdDt[Tue Nov 06 07:34:13 UTC 2018]
insertDt[Tue Nov 06 07:34:13 UTC 2018]
insertBy[203]
lastUpdBy[203]
orgId[102]
recVersionNum[1]
dataSrc[SOURCE1]
name[Europe/Vienna]
javaName[Europe/Vienna]
displayStdOffset[GMT+01:00]
hasDst[Y] from the boBag
2020-01-01 08:01:02,579 [DefaultMessageListenerContainer-1] DEBUG - External measurement code: <99>
2020-01-01 08:01:02,580 [DefaultMessageListenerContainer-1] DEBUG - DtsContext SvcPtId: <8383>

```

All would be excellent if the message with the multi line would be displayed:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/c/2c85095a07edbfcfe4cb7b253364e3ca37647164.png)

The only thing I found is when selecting "Show missing values" the multi line message that is supposed to be there appears as "Missing":

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/0/40636527f1e70873ad7e39d0cdf8ea042d60ae51.png)

---

<div class="post-metadata">

**Author:** ![totopo\_loco](https://avatars.discourse-cdn.com/v4/letter/t/71e660/32.png) [@totopo\_loco](https://discuss.elastic.co/u/totopo_loco)\
**Post date:** [January 24, 2020, 7:37pm UTC](https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427/4 "2020-01-24T19:37:47Z")

</div>

I am testing with a single multi line message, somehow appears in the dashboard:

```
[root@ubuntu1804 20:23:19] vagrant $ cat mul.log
2020-01-24 08:01:02,578 [DefaultMessageListenerContainer-1] INFO - a
2020-01-24 08:01:02,579 [DefaultMessageListenerContainer-1] ERROR - b
2020-01-24 08:01:02,580 [DefaultMessageListenerContainer-1] WARN - c
2020-01-24 08:01:02,581 [DefaultMessageListenerContainer-1] WARN - kd kf
kfk
2020-01-24 08:01:02,582 [DefaultMessageListenerContainer-1] DEBUG - d

```

Result:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/9/c956159559dd9c685562f98e6130eeea9b00ba7a.png)

So, seems the problem is a multi line message, does someone know how to display something bigger or please advise which visualization is better?

Regards

~M

---

<div class="post-metadata">

**Author:** ![totopo\_loco](https://avatars.discourse-cdn.com/v4/letter/t/71e660/32.png) [@totopo\_loco](https://discuss.elastic.co/u/totopo_loco)\
**Post date:** [January 24, 2020, 8:11pm UTC](https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427/5 "2020-01-24T20:11:00Z")

</div>

Hello,

I am getting deeper into this issue, looks like strings like this:

```
id[2307]
lastUpdDt[Tue Nov 06 07:34:13 UTC 2018]
insertDt[Tue Nov 06 07:34:13 UTC 2018]
insertBy[203]
lastUpdBy[203]
orgId[102]
recVersionNum[1]
dataSrc[SOURCE1]
name[Europe/Vienna]
javaName[Europe/Vienna]
displayStdOffset[GMT+01:00]

```

Cause a problem, the advice of an export is very much welcomed 🙂

I can see that if I have logs like this, things work quite normal, how to clean the multi lines in logstash?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/5/d5306f705929b52da93a98c5ec831a9dfb026e73.png)

---

<div class="post-metadata">

**Author:** ![totopo\_loco](https://avatars.discourse-cdn.com/v4/letter/t/71e660/32.png) [@totopo\_loco](https://discuss.elastic.co/u/totopo_loco)\
**Post date:** [January 24, 2020, 10:25pm UTC](https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427/6 "2020-01-24T22:25:13Z")

</div>

Kind of works with the following filter in logstash:

```
filter {

mutate {
    gsub => ["message", "\n", " (LF) "]
}

fingerprint {
    method => "SHA1"
    key => "KEY"
}

  grok {
    #usefull is http://grokdebug.herokuapp.com/
    break_on_match => false
    #match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp}\s{0,4}\[%{DATA:thread}\]\s{0,4}%{LOGLEVEL:loglevel}\s{0,4}-\s{0,4}%{GREEDYDATA:message}" ]
    match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp}\s{0,4}\[%{DATA:thread}\]\s{0,4}%{LOGLEVEL:loglevel}\s{0,4}-\s{0,4}(?<message>(.|\r|\n)*)" ]
    overwrite => ["message"]
  }
  date {
    match => ["timestamp" , "yyyy-MM-dd HH:mm:ss,SSS"]
  }
  truncate {
    length_bytes => 100
    fields => "message"
    add_tag => ["shortened_message"]
  }
}

```

Maybe someone has a better idea?

Regards

~M

---

<div class="post-metadata">

**Author:** ![totopo\_loco](https://avatars.discourse-cdn.com/v4/letter/t/71e660/32.png) [@totopo\_loco](https://discuss.elastic.co/u/totopo_loco)\
**Post date:** [February 7, 2020, 11:51pm UTC](https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427/7 "2020-02-07T23:51:34Z")

</div>

The discover view is just another visualisation that can be added to the Dashboard, issue solved.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 6, 2020, 11:51pm UTC](https://discuss.elastic.co/t/multiline-is-not-displayed-in-the-visualize-or-dashboard-screen/216427/8 "2020-03-06T23:51:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.