Split message in Logstash does not display complete message

Elastic Stack Logstash
1

Hi All,

The stdout log of an application is tokenized with a delimiter ~|~. There are a total of 5 delimiters. An excerpt from log is as below:

2023-10-27 11:03:41,294~|~INFO~|~host.com~|~com.controller.LoginController~|~Key, Value :  cookie = BCSID=9f347f4e-475b-45b1-b2d5-8d94cd65e608; _cls_v=f492d25b-c0ff-4417-bb45-eee8eb12ff45; _cls_s=f4c06851-b23f-4ae4-8b1a-2febe4c48ae4:0; TU7NGTeXTlLCIxKTEy3M4kU4uGl58QAwyCc2Dmks/SHnWUcy5b6YYm0UWtyV5ipSjCAgA==~-1~-1~-1; ak_bmsc=7ADB4FB55525821AD4469FC6CC9D83BB~000000000000000000000000000000~YAAQcpAFt0LXawYuYXNe9/pvebJHLWHHgLgOqLlaUjwa9nDr5J5tXryUbHFCDyagXeR/kr657VDa3MrckLiKVx5kg==; bm_sz=7CBFD8FECC1D9F4611FCCA8BB0B3D629~YAAQczsvF4kmtlSLAQAAdSOfcRUTDvX9C6Djc4BmniUser=%D2g%u0118%A9%u0122%B7%D8i%B1z%98h%B1%7D%7EJ; prtlnkt=20ec3ef6-161a-43e3-ab35-c2f31a82bce0; AuditUserId=koko7044; E2eeUserID=a29rbzcwNDQ; AuthType=fam; ADRUM_BTa=R:46|g:aec13a90-53ed-496d-93cc-4d781100b238|n:customer1_a6584e1e-55e3-4797-bf22-e75ff520b393; ADRUM_BT1=R:46|i:60813|e:34; bm_sv=C893149D924163E571F206802682573D~YAAQjXjOF1dK4mqLAQAANpGpcRXSejUixVg2sTzxdKRU5D03ZviQBFm26TgT7YWtNv1Eb8EETlC3n8mVVWtQ7u56Cj7EkY41oKafhXvEp1uc=~1~|~

Logstash receives message from filebeat and its config file splits the message in 5 tokens as follows:

mutate {
                        split => ["message", "~|~"]
                        add_field =>{
                          "createdTime" => "%{[message][0]}"
                          "logLevel" => "%{[message][1]}"
                          "hostname" => "%{[message][2]}"
                          "class" => "%{[message][3]}"
                          "Message" => "%{[message][4]}"

I understand multiline option could be exercised at filebeat level, but my question is when we are tokenizing the entire message in 5 distinct tokens, shouldn't the complete message get displayed as %{[message][4]}" ?

Because %{[message][4]}" is long text with a number of lines, it results in it not showing completely. Also the lines are not the same in number and pattern every time.

Please guide

2

What is the output you expect?

From what you shared the value [message][4] will be:

Key, Value : cookie = BCSID=9f347f4e-475b-45b1-b2d5-8d94cd65e608; _cls_v=f492d25b-c0ff-4417-bb45-eee8eb12ff45; _cls_s=f4c06851-b23f-4ae4-8b1a-2febe4c48ae4:0; TU7NGTeXTlLCIxKTEy3M4kU4uGl58QAwyCc2Dmks/SHnWUcy5b6YYm0UWtyV5ipSjCAgA==~-1~-1~-1; ak_bmsc=7ADB4FB55525821AD4469FC6CC9D83BB~000000000000000000000000000000~YAAQcpAFt0LXawYuYXNe9/pvebJHLWHHgLgOqLlaUjwa9nDr5J5tXryUbHFCDyagXeR/kr657VDa3MrckLiKVx5kg==; bm_sz=7CBFD8FECC1D9F4611FCCA8BB0B3D629~YAAQczsvF4kmtlSLAQAAdSOfcRUTDvX9C6Djc4BmniUser=%D2g%u0118%A9%u0122%B7%D8i%B1z%98h%B1%7D%7EJ; prtlnkt=20ec3ef6-161a-43e3-ab35-c2f31a82bce0; AuditUserId=koko7044; E2eeUserID=a29rbzcwNDQ; AuthType=fam; ADRUM_BTa=R:46|g:aec13a90-53ed-496d-93cc-4d781100b238|n:customer1_a6584e1e-55e3-4797-bf22-e75ff520b393; ADRUM_BT1=R:46|i:60813|e:34; bm_sv=C893149D924163E571F206802682573D~YAAQjXjOF1dK4mqLAQAANpGpcRXSejUixVg2sTzxdKRU5D03ZviQBFm26TgT7YWtNv1Eb8EETlC3n8mVVWtQ7u56Cj7EkY41oKafhXvEp1uc=~1

This is the output that the filters you shared will created based on that sample message:

[2023-10-27T13:30:11,122][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
       "logLevel" => "INFO",
        "Message" => "Key, Value :  cookie = BCSID=9f347f4e-475b-45b1-b2d5-8d94cd65e608; _cls_v=f492d25b-c0ff-4417-bb45-eee8eb12ff45; _cls_s=f4c06851-b23f-4ae4-8b1a-2febe4c48ae4:0; TU7NGTeXTlLCIxKTEy3M4kU4uGl58QAwyCc2Dmks/SHnWUcy5b6YYm0UWtyV5ipSjCAgA==~-1~-1~-1; ak_bmsc=7ADB4FB55525821AD4469FC6CC9D83BB~000000000000000000000000000000~YAAQcpAFt0LXawYuYXNe9/pvebJHLWHHgLgOqLlaUjwa9nDr5J5tXryUbHFCDyagXeR/kr657VDa3MrckLiKVx5kg==; bm_sz=7CBFD8FECC1D9F4611FCCA8BB0B3D629~YAAQczsvF4kmtlSLAQAAdSOfcRUTDvX9C6Djc4BmniUser=%D2g%u0118%A9%u0122%B7%D8i%B1z%98h%B1%7D%7EJ; prtlnkt=20ec3ef6-161a-43e3-ab35-c2f31a82bce0; AuditUserId=koko7044; E2eeUserID=a29rbzcwNDQ; AuthType=fam; ADRUM_BTa=R:46|g:aec13a90-53ed-496d-93cc-4d781100b238|n:customer1_a6584e1e-55e3-4797-bf22-e75ff520b393; ADRUM_BT1=R:46|i:60813|e:34; bm_sv=C893149D924163E571F206802682573D~YAAQjXjOF1dK4mqLAQAANpGpcRXSejUixVg2sTzxdKRU5D03ZviQBFm26TgT7YWtNv1Eb8EETlC3n8mVVWtQ7u56Cj7EkY41oKafhXvEp1uc=~1",
    "createdTime" => "2023-10-27 11:03:41,294",
     "@timestamp" => 2023-10-27T16:30:11.114680581Z,
           "host" => "lab",
        "message" => [
        [0] "2023-10-27 11:03:41,294",
        [1] "INFO",
        [2] "host.com",
        [3] "com.controller.LoginController",
        [4] "Key, Value :  cookie = BCSID=9f347f4e-475b-45b1-b2d5-8d94cd65e608; _cls_v=f492d25b-c0ff-4417-bb45-eee8eb12ff45; _cls_s=f4c06851-b23f-4ae4-8b1a-2febe4c48ae4:0; TU7NGTeXTlLCIxKTEy3M4kU4uGl58QAwyCc2Dmks/SHnWUcy5b6YYm0UWtyV5ipSjCAgA==~-1~-1~-1; ak_bmsc=7ADB4FB55525821AD4469FC6CC9D83BB~000000000000000000000000000000~YAAQcpAFt0LXawYuYXNe9/pvebJHLWHHgLgOqLlaUjwa9nDr5J5tXryUbHFCDyagXeR/kr657VDa3MrckLiKVx5kg==; bm_sz=7CBFD8FECC1D9F4611FCCA8BB0B3D629~YAAQczsvF4kmtlSLAQAAdSOfcRUTDvX9C6Djc4BmniUser=%D2g%u0118%A9%u0122%B7%D8i%B1z%98h%B1%7D%7EJ; prtlnkt=20ec3ef6-161a-43e3-ab35-c2f31a82bce0; AuditUserId=koko7044; E2eeUserID=a29rbzcwNDQ; AuthType=fam; ADRUM_BTa=R:46|g:aec13a90-53ed-496d-93cc-4d781100b238|n:customer1_a6584e1e-55e3-4797-bf22-e75ff520b393; ADRUM_BT1=R:46|i:60813|e:34; bm_sv=C893149D924163E571F206802682573D~YAAQjXjOF1dK4mqLAQAANpGpcRXSejUixVg2sTzxdKRU5D03ZviQBFm26TgT7YWtNv1Eb8EETlC3n8mVVWtQ7u56Cj7EkY41oKafhXvEp1uc=~1"
    ],
       "@version" => "1",
       "hostname" => "host.com",
          "class" => "com.controller.LoginController"
}
[2023-10-27T13:30:11,289][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
3

thanks. So in Kibana it does not show [message][4] as a complete message.

It seems to break it down into different sub-messages. Also as [message][4] is non-standard so different responses are seen. In the case below, it seems that it has taken the message (which is well within the token) as new line and lost formatting.

I expect the entire message (regardless of complexity) to appear completely as [message][4]

Capture
Capture543×868 38 KB

4

This image shows that your message is completely different from the one you shared in your first post.

As you see the value for the message field is just "name": "displayPassword",, so there is nothing to split because the string ~|~ does not exist in your message, so you will not have [message][X].

It seems that your source message is a multiline message, you would need to configure filebeat to consume the multiline message as one single message.

5

thanks @leandrojmp

Yes the content of message field was different the second time around. The message comes in various sizes and formats, because of which I thought I would be able to encapsulate the entire message within one delimiter i.e. ~|~

Also you are right that the message in stdout is a multiline message. But as this message does not conform to a set format, it is hard to configure multiline message pattern in filebeat.

Is there a "one size fits all" kind of multiline pattern that I could follow at the filebeat level?

TIA

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.