Json { source => "message" } not parsing all of it

Elastic Stack Logstash
1

Hello, I'm new to Logstash and was wondering if someone could help with my filter and how can I parse the rest of my json fields the intent is to bring out the other fields in "message" such that I can map them to ECS:

here is what I have for my filter:

filter {
  if "redacted_platform" in [ls-source] {
    json {
      source => "message"
    }
    json {
      source => "message"
    }

    }
  }

here is what the result is coming out as the result:

       "message" => "<165>Feb 15 00:0000 redacted_platform.net.Example.com redacted_platform {\"model\":{\"description\":\"Test model used for testing alerting configuration.\",\"created\":{\"by\":\"System\"},\"edited\":{\"by\":\"Nobody\"},\"name\":\"Unrestricted Test Model\",\"priority\":5},\"device\":{\"ip\":\"0.1.2.3\",\"hostname\":\"test-device.example.com\",\"macaddress\":\"00:11:22:33:44:55\",\"vendor\":\"Test Vendor\",\"label\":\"Test Device\"},\"triggeredComponents\":[{\"metric\":{\"label\":\"Test Metric\"},\"triggeredFilters\":[{\"comparatorType\":\"display\",\"filterType\":\"Test Metric Filter\",\"trigger\":{\"value\":\"Test filter value\"}}]}],\"breachUrl\":\"\",\"pbid\":123,\"score\":1,\"creationTime\":123456,\"time\":123456}",
          "host" => "hostname.example.com",
      "log_type" => "redacted_platform",
    "@timestamp" => 2022-02-15 blah blah blah,
          "type" => "redacted_platform",
        "fields" => {
        "event_type" => "redacted_platform"
    },
     "ls-source" => "redacted_platform",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
          "port" => 12345,
      "@version" => "1",
           "ecs" => {
        "version" => "1.2.3"
    },
        "origin" => "redacted_platform",
         "input" => {
        "type" => "log"
    },
         "agent" => {
                "name" => "123b456c",
                  "id" => "123b456c",
             "version" => "1.2.3",
            "hostname" => "123b456c",
                "type" => "filebeat",
        "ephemeral_id" => "123b456c"
    },
           "log" => {
        "offset" => 0,
          "file" => {
            "path" => "/var/log/logstash/redacted_platform/test.json"
        }
    }
}
2

You need to remove the prefix from [message]. I would suggest something like this, but if you do not care about the prefix it could be something as simple as

mutate { gsub => [ "message", ".*{", "{" ] }
3

Thanks I'll be sure to try both those solutions out today!

what if i just wanted to remove below from [message]:

<165>Feb 15 00:0000 redacted_platform.net.Example.com redacted_platform

then do this a 3rd time:

   json {
      source => "message"
    }

would that work as well?
would that be sloppy and not best practice?

4

That's what the mutate+gsub I suggested would do. Then do the third json filter.

5

will do thank you so much! you people are so much help here!

6

If i did care about the prefix and wanted to parse out that data would it be the same thing as what you linked?

dissect { mapping => { "message" => "<%{pri}>%{f1} %{ts} [%{f2}] %{f3} ,%{[@metadata][json]}" } }
json { source => "[@metadata][json]" }

my current result came out with more back slashes

[2022-02-15T some time ][WARN ][org.logstash.dissect.Dissector][redacted_platform][123a456b789c] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"{pri}>%{f1} %{ts} [%{f2}] %{f3} ,%{[@metadata][json]}", "event"=>{"@version"=>"1", "ls-source"=>"redacted_platform", "message"=>"{\"host\":{\"name\":\"123a456b789c\"},\"@version\":\"1\",\"@timestamp\":\"2022-02-15T some time Z\",\"agent\":{\"version\":\"1.2.3\",\"name\":\"123a456b789c\",\"hostname\":\"12345abc\",\"type\":\"filebeat\",\"ephemeral_id\":\"123a456b789c\",\"id\":\"123a456b789c\"},\"log\":{\"offset\":123,\"file\":{\"path\":\"/var/log/logstash/redacted_platform.json\"}},\"fields\":{\"event_type\":\"redacted_platform\"},\"ecs\":{\"version\":\"1.2.3\"},\"log_type\":\"redacted_platform\",\"tags\":[\"some_company\",\"beats_input_codec_plain_applied\"],\"input\":{\"type\":\"log\"},\"message\":\"{\\\"type\\\":\\\"redacted_platform\\\",\\\"host\\\":\\\"sample.id.some_company\\\",\\\"port\\\":12345,\\\"origin\\\":\\\"redacted_platform\\\",\\\"@version\\\":\\\"1\\\",\\\"tags\\\":[\\\"_jsonparsefailure\\\"],\\\"@timestamp\\\":\\\"2022-02-15Tsome timeZ\\\",\\\"message\\\":\\\"<165>Feb 15 redacted_platform-01.example.some_company redacted_platform {\\\\\\\"model\\\\\\\":{\\\\\\\"description\\\\\\\":\\\\\\\"Test model used for testing alerting configuration.\\\\\\\",\\\\\\\"created\\\\\\\":{\\\\\\\"by\\\\\\\":\\\\\\\"System\\\\\\\"},\\\\\\\"edited\\\\\\\":{\\\\\\\"by\\\\\\\":\\\\\\\"Nobody\\\\\\\"},\\\\\\\"name\\\\\\\":\\\\\\\"Unrestricted Test Model\\\\\\\",\\\\\\\"priority\\\\\\\":5},\\\\\\\"device\\\\\\\":{\\\\\\\"ip\\\\\\\":\\\\\\\"0.1.2.3\\\\\\\",\\\\\\\"hostname\\\\\\\":\\\\\\\"test-device.example.com\\\\\\\",\\\\\\\"macaddress\\\\\\\":\\\\\\\"00:11:22:33:44:55\\\\\\\",\\\\\\\"vendor\\\\\\\":\\\\\\\"Test Vendor\\\\\\\",\\\\\\\"label\\\\\\\":\\\\\\\"Test Device\\\\\\\"},\\\\\\\"triggeredComponents\\\\\\\":[{\\\\\\\"metric\\\\\\\":{\\\\\\\"label\\\\\\\":\\\\\\\"Test Metric\\\\\\\"},\\\\\\\"triggeredFilters\\\\\\\":[{\\\\\\\"comparatorType\\\\\\\":\\\\\\\"display\\\\\\\",\\\\\\\"filterType\\\\\\\":\\\\\\\"Test Metric Filter\\\\\\\",\\\\\\\"trigger\\\\\\\":{\\\\\\\"value\\\\\\\":\\\\\\\"Test filter value\\\\\\\"}}]}],\\\\\\\"breachUrl\\\\\\\":\\\\\\\"\\\\\\\",\\\\\\\"pbid\\\\\\\":123,\\\\\\\"score\\\\\\\":1,\\\\\\\"creationTime\\\\\\\":12345,\\\\\\\"time\\\\\\\":12345}\\\"}\"}", "@timestamp"=>2022-02-15, "tags"=>["_dissectfailure"]}}
[2022-02-15T][WARN ][logstash.filters.json    ][redacted_platform][1234a456b789c] Error parsing json {:source=>"message", :raw=>"{\"value\":\"Test filter value\"}}]}],\"breachUrl\":\"\",\"pbid\":123,\"score\":1,\"creationTime\":12345,\"time\":`12345}", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"{"value":"Test filter value"}}]}],"breachUrl":"","pbid":123,"score":1,"creationTime":12345,"time":12345}"; line: 1, column: 0])
 at [Source: (byte[])"{"value":"Test filter value"}}]}],"breachUrl":"","pbid":123,"score":1,"creationTime":12345,"time":12345}"; line: 1, column: 31]>}
{
      "@version" => "1",
         "agent" => {
                "name" => "123b456c",
            "hostname" => "123b456c",
                "type" => "filebeat",
                  "id" => "123b456c",
             "version" => "1.2.3",
        "ephemeral_id" => "123b456c"
    },
        "origin" => "redacted_platform",
           "ecs" => {
        "version" => "1.2.3"
    },
      "log_type" => "redacted_platform",
       "message" => "{\"value\":\"Test filter value\"}}]}],\"breachUrl\":\"\",\"pbid\":123,\"score\":1,\"creationTime\":12345,\"time\":12345}",
     "ls-source" => "redacted_platform",
           "log" => {
          "file" => {
            "path" => "/var/log/logstash/redacted_platform.json"
        },
        "offset" => 123
    },
          "port" => 123,
          "type" => "redacted_platform",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
         "input" => {
        "type" => "log"
    },
          "host" => "sample.id.some_company",
        "fields" => {
        "event_type" => "redacted_platform"
    },
    "@timestamp" => 2022-02-15 it's 5'oclock somewhere
}
7

I would suggest

dissect { mapping => { "message" => "<%{pri}>%{ts} %{+ts} %{+ts} %{h} %{p} %{[@metadata][json]}" } }
8

would my filter look like this?

filter {
  if "some_platform" in [ls-source] {
    json {
      source => "message"
    }
    json {
      source => "message"
    }
    dissect { mapping => { "message" => "<%{pri}>%{ts} %{+ts} %{+ts} %{h} %{p} %{[@metadata][json]}" } }
    
    json {
      source => "message"
    }
 

    }
  }

if so i'm still getting jsonparsefailure and dissectfailure.

is my formatting wrong?

9

The third json filter should have source => "[@metadata][json]". But at the moment the third duplicates the second, so that must be failing.

10 
filter {
  if "darktrace" in [ls-source] {
    json {
      source => "message"
    }
    json {
      source => "[@metadata][json]"
    }
    dissect { mapping => { "message" => "<%{pri}>%{ts} %{+ts} %{+ts} %{h} %{p} %{[@metadata][json]}" } }
 

    }
  }

so something like this?

yeah it's still failing. would it help if I posted the output again?

11

No, I am saying

json { source => "message" }
json { source => "message" }
dissect { mapping => { "message" => "<%{pri}>%{ts} %{+ts} %{+ts} %{h} %{p} %{[@metadata][json]}" } }
json { source => "[@metadata][json]" }
12

Hello, just circling back to this to make sure i'm understanding so far is good,
the code below takes out the prefix but parsed out the json correct?

json { source => "message" }
json { source => "message" }
mutate { gsub => [ "message", ".*{", "{" ] }
json { source => "message" }

the intent of the code block is that the teal color portion is parsed and the red lined text gone:
image

and if I wanted the prefix that is red lined out in the picture above I would want to do the code block at the bottom correct?:

json { source => "message" }
json { source => "message" }
dissect { mapping => { "message" => "<%{pri}>%{ts} %{+ts} %{+ts} %{h} %{p} %{[@metadata][json]}" } }
json { source => "[@metadata][json]" }
13

I think both are correct.

14

Sounds good, what seems to be happening as of a result for the gsub solution doesn't seem to match the intent.

here is what i'm getting as a result:

"message" => "{\"value\":\"Test filter value\"}}]}],\"breachUrl\":\"\",\"pbid\":123,\"score\":1,\"creationTime\":123456,\"time\":123456}",
           "log" => {
        "offset" => 5232,
          "file" => {
            "path" => "/var/log/logstash/redacted_platform/redacted_platform-2022-02-22.json"
        }
    },
    "@timestamp" => 2022-02-22T19:30:01.547Z,
          "type" => "redacted_platform",
         "input" => {
        "type" => "log"
    },
          "port" => 4321,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "example_host.com",
      "log_type" => "redacted_platform",
        "fields" => {
        "event_type" => "redacted_platform"
    },
        "origin" => "redacted_platform",
     "ls-source" => "redacted_platform",
           "ecs" => {
        "version" => "1.11.0"
    }
}

based off the result above and comparing it to the picture the fields that should have been parsed but instead have disappeared. Is there something we are missing? it get's rid of the prefix but I don't see the other fields leading up to "value":"Test filter value"}}]}]

image

15

Try changing that to

mutate { gsub => [ "message", "^[^{]+", "" ] }
17

Looks like it worked, is there anything that sticks out to you that I may have missed?
It looks like now I can convert these fields to ECS now & delete "message" ! :slight_smile:

"message" => "{\"model\":{\"description\":\"Test model used for testing alerting configuration.\",\"created\":{\"by\":\"System\"},\"edited\":{\"by\":\"Nobody\"},\"name\":\"Unrestricted Test Model\",\"priority\":5},\"device\":{\"ip\":\"0.1.2.3\",\"hostname\":\"test-device.example.com\",\"macaddress\":\"00:11:22:33:44:55\",\"vendor\":\"Test Vendor\",\"label\":\"Test Device\"},\"triggeredComponents\":[{\"metric\":{\"label\":\"Test Metric\"},\"triggeredFilters\":[{\"comparatorType\":\"display\",\"filterType\":\"Test Metric Filter\",\"trigger\":{\"value\":\"Test filter value\"}}]}],\"breachUrl\":\"\",\"pbid\":123,\"score\":1,\"creationTime\":123456,\"time\":123456}",
"creationTime" => 123456,
      "fields" => {
"event_type" => "redacted_platform"
},
        "pbid" => 123,
"triggeredComponents" => [
[0] {
           "metric" => {
     "label" => "Test Metric"
 },
 "triggeredFilters" => [
     [0] {
             "filterType" => "Test Metric Filter",
         "comparatorType" => "display",
                "trigger" => {
             "value" => "Test filter value"
         }
     }
 ]
}
],
        "time" => 123456,
        "port" => 123,
  "@timestamp" => 2022-02-22T19:49:42.855Z,
    "log_type" => "redacted_platform",
       "score" => 1,
   "ls-source" => "redacted_platform",
   "breachUrl" => "",
      "device" => {
  "label" => "Test Device",
"hostname" => "test-device.example.com",
 "vendor" => "Test Vendor",
     "ip" => "0.1.2.3",
"macaddress" => "00:11:22:33:44:55"
},
       "agent" => {
     "name" => "12a34b56c",
"ephemeral_id" => "12a34b56c",
       "id" => "12a34b56c",
  "version" => "1.2.3",
 "hostname" => "12a34b56c",
     "type" => "filebeat"
},
        "type" => "redacted_platform",
        "tags" => [
[0] "_jsonparsefailure"
],
         "ecs" => {
"version" => "1.11.0"
},
       "input" => {
"type" => "log"
},
         "log" => {
"file" => {
 "path" => "/var/log/logstash/redacted_platform/redacted_platform-2022-02-22.json"
},
"offset" => 123
},
    "@version" => "1",
      "origin" => "redacted_platform",
        "host" => "example_host.com",
       "model" => {
"description" => "Test model used for testing alerting configuration.",
 "created" => {
 "by" => "System"
},
  "edited" => {
 "by" => "Nobody"
},
    "name" => "Unrestricted Test Model",
"priority" => 5
}
}
18

Looks like you are set.

19

TYSM you've been so much help! One last question: would a 4th json { source => "message" } be needed or is that unnecessary?

20

Here are the current results with the 4th json source message added:

{
    "triggeredComponents" => [
        [0] {
                      "metric" => {
                "label" => "Test Metric"
            },
            "triggeredFilters" => [
                [0] {
                    "comparatorType" => "display",
                           "trigger" => {
                        "value" => "Test filter value"
                    },
                        "filterType" => "Test Metric Filter"
                }
            ]
        }
    ],
             "@timestamp" => 2022-02-22T20:11:51.675Z,
                  "score" => 1,
                   "pbid" => 123,
                    "ecs" => {
        "version" => "1.11.0"
    },
               "log_type" => "redacted_platform",
                   "tags" => [
        [0] "_jsonparsefailure"
    ],
                   "type" => "redacted_platform",
                   "port" => 123456,
                 "origin" => "redacted_platform",
              "ls-source" => "redacted_platform",
              "breachUrl" => "",
                   "time" => 123456,
                   "host" => "example_host.com",
                  "input" => {
        "type" => "log"
    },
                  "model" => {
           "priority" => 5,
            "created" => {
            "by" => "System"
        },
             "edited" => {
            "by" => "Nobody"
        },
               "name" => "Unrestricted Test Model",
        "description" => "Test model used for testing alerting configuration."
    },
                  "agent" => {
             "version" => "123456",
                "name" => "123456",
                "type" => "filebeat",
        "ephemeral_id" => "123456",
                  "id" => "123456",
            "hostname" => "123456"
    },
               "@version" => "1",
                    "log" => {
          "file" => {
            "path" => "/var/log/logstash/redacted_platform/redacted_platform-2022-02-22.json"
        },
        "offset" => 123456
    },
           "creationTime" => 123456,
                 "device" => {
        "macaddress" => "00:11:22:33:44:55",
          "hostname" => "test-device.example.com",
            "vendor" => "Test Vendor",
                "ip" => "0.1.2.3",
             "label" => "Test Device"
    },
                "message" => "{\"model\":{\"description\":\"Test model used for testing alerting configuration.\",\"created\":{\"by\":\"System\"},\"edited\":{\"by\":\"Nobody\"},\"name\":\"Unrestricted Test Model\",\"priority\":5},\"device\":{\"ip\":\"0.1.2.3\",\"hostname\":\"test-device.example.com\",\"macaddress\":\"00:11:22:33:44:55\",\"vendor\":\"Test Vendor\",\"label\":\"Test Device\"},\"triggeredComponents\":[{\"metric\":{\"label\":\"Test Metric\"},\"triggeredFilters\":[{\"comparatorType\":\"display\",\"filterType\":\"Test Metric Filter\",\"trigger\":{\"value\":\"Test filter value\"}}]}],\"breachUrl\":\"\",\"pbid\":123,\"score\":1,\"creationTime\":123456,\"time\":123456}",
                 "fields" => {
        "event_type" => "redacted_platform"
    }
}
21

A 4th json filter would only be needed if the previous json filter had overwritten the [message] field because there was nested JSON.

1 Like