Parsing result from custom Beats

Elastic Stack Logstash
1

I have created my own Beats and I am passing data from that Beats to logstash.

This is what I get when I print to stdout in logstash(stdout { codec => "rubydebug"})

{
         "agent" => {
                "type" => "custombeat",
        "ephemeral_id" => "xxxxxxxxxxxx",
                  "id" => "xxxxxxxxxxxx",
             "version" => "8.0.0",
                "name" => "helloWorld"
    },
 
          "tags" => [
        [0] "beats_input_raw_event"
    ],
      "log_line" => {
          "datetime" => "2021-06-10 00:15:16.152713",
        "data" => {
            "value" => "4",
              "type" => "LOSS"
        }
    },
           "ecs" => {
        "version" => "1.8.0"
    },
          "host" => {
        "containerized" => false,
         "architecture" => "x86_64",
                 "name" => "helloWorld",
                  "mac" => [
            [0] "xxxxxxxx",
            [1] "xxxxxxxx",
            [2] "xxxxxxxx"
        ],
             "hostname" => "helloWorld",
                   "os" => {
              "kernel" => "5.4.72-microsoft-standard-WSL2",
              "family" => "debian",
                "name" => "Ubuntu",
            "platform" => "ubuntu",
                "type" => "linux",
            "codename" => "focal",
             "version" => "20.04.2 LTS (Focal Fossa)"
        },
                   "ip" => [
            [0] "xxxxxxxxxxx",
            [1] "xxxxxxxxxxxxxxx"
        ]
    },
      "@version" => "1",
    "@timestamp" => 2021-06-10T12:52:43.867Z
}

All I want to print as output is the log_line bit and nothing else. I also want to flatten the result when I print it (Right now, log_line is a nested json).

So for this example, I want to print-

 "datetime" => "2021-06-10 00:15:16.152713",
 "value" => "4",
 "type" => "LOSS"

I have been having a hard time even referring to the log_line bit in logstash and I've tried a lot of things. At this point, I am not even sure what the correct first step is. I have been stuck for a while... I'd appreciate any help or direction with the logstash filters.

2

Use prune to delete the fields you do not want

prune { whitelist_names => [ "log_line" ] }

Then mutate

mutate {
    add_field => {
        "datetime" => "%{[log_line][datetime]}"
        "type" => "%{[log_line][date][type]}"
        "value" => "%{[log_line][date][value]}"
    }
}
mutate { remove_field => [ "log_line" ] }
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.