# Unexpected error with Beats input plugin

**URL:** <https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255>\
**Category:** Logstash\
**Created:** [August 12, 2021, 7:58pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255 "2021-08-12T19:58:32Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![panagiss](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@panagiss](https://discuss.elastic.co/u/panagiss)\
**Post date:** [August 12, 2021, 7:58pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/1 "2021-08-12T19:58:32Z")

</div>

Previously i used JSON data from a file (that were previously taken by beats) to test my logstash conf and filtering. I had no problems, but when i try to read straight from auditbeat with the Beats input plugin, Logstash prints few errors and also the output doesn't appear.

here is sample of the error form logstash while receiving auditbeat events:

```auto

[2021-08-12T19:46:31,639][WARN][logstash.filters.json][main][699e35f42d506ad9de208d3e678ec089b2419d6ee7373c8b174f7f900352ed56] Error parsing json {:source=>"message", :raw=>"Process containerd-shim (PID: 5305) by user root is RUNNING", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Process': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"Process containerd-shim (PID: 5305) by user root is RUNNING"; line: 1, column: 9]>}
[2021-08-12T19:46:31,640][WARN][logstash.filters.json][main][699e35f42d506ad9de208d3e678ec089b2419d6ee7373c8b174f7f900352ed56] Error parsing json {:source=>"message", :raw=>"ERROR for PID 5323: failed to hash executable /pause for PID 5323: failed to stat file /pause: stat /pause: no such file or directory", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'ERROR': was expecting ('true', 'false' or 'null')

```

and another sample:

```auto
[2021-08-12T19:51:56,036][WARN][logstash.filters.json][main][699e35f42d506ad9de208d3e678ec089b2419d6ee7373c8b174f7f900352ed56] Error parsing json {:source=>"message", :raw=>"Process sleep (PID: 26212) by user root STOPPED", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Process': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"Process sleep (PID: 26212) by user root STOPPED"; line: 1, column: 9]>}
[2021-08-12T19:52:06,041][WARN][logstash.filters.json][main][699e35f42d506ad9de208d3e678ec089b2419d6ee7373c8b174f7f900352ed56] Error parsing json {:source=>"message", :raw=>"Process sleep (PID: 26625) by user root STARTED", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Process': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"Process sleep (PID: 26625) by user root STARTED"; line: 1, column: 9]>}

```

Here is the logstash conf:

```auto
input{
    # file{
    # path => "/usr/share/logstash/config/auditbeat.json"
    # start_position => "beginning"
    # ecs_compatibility => "v1"
    # # codec => "json"
    # }
    beats {
        port => 5044
        ecs_compatibility => "v1"
    }
}

filter{

    json {
        source => "message"
        target => "[@metadata][message]"
        remove_field => "message"
    }
    if [@metadata][message][destination][ip] {
        mutate{
            add_field => { "[dst_ip]" => "%{[@metadata][message][destination][ip]}"}
        }
    }

    if [@metadata][message][source][ip] {
        mutate{
            add_field => { "[src_ip]" => "%{[@metadata][message][source][ip]}"}
        }
    }
    if ![src_ip] and ![dst_ip]{
        drop {}
    }

    mutate{
        remove_field => ["[@metadata][message]" ]
    }

.
.
.
.

output{
    # stdout {
    # codec => rubydebug
    # { metadata => true }
    # }
    file{
        path => "/usr/share/logstash/config/enriched-auditbeat.json"
        codec => "json"
    }
}

```

---

<div class="post-metadata">

**Author:** ![panagiss](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@panagiss](https://discuss.elastic.co/u/panagiss)\
**Post date:** [August 12, 2021, 8:41pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/2 "2021-08-12T20:41:53Z")

</div>

What i want to achieve is to take specific fields from the beats input in order to check them later in the filter. So i don't want all the data from the input. Just specific JSON fields.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 12, 2021, 8:51pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/3 "2021-08-12T20:51:40Z")

</div>

> "Process containerd-shim (PID: 5305) by user root is RUNNING"

That is not JSON, so your json filter warns you about that. Not sure what else you would expect. It does not look like auditbeat data either.

---

<div class="post-metadata">

**Author:** ![panagiss](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@panagiss](https://discuss.elastic.co/u/panagiss)\
**Post date:** [August 12, 2021, 9:53pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/4 "2021-08-12T21:53:37Z")

</div>

It is Auditbeat and it IS JSON. But that JSON has also a message field I guess.  
Tomorrow I will post you what I get as pure input without the filter which causes the errors.

---

<div class="post-metadata">

**Author:** ![panagiss](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@panagiss](https://discuss.elastic.co/u/panagiss)\
**Post date:** [August 13, 2021, 6:40am UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/5 "2021-08-13T06:40:32Z")

</div>

So here is the raw input i receive from auditbeat

```auto
{
       "process" => {
        "working_directory" => "/home/user",
                      "pid" => 27377
    },
         "agent" => {
            "hostname" => "lent",
                "type" => "auditbeat",
             "version" => "7.13.2"
    },
     "@metadata" => {
          "input" => {
            "beats" => {
                "host" => {
                    "ip" => "20.86.153.197"
                }
            }
        },
           "type" => "_doc",
           "beat" => "auditbeat",
        "version" => "7.13.2"
    },
        "auditd" => {
              "result" => "success",
             "summary" => {
             "actor" => {
                "secondary" => "azureuser",
                  "primary" => "azureuser"
            },
            "object" => {
                   "type" => "process",
                "primary" => "systemctl stop auditbeat.service"
            }
        },
            "sequence" => 163,
                "data" => {
            "terminal" => "pts/0",
                 "cmd" => "systemctl stop auditbeat.service"
        },
             "session" => "1",
        "message_type" => "user_cmd"
    },
          "tags" => [
        [0] "beats_input_raw_event"
    ],
         "cloud" => {
        "instance" => {
            "name" => "lent",
 
        },
        "provider" => "azure",
         "machine" => {
            "type" => "Standard_B2s"
        },
         "service" => {
            "name" => "Virtual Machines"
        },
          "region" => "europe",
         "account" => {}
    },
    "@timestamp" => 2021-08-13T06:37:14.359Z,
           "ecs" => {
        "version" => "1.9.0"
    },
       "service" => {
        "type" => "auditd"
    },
      "@version" => "1",
          "host" => {
             "hostname" => "lent",
                   "os" => {
            
            "codename" => "bionic",
                "name" => "Ubuntu",
                "type" => "linux",
            "platform" => "ubuntu"
        },
        "containerized" => false,
                   "ip" => [
            [0] "10.0.1.5",
    
        ],
                 "name" => "lent",
                  "mac" => [
              [1] "ee:ee:ee:ee:ee:ee",
            
        ],
         "architecture" => "x86_64"
    },
         "event" => {
            "kind" => "event",
          "module" => "auditd",
          "action" => "ran-command",
            "type" => [
            [0] "start"
        ],
        "category" => [
            [0] "process"
        ],
         "outcome" => "success"
    },
          "user" => {
        "audit" => {
            "name" => "user",
              "id" => "100"
        },
         "name" => "user",
           "id" => "100"
    },
    "message": "Process sleep (PID: 26212) by user root STOPPED"
}

```

Look at the end of this JSON. There is a message field! Maybe that is causing the damage. Do you know how can i deal with it?

As a stated earlier what i want is just to get few specific fields from that Json an "build" my own event via my pipeline.

What should i do? What i had in mind is that when an event comes ALL this JSON data would be inside a MESSAGE filed. That's why in my conf in my Json Filter i do ` source => "message"`.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2021, 5:54pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/6 "2021-08-13T17:54:05Z")

</div>

You could use mutate+add\_field to copy all the fields you actually do want, then

```
prune { blacklist_names => ["agent", "auditd", "cloud", "ecs", "event", "host", "message", "process", "service", "user"] }

```

Or whitelist the fields you want to keep after adding them.

---

<div class="post-metadata">

**Author:** ![panagiss](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@panagiss](https://discuss.elastic.co/u/panagiss)\
**Post date:** [August 13, 2021, 6:36pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/7 "2021-08-13T18:36:02Z")

</div>

Ok, got it.  
Two questions then, so when reading from a file i get all the data into the "message" field while when i get input from beats, data comes straight without being into a "message" field right ?

Second, about the prune filter, when i whitelist a filed which has many "son" fields, i hope it keeps also the son fields right?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2021, 7:05pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/8 "2021-08-13T19:05:43Z")

</div>

Yes a file input puts the line from the file into the message field. beats data is usually much more structured. For other inputs it varies between the two extremes.

prune only operates on top level fields. If a top-level field is blacklisted anything inside it is also removed, if whitelisted the contents stay.

---

<div class="post-metadata">

**Author:** ![panagiss](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@panagiss](https://discuss.elastic.co/u/panagiss)\
**Post date:** [August 13, 2021, 7:10pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/9 "2021-08-13T19:10:40Z")

</div>

So the solution to my problem was that **BEATS** input plugin **DOES NOT** put the data into a "message" field as **FILE** input does.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 10, 2021, 7:11pm UTC](https://discuss.elastic.co/t/unexpected-error-with-beats-input-plugin/281255/10 "2021-09-10T19:11:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.