Messages escaped in Filebeat

Elastic Stack Beats
1

Hi,

I have an issue with Filebeat, I have escaping characters that I can't remove from the message value, or I don't know how. I know there already are some topics on that matter but none solved my issue, that's why I'm allowing myself to open this one. Please kindly tell me if there is already a solution I wouldn't have found.

I've configured a Filebeat to query a MISP instance (without using the MISP module) and to send the results to a Logstash instance in UDP.

Here is the Filebeat conf:

    filebeat.inputs:
    - type: httpjson
      url: "https://10.0.0.5/events/restSearch"
      http_method: "POST"
      json_objects_array: "response"
      http_headers:
              Authorization: "XXXXXXXXXXXXXXXX"
              Accept: "application/json"
              Content-Type: "application/json"
      interval: 15
      pagination.enabled: "true"
      pagination.extra_body_content:
              max_items: "1"
      ssl:
              verification_mode: "none"
              client_authentication: "none"
      # fields_under_root: true
      # json.keys_under_root: true

    output.logstash:
      hosts: ["10.0.0.1:5044"]

Here is the Logstash beat's input conf:

    input {
      udp {
        host => "10.0.0.2"
        port => 6044
        }
    }

    filter {
      json {
        source => "message"
      }

      prune {
        whitelist_names => ["message"]
      }

      mutate {
        gsub => ["message",'\"','"']
      }

    }

    output {
      #for debug purposes only
      stdout { codec => rubydebug }
    }

I've tried the use of fields_under_root and json.keys_under_root set to true but none of that helped. I've also tried to modify the message value on the second Logstash in the filter but it does not work either. In the end, I would like to extract the message value without the escaping.

Here is the output I get :

    {
              "host" => {
                       "os" => {
                "codename" => "bionic",
                  "kernel" => "4.15.0-55-generic",
                  "family" => "debian",
                 "version" => "18.04.4 LTS (Bionic Beaver)",
                    "name" => "Ubuntu",
                "platform" => "ubuntu"
            },
                 "hostname" => "misp",
            "containerized" => false,
                       "id" => "cd4d32abc1224be991e9394f8ddfeef6",
             "architecture" => "x86_64",
                     "name" => "misp"
        },
           "message" => "{\"Event\":{\"Attribute\":[{\"Galaxy\":[],\"ShadowAttribute\":[],\"category\":\"Artifacts dropped\",\"comment\":\"c'est un hash\",\"deleted\":false,\"disable_correlation\":false,\"distribution\":\"5\",\"event_id\":\"3\",\"first_seen\":null,\"id\":\"4\",\"last_seen\":null,\"object_id\":\"0\",\"object_relation\":null,\"sharing_group_id\":\"0\",\"timestamp\":\"1591806080\",\"to_ids\":true,\"type\":\"text\",\"uuid\":\"5ee10880-656c-44ce-8389-02eec0a83c0f\",\"value\":\"hashhashhash\"}],\"Galaxy\":[],\"Object\":[],\"Org\":{\"id\":\"1\",\"local\":true,\"name\":\"ORGNAME\",\"uuid\":\"5ec6f868-6020-45b5-b9d5-503a84db2e2f\"},\"Orgc\":{\"id\":\"1\",\"local\":true,\"name\":\"ORGNAME\",\"uuid\":\"5ec6f868-6020-45b5-b9d5-503a84db2e2f\"},\"RelatedEvent\":[],\"ShadowAttribute\":[],\"analysis\":\"0\",\"attribute_count\":\"1\",\"date\":\"2020-06-10\",\"disable_correlation\":false,\"distribution\":\"1\",\"extends_uuid\":\"\",\"id\":\"3\",\"info\":\"Malware\",\"locked\":false,\"org_id\":\"1\",\"orgc_id\":\"1\",\"proposal_email_lock\":false,\"publish_timestamp\":\"0\",\"published\":false,\"sharing_group_id\":\"0\",\"threat_level_id\":\"1\",\"timestamp\":\"1591806080\",\"uuid\":\"5ee10817-5a74-4d8a-8c58-02e2c0a83c0f\"}}",
               "ecs" => {
            "version" => "1.4.0"
        },
        "@timestamp" => 2020-06-15T11:12:17.223Z,
          "@version" => "1",
             "input" => {
            "type" => "httpjson"
        },
             "agent" => {
                      "id" => "8268922d-2b13-40fe-94a0-026aa9faa72d",
                "hostname" => "misp",
            "ephemeral_id" => "9cad9c43-7973-4caa-a2e9-0ba2169df152",
                 "version" => "7.6.2",
                    "type" => "filebeat"
        },
             "event" => {
            "created" => "2020-06-15T11:12:17.223Z"
        },
              "tags" => [
            [0] "beats_input_codec_plain_applied"
        ]
    }

Would anyone have any idea of what could cause this and what could help remediate it?

Thank you in advance for your help.

Edit: I have clean my configuration.

2

Hey @JeanN,

I think that processing MISP can be complicated. Is there any reason why you are not using the MISP module?

Most modules use ingest pipelines, that can be tricky to use with Logstash, but in the case of the MISP module processing is done client side, in filebeat, using these processors, and this javascript pipeline.

3

Hello @jsoriano ,

I simplified my configuration.

I don't want to use the MISP module, and I would like to keep the original raw message, so that I can send it later.

As in this example:

MISP with filebeat -> logstash -> MISP

4

fields_under_root only applies to the custom fields added with fields. json.keys_under_root is not supported by the httpjson input.

Something you can try is to decode json already in filebeat by using the decode_json_fields processor, as in the example in the httpjson input docs: HTTP JSON input | Filebeat Reference [7.7] | Elastic
To merge the decoded fields at the root level, you need to use target: "" in decode_json_fields.

5

Thanks for your help. You solved my problem.

6

Hi Jean,

Can you share your final config file of filebeat.yml to get MISP events? I don't want to use the MISP module too, is not working right for me.

Thanks in advance.

Regards,
Alejandro

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.