Filebeat MISP module error

Previously had the MISP module working although not used frequently. Recently I've noticed that it seems to be broken, I've updated to Filebeat (7.7) recently.

The logs say "key not found".

Filebeat logs:

    May 22 10:18:15 yyyy filebeat[15321]: 2020-05-22T10:18:15.913+0100#011INFO#011[esclientleg]#011eslegclient/connection.go:263#011Attempting to connect to Elasticsearch version 7.7.0
    May 22 10:18:15 yyyy filebeat[15321]: 2020-05-22T10:18:15.981+0100#011INFO#011input/input.go:114#011Starting input of type: httpjson; ID: 4285923797870508923
    May 22 10:18:15 yyyy filebeat[15321]: 2020-05-22T10:18:15.982+0100#011INFO#011[httpjson]#011httpjson/input.go:118#011httpjson input worker has started.#011{"url": "http://x.x.x.x/attributes/restSearch"}
    May 22 10:18:16 yyyy filebeat[15321]: 2020-05-22T10:18:16.077+0100#011ERROR#011[httpjson]#011httpjson/input.go:123#011key not found#011{"url": "http://x.x.x.x/attributes/restSearch"}
    May 22 10:18:16 yyyy filebeat[15321]: 2020-05-22T10:18:16.078+0100#011INFO#011[httpjson]#011httpjson/input.go:124#011httpjson input worker has stopped.#011{"url": "http://x.x.x.x/attributes/restSearch"}

Misp module config:

    - module: misp
      threat:
        enabled: true
        var.api_key: "xxxxxx"
        var.json_objects_array: "response.Attribute"
        var.url: "http://x.x.x.x/attributes/restSearch"

I've tested using a rest client and can verify the API key and endpoint work and return the expected json object.

I've also done pcap to check what is sent and returned - this confirms that it looks like it should work:

    y.y.y.y.54416 > x.x.x.x.http: Flags [P.], cksum 0xa298 (incorrect -> 0x93af), seq 1:340, ack 1, win 502, options [nop,nop,TS val 2066588609 ecr 3090069452], length 339: HTTP, length: 339
        GET /attributes/restSearch HTTP/1.1
        Host: x.x.x.x
        User-Agent: Elastic Filebeat/7.7.0 (linux; amd64; 5e69e25b920e3d93bec76a09a31da3ab35a55607; 2020-05-12 00:53:16 +0000 UTC)
        Accept: application/json
        Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        Content-Type: application/json
        Accept-Encoding: gzip
        Connection: close

    x.x.x.x.http > y.y.y.y.54416: Flags [.], cksum 0xfcfc (correct), seq 1:1369, ack 340, win 219, options [nop,nop,TS val 3090069496 ecr 2066588609], length 1368: HTTP, length: 1368
        HTTP/1.1 200 OK
        Date: Fri, 22 May 2020 09:18:15 GMT
        Server: Apache/2.4.29 (Ubuntu)
        Set-Cookie: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; expires=Fri, 22-May-2020 19:18:15 GMT; Max-Age=36000; path=/; HttpOnly
        Set-Cookie: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; expires=Fri, 22-May-2020 19:18:16 GMT; Max-Age=36000; path=/; HttpOnly
        X-Result-Count: 1059
        X-Export-Module-Used: json
        X-Response-Format: json
        Content-Length: 563677
        X-Content-Type-Options: nosniff
        X-Frame-Options: DENY
        Connection: close
        Content-Type: application/json; charset=UTF-8

        {"response": {"Attribute": [{"id":"1025","event_id":"2","object_id":"0","object_relation":null,"category":"Network activity","type":"ip-dst","to_ids":false,"uuid":"5dfa55f6-e954-454e-8479-552fe80e45af","timestamp":"1576687094","distribution":"0","sharing_group_id":"0","comment":"","deleted":false,"disable_correlation":false,"first_seen":null,"last_seen":null,"value":"164.52.24.164","Event":{"org_id":"2","distribution":"0","id":"2","info":"blockrules of rules.emergingthreats.net feed","orgc_id":"2","uuid":"5d42d276-aed4-4a4f-b87f-3504e80e45af"}},{"id":"1130","event_id":"2","object_id":"0","object_relation":null,"category":"Network activity","type":"ip-dst","to_ids":false,"uuid":"5dfa55f7-28ec-4b45-8f11-552fe80e45af","timestamp":"1576687095","[!http]

Anyone else seeing this? any ideas how to fix?

Just to note, I've tested the same config on Filebeat 7.6.0 and it works, so seems like something broke it in 7.7.0.

Hi Dan,

as MISP module is relying on HTTPJSON input, there is an open issue here and it looks like json_objects_array is the issue here: