Previously had the MISP module working although not used frequently. Recently I've noticed that it seems to be broken, I've updated to Filebeat (7.7) recently.
The logs say "key not found".
Filebeat logs:
May 22 10:18:15 yyyy filebeat[15321]: 2020-05-22T10:18:15.913+0100#011INFO#011[esclientleg]#011eslegclient/connection.go:263#011Attempting to connect to Elasticsearch version 7.7.0
May 22 10:18:15 yyyy filebeat[15321]: 2020-05-22T10:18:15.981+0100#011INFO#011input/input.go:114#011Starting input of type: httpjson; ID: 4285923797870508923
May 22 10:18:15 yyyy filebeat[15321]: 2020-05-22T10:18:15.982+0100#011INFO#011[httpjson]#011httpjson/input.go:118#011httpjson input worker has started.#011{"url": "http://x.x.x.x/attributes/restSearch"}
May 22 10:18:16 yyyy filebeat[15321]: 2020-05-22T10:18:16.077+0100#011ERROR#011[httpjson]#011httpjson/input.go:123#011key not found#011{"url": "http://x.x.x.x/attributes/restSearch"}
May 22 10:18:16 yyyy filebeat[15321]: 2020-05-22T10:18:16.078+0100#011INFO#011[httpjson]#011httpjson/input.go:124#011httpjson input worker has stopped.#011{"url": "http://x.x.x.x/attributes/restSearch"}
Misp module config:
- module: misp
threat:
enabled: true
var.api_key: "xxxxxx"
var.json_objects_array: "response.Attribute"
var.url: "http://x.x.x.x/attributes/restSearch"
I've tested using a rest client and can verify the API key and endpoint work and return the expected json object.
I've also done pcap to check what is sent and returned - this confirms that it looks like it should work:
y.y.y.y.54416 > x.x.x.x.http: Flags [P.], cksum 0xa298 (incorrect -> 0x93af), seq 1:340, ack 1, win 502, options [nop,nop,TS val 2066588609 ecr 3090069452], length 339: HTTP, length: 339
GET /attributes/restSearch HTTP/1.1
Host: x.x.x.x
User-Agent: Elastic Filebeat/7.7.0 (linux; amd64; 5e69e25b920e3d93bec76a09a31da3ab35a55607; 2020-05-12 00:53:16 +0000 UTC)
Accept: application/json
Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
Accept-Encoding: gzip
Connection: close
x.x.x.x.http > y.y.y.y.54416: Flags [.], cksum 0xfcfc (correct), seq 1:1369, ack 340, win 219, options [nop,nop,TS val 3090069496 ecr 2066588609], length 1368: HTTP, length: 1368
HTTP/1.1 200 OK
Date: Fri, 22 May 2020 09:18:15 GMT
Server: Apache/2.4.29 (Ubuntu)
Set-Cookie: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; expires=Fri, 22-May-2020 19:18:15 GMT; Max-Age=36000; path=/; HttpOnly
Set-Cookie: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; expires=Fri, 22-May-2020 19:18:16 GMT; Max-Age=36000; path=/; HttpOnly
X-Result-Count: 1059
X-Export-Module-Used: json
X-Response-Format: json
Content-Length: 563677
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Connection: close
Content-Type: application/json; charset=UTF-8
{"response": {"Attribute": [{"id":"1025","event_id":"2","object_id":"0","object_relation":null,"category":"Network activity","type":"ip-dst","to_ids":false,"uuid":"5dfa55f6-e954-454e-8479-552fe80e45af","timestamp":"1576687094","distribution":"0","sharing_group_id":"0","comment":"","deleted":false,"disable_correlation":false,"first_seen":null,"last_seen":null,"value":"164.52.24.164","Event":{"org_id":"2","distribution":"0","id":"2","info":"blockrules of rules.emergingthreats.net feed","orgc_id":"2","uuid":"5d42d276-aed4-4a4f-b87f-3504e80e45af"}},{"id":"1130","event_id":"2","object_id":"0","object_relation":null,"category":"Network activity","type":"ip-dst","to_ids":false,"uuid":"5dfa55f7-28ec-4b45-8f11-552fe80e45af","timestamp":"1576687095","[!http]
Anyone else seeing this? any ideas how to fix?