Threatintel anomali error: the requested root field is empty

Elastic Stack Beats
1

Hi,

I gave the threatintel module a try. Successfully ingesting abuseurl, abusemalware and otx. I've a problem with getting anomali to work.

First I figured, I have to uncomment the username/password fields to send default guest credentials.

But then I get:

Apr 21 12:14:47 ingester filebeat[29141]: 2021-04-21T12:14:47.524Z#011INFO#011[input.httpjson-cursor]#011v2/input.go:145#011Input stopped because context
 was cancelled with: context canceled#011{"input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects", "input_url": "https://li
mo.anomali.com/api/v1/taxii2/feeds/collections/41/objects"}
Apr 21 12:14:50 ingester filebeat[3205]: 2021-04-21T12:14:50.228Z#011INFO#011[input.httpjson-cursor]#011v2/input.go:126#011Process another repeated reque
st.#011{"input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects", "input_url": "https://limo.anomali.com/api/v1/taxii2/feeds
/collections/41/objects"}
Apr 21 12:14:50 ingester filebeat[3205]: 2021-04-21T12:14:50.938Z#011ERROR#011[input.httpjson-cursor]#011v2/request.go:186#011error processing response:
the requested root field is empty#011{"input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects", "input_url": "https://limo.a
nomali.com/api/v1/taxii2/feeds/collections/41/objects"}
Apr 21 12:14:50 ingester filebeat[3205]: 2021-04-21T12:14:50.938Z#011INFO#011[input.httpjson-cursor]#011v2/request.go:209#011request finished: 0 events p
ublished#011{"input_source": "https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects", "input_url": "https://limo.anomali.com/api/v1/taxii2/
feeds/collections/41/objects"}

This is my threatinel module anomali section:

anomali:
    enabled: true
    var.input: httpjson
    var.url: https://limo.anomali.com/api/v1/taxii2/feeds/collections/41/objects
    var.username: guest
    var.password: guest
    var.first_interval: 400h
    var.interval: 5m

I also tried a few other collections, but with no difference. Anything I'm doing wrong here?

Sebastian

2

I am having the same problem when trying to configure filebeat to work with my MISP instance. However, I am using an API key to access my site.

EDIT:
-Apologies, I was using the MISP.yml instead of threatintel...when I moved over to threatintel, ti worked correctly for me.

3

Hi,

I'm getting the same error while trying to use the misp threat intel module:
[input.httpjson-cursor] v2/request.go:186 error processing response: the requested root field is empty {"input_source": "https://X.X.X.X/events/restSearch/", "input_url": "https://X.X.X.X/events/restSearch/"}

This is the config in filebeat.yml:

    - module: threatintel
  abuseurl.enabled: false
  abusemalware.enabled: false
  malwarebazaar.enabled: false
  anomali.enabled: false
  otx.enabled: false
  misp:
    enabled: true
    var.input: httpjson
    var.url: "https://X.X.X.X/events/restSearch"
    var.api_token: APIKEY
    var.ssl.verification_mode: none
    var.filters:
      - type: ["md5", "sha256", "url", "ip-src", "ip-dst", "domain"]
    var.first_interval: 72h
    var.interval: 60m

Any idea what causes it?

4

The only differences I can see between my threatintel.yml and yours is that I don't have quotes around my var.url entry and I'm not using any filters.
That is assuming that you have an actual API token in your yml file instead of the placeholder APIKEY.

5

Do you receive any errors when turning on filters? In my case, the service doesn't start when I use, for example:

var.filters:
      - type: ["domain"]
6

The mystery solved for me. I usually point all my beats to logstash for enrichment of the data.
So, all the json.id field of the anomali clashed with the json.id field that already was due to other
data sources in the index, so ES rejected it because of string vs. long type.
It took me a while, I figured it out the time I updated the filebeat to 7.12.1. That was the first filebeat that updated, and so the anomali data easily went in, because the json.id field type now was automatically set to a string.

Note: I still see these messages in the filebeat logs, but I was on a totally wrong path, they were not the reason the data didn't properly got ingested.

7

Hello all,

I know that this error has been causing some confusion, the issue is that the message is not actually an error, it just means that either there was no events to retrieve, or it has paginated over multiple pages of events, and reached an empty page.

In 7.13 this "error" has been moved to the debugging level to stop causing the confusions.

While you might not be getting any events in, the error message itself is not the cause, I would start with setting the "first_interval" higher, in some cases there might not be any new events in the threat intel feed, for the 72h you have configured, so it will just return that error instead.

Hope this helps @buzzdeee @AndreiRD @tcaudill

8

Marius, I think you are right. Thanks for this!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.