HTTP JSON input: one array one document

Elastic Stack Beats
1

Hello,

I am using the HTTP JSON input of Filebeat to query some APIs and get some metrics. The problem is for some of them to send a JSON array instead of a JSON object. If I want to make a count of a specific metric, I can't sum the values for each object in the array. I went through all the above documentation, but failed to find a solution to my problem.

Has anyone found a workaround?

Thanks!

2

Can u clarify what you're trying to do with the array of data?

3

Sure!

For example, if the API request returns this type of data :

[{"username": "foo",
  "messages": "x"},
 {"username": "bar",
  "messages": "y"}]

I would like to sum the messages values to get the total number of messages (x+y).

From what I've experimented, this kind of response is processed in two times :

  1. First with foo
  2. Then with bar

I can use response.transforms but only in the scope of one JSON object. So I'm stuck.

4

Can you post the config for your httpjson input?

5

Hm no I can't because I didn't find any that could work. But I can try to give a more meaningful example.

For example for collecting emails count on a daily basis for Mailcow, I could use this endpoint. But the output will be an array with all the mailboxes details (below is only one mailbox as the API example) :

[
  {
    "max_new_quota": 10737418240,
    "username": "info@doman3.tld",
    "rl": false,
    "is_relayed": 0,
    "name": "Full name",
    "active": "1",
    "domain": "doman3.tld",
    "local_part": "info",
    "quota": 3221225472,
    "attributes": {
      "force_pw_update": "0",
      "tls_enforce_in": "0",
      "tls_enforce_out": "0",
      "sogo_access": "1",
      "mailbox_format": "maildir:",
      "quarantine_notification": "never"
    },
    "quota_used": 0,
    "percent_in_use": 0,
    "messages": 0,          <-- I want to sum this field for all mailboxes
    "spam_aliases": 0,
    "percent_class": "success"
  }
]

Actually I realize there are 2 questions in my post:

  1. How to have access to all the mailboxes details within one response and be able to process this response with processors;
  2. With making use of processors, how could I end up with the sum of all messages count?

Maybe my case is too specific, I can keep searching, but the first step is to know if it is possible to process all json documents within one response?

6

to split the response use HTTP JSON input | Filebeat Reference [7.13] | Elastic. The MISP module does, beats/config.yml at master · elastic/beats · GitHub, to split the API response.

response.split:
  target: body.response
  split:
    target: body.Event.Attribute
    keep_parent: true

You'd probably just do

response.split:
  target: body.response
  split:
    target: body
    keep_parent: true
7

I thought I've tried to split the response (in a lot of different ways) but this operation only applied to the JSON objects and not to the whole array. Anyway, I will give it another try with what you provided, but in a few days (maybe weeks...) unfortunately. I will keep you posted. Thank you

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.