Logstash, http filter, problem with sending arrays in body

Elastic Stack Logstash
1

I am trying to use http filter plugin to make POST requests to third party API while processing evens in logstash.

This API requires requests to be json-encoded in following format:

{
    "type": "alert",
    "source": "elasticsearch",
    "title": "default title",
    "description": "default description",
    "sourceRef": "default_id",
    "artifacts": [
        {
            "data": "8.8.8.8",
            "message": "not empty",
            "dataType": "ip"
        },
        {
            "data": "11.11.11.11",
            "message": "not empty",
            "dataType": "ip"
        }
    ]
}

In my logstash config I use following filter configuration:

http {
  body_format => "json"
  follow_redirects => false
  body => {
    "title" => "%{[title]}"
    "description" => "%{[description]}"
    "type" => "%{[type]}"
    "source" => "%{[source]}"
    "sourceRef" => "%{[sourceRef]}"
    "artifacts" => "%{[array_of_objects]}"
  }
  url => "http://${HIVE_IP}/api/alert"
  verb => "POST"
  headers => [ "Authorization", "Bearer ${HIVE_API_KEY}" ]
  target_body => "[@metadata][hive_response]"
  target_headers => "[@metadata][hive_headers]" 
}

All required data is prepared in right format in previous sections of the filter.

The problem I am now fasing is that I do not understand, how to pass an array of objects to "artifacts" field of http body. I tried "%{[array_of_objects]}" and ["%{[array_of_objects]}"], but it seems that using %{[field]} is leading to content of the field becoming a string.

Here is what I got in response (using "artifacts" => ["%{[array_of_objects]}"] syntax):

[DEBUG] 2019-07-19 10:32:13.769 [[main]>worker0] http - processing request {:body=>" {\"title\":\"default title\",\"description\":\"default description\",\"type\":\"alert\",\"source\":\"elasticsearch\",\"sourceRef\":\"default_id\",\"artifacts\":[\"{message=not empty, data=8.8.8.8, dataType=ip},{message=not empty, data=11.11.11.11, dataType=ip}\"]}", :url=>"http://<<IP>>/api/alert", :headers=>{"Authorization"=>"Bearer <<KEY>>", "content-type"=>"application/json"}}
[ERROR] 2019-07-19 10:32:14.238 [[main]>worker0] http - error during HTTP request {:url=>"http://<<IP>>/api/alert", :code=>400, :response=>"{\"tableName\":\"alert\",\"type\":\"AttributeCheckingError\",\"errors\":[{\"name\":\"alert.artifacts\",\"format\":\"nested\",\"type\":\"InvalidFormatAttributeError\",\"message\":\"Invalid format for alert.artifacts: JsonInputValue(\\\"{message=not empty, data=8.8.8.8, dataType=ip},{message=not empty, data=11.11.11.11, dataType=ip}\\\"), expected nested\",\"value\":{\"type\":\"JsonInputValue\",\"value\":\"{message=not empty, data=8.8.8.8, dataType=ip},{message=not empty, data=11.11.11.11, dataType=ip}\"}}]}"}

Here is what I got in response (using "artifacts" => "%{[array_of_objects]}" syntax):

[DEBUG] 2019-07-19 10:38:09.743 [[main]>worker2] http - processing request {:body=>{\"title\":\"default title\",\"description\":\"default description\",\"type\":\"alert\",\"source\":\"elasticsearch\",\"sourceRef\":\"default_id\",\"artifacts\":\"{data=8.8.8.8, message=not empty, dataType=ip},{data=11.11.11.11, message=not empty, dataType=ip}\"}", :url=>"http://<<IP>>/api/alert", :headers=>{"Authorization"=>"Bearer <<KEY>>", "content-type"=>"application/json"}}
[ERROR] 2019-07-19 10:38:10.233 [[main]>worker2] http - error during HTTP request {:url=>"http://<<IP>>/api/alert", :code=>400, :response=>"{\"type\":\"ConflictError\",\"message\":\"[alert][2dda1ae3dc11f5810504ca683c915dbb]: version conflict, document already exists (current version [2])\",\"object\":{\"type\":\"alert\",\"customFields\":{},\"sourceRef\":\"default_id\",\"date\":1563521890175,\"lastSyncDate\":1563521890175,\"description\":\"default description\",\"follow\":true,\"createdBy\":\"siem\",\"artifacts\":[ ],\"status\":\"New\",\"_id\":\"2dda1ae3dc11f5810504ca683c915dbb\",\"source\":\"elasticsearch\",\"severity\":2,\"title\":\"default title\",\"tlp\":2,\"createdAt\":1563521890153}}"}

Any advice?

1 Like
2

Are you sure the second error is a problem with the body and its not just telling you that you are posting duplicate data?

There is nothing special about sprintf in the http filter, so you can use mutate+add_field to see what it does. If we have

   "anArray" => [
    [0] "a",
    [1] 1
],

Then

mutate { add_field => { "text1" => "%{anArray}" "text2" => "[%{anArray}]" } }

produces

     "text1" => "a,1",
     "text2" => "[a,1]",

So the second one looks like an array, but there are no quotes around the text string inside it. If that is a problem you will have to resort to ruby to create the text string that you are going to insert into the body. Remember, you are not sending an array in the body, you are sending a text representation of an array.

1 Like
3

Hello @Badger,

Thank you for the response.
Missing quotes aroung keys and its values is an exact problem I am fasing. I thought I would be able to put an exact array containing objects via standard syntax, but not an array with textual representation of my objects, and I am still wandering if it is possible without changing plugin code.

I have bulit a string representation of a payload outside of http filter as a workaround.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.