Sending multiple events using a single POST with http output codec

Elastic Stack Logstash
1

Good Day,
How do you send multiple events in a single POST using the http output codec? I can send the below events but they are sent as 2 individual POSTS and I need to send them as a single post due to restrictions on the receiving servers API.
My existing filter parses the data I want out of a large JSON file as per below and outputs to stdout. I have tried setting the filed format => "json_batch" but I end up with a "Server 500 error response."

{
         "sha1" => "410e66f0dc83f98f2ab600ce053cb8240e817efd",
       "event_data" => "748b6d47c5aa6ee1b4d2485a0ef3b6ae, 410e66f0dc83f98f2ab600ce053cb8240e817efd",
       "@timestamp" => 2024-08-31T12:04:33.437Z,
        "Published" => "2024-08-27T07:09:26Z",
          "md5" => "748b6d47c5aa6ee1b4d2485a0ef3b6ae"


}
{
         "sha1" => "dea0e8e92454c0cf77477b4620b1c3136276bac7",
       "event_data" => "2760c8bc1d1ba3a14bbaa919d1c84a98, dea0e8e92454a0cf77477b4620a1c3136276bac7",
       "@timestamp" => 2024-08-31T12:04:33.437Z,
        "Published" => "2024-08-18T17:53:43Z",
          "md5" => "2760c8bc1e1ba3a14bdaa919d1c84a98"

}

My filter is as follows:

input {
        file {
        path => "/usr/share/logstash/bin/example.json"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        codec => "json"
        }
}

filter {

  split {
    field => "[alerts]"
  }

  mutate {
    rename => { "[alerts][doc][timestamp]" => "Published" }
    rename => { "[alerts][doc][source_file][hashes][sha1]" => "sha1" }
    rename => { "[alerts][doc][source_file][hashes][md5]" => "md5" }
  }

  mutate {
    remove_field => [
      "[alerts]",
      "[port]",
      "[path]",
      "[host]",
      "[@version]"
    ]
  }

mutate {
  add_field => {
        "event_data" => "%{md5}, %{sha1}"
       }
}

}

output {
  stdout {
    codec => rubydebug
  }

http {
        url => "https://api.example.com/api/v2/example/import/"
        http_method => "post"
        headers => {
                "authorization" => "apikey <api-username>:<api-key>"
                   }
        format => "json"
        mapping => {
                "confidence" => "10"
                "classification" => "example"
                "type" => "type_example"
                "datatext" => "%{event_data}"
                    }
     }
}

How do I get 2 or more events to be sent as a single HTTP POST? instead of each event being sent as multiple HTTP POSTS? We could have 2 events or we could have 60 events.

2

You use format => json_batch, but your HTTP server may not like the format it receives.

When a batch of events gets through the pipeline filters and reaches the output it is passed to the multi_receive method of the output as an array of events. The http output then either sends that entire array as a single HTTP call (json_batch), or iterates over it and sends each array entry as a separate HTTP call.

If you have a couple of events like

{
    "keepMe" => 0,
"@timestamp" => 2024-08-31T20:47:16.102845153Z,
  "@version" => "1"
}
{
    "keepMe" => 1,
"@timestamp" => 2024-08-31T20:47:16.112518840Z,
  "@version" => "1"
}

and use a logstash http output to send it to a logstash http input then the message received by the input will be a JSON array

"message"=>"[{\"keepMe\":0,\"@timestamp\":\"2024-08-31T20:47:16.102845153Z\",\"@version\":\"1\"},{\"keepMe\":1,\"@timestamp\":\"2024-08-31T20:47:16.112518840Z\",\"@version\":\"1\"}]"

If you changed the format to json instead then the received message would be

"message"=>"{\"keepMe\":0,\"@timestamp\":\"2024-08-31T20:51:19.006632423Z\",\"@version\":\"1\"}"
4

Thanks.
I have tried format => json and it gets the events to the end system successfully but it sends each event in separate POST. So if we have 100 events that's going to be 100 separate http POST's that's going to max out the number of API calls we can make to the system that's going to receive the events. We can't change this as we don't own this system. The issue as well is that the system receiving the events has to have specific mappings. So from you example the "keepMe" field would have to be present for the API on the receiving system to work. I need some way to basically combine "keepMe" => 0, "keepMe" => 1, "keepMe" => 2, etc" in a single field, along with ""alpha" => 0, "alpha" => 1, "alpha" => 2.
In my original example it would be multiple sha1 and md5 data....if that makes sense?

5

For the two events you showed in your first post can you show what JSON should be POSTed with them combined? I can generalize from two events upwards :smiley:

7

Sure thing. So it would be the SHA1 and MD5 Fields combined in a field called "event_data"....Now there could be more then just 2 of the sha1's and 2 of the md5's that need to be sent in a single post. It could be upwards of like 30 of these fields

"sha1" => "410e66f0dc83f98f2ab600ce053cb8240e817efd",
"sha1" => "dea0e8e92454c0cf77477b4620b1c3136276bac7",
"md5" => "748b6d47c5aa6ee1b4d2485a0ef3b6ae",
"md5" => "2760c8bc1e1ba3a14bdaa919d1c84a98"

They have to be written to a filed called "datatext" otherwise the system receiving the events wont accept them as it expects to have the events in the "datatext" field. So that's why I was mapping them with "datatext" => '%{event_data}"

mutate {
  add_field => {
        "event_data" => "%{md5}, %{sha1}"
       }
}

mapping => {
                "confidence" => "10"
                "classification" => "example"
                "threat_type" => "malware"
                "datatext" => "%{event_data}**"
                    }

Or were you asking for the original JSON from the log file?

8

I am asking about the JSON you want to send to the HTTP server. With this

input { generator { count => 1 lines => [
    '{ "keepMe": 0, "sha1": "410e66f0", "md5": "748b6d47" }',
    '{ "keepMe": 1, "sha1": "dc83f98f", "md5": "c5aa6ee1" }'
] codec => json } }
filter {
    ...
    mutate { add_field => { "event_data" => "%{md5}, %{sha1}" } }
}
output {
    http {
        url => "http://127.43.9.201:22334/"
        format => json
        http_method => "post"
        mapping => { "type" => "type_example" "datatext" => "%{event_data}" }
    }
}

we will get two POSTs, with the JSON content being

{"datatext":"748b6d47, 410e66f0","type":"type_example"}

{"datatext":"c5aa6ee1, dc83f98f","type":"type_example"}

To combine them into one POST we could send an array of hashes, but that is exactly what json_batch does, and it sounds like that does not work for you.

We cannot just concatente them because that is not valid JSON

{"datatext":"748b6d47, 410e66f0","type":"type_example", "datatext":"c5aa6ee1, dc83f98f","type":"type_example"}

We could make datatext and type arrays

{"datatext": ["748b6d47, 410e66f0","c5aa6ee1, dc83f98f"],"type":"type_example","type_example"}

and there are probably other possibilities. I would like to know exactly what JSON you want sent to the HTTP server for those two events.

9

from your above example I would want to send sha1 fields and md5 fields I don't need to send the "keepMe" data.

"748b6d47, 410e66f0","c5aa6ee1, dc83f98f"
10

I get that. And my previous post shows that if you use the mapping option you can do that (as you were trying to do). You still have not shown me the exact form of JSON you want used to send two events in a single POST.

11

Hi,
The JSON is as follows and I am looking to extract md5 and sha1 data. The thing to keep in mind is its not always going to be 2 events....some times it could be 2 or it could be 10 or it could be 30+. It will vary.

{"alerts":[{"id":"cr71h0nr4h0u57cn1nj0","monitor_id":"cpvq246rctin8luth120","doc":{"__derived_from":"9ebe4dc8-e895-4f23-950e-fcd6e8063719","__id":"de0e17f4-d113-4d39-bdc7-bf03f6344e43","__type":"account_discovery","ingested":"2024-08-27T18:17:53Z","test_account":{"login":"lmprashid","mean":{"plain_text":"********"},"profile":{"identity":{"name":"lmprashid"}},"test":{"inet_location":{"domain":"identity.auth.abc.com","path":"/login","protocol":"https","url":"https://identity.auth.abc.com/login"},"name":"identity.auth.abc.com"}},"source":"ccmp","source_file":{"filename":"CA_V92sc2f-2108mcH-HN2_startup_27-08-2024 07-09-26","hashes":{"md5":"748b6d47c5aa6ee1b4d2485a0ef3b9cf","sha1":"410e44f0dc83f98f2ab600ce053cb8240e817efd","sha256":"37240d4f11dc92143540602a11283d399ac1a6793ccad3cdba3c96378b09f319"},"size":2013938},"source_threat":{"type":"malware"},"source_url":"https://api.test.org/bot6735509373/21","timestamp":"2024-08-27T07:09:26Z"},"labels":[{"id":"372a4654-798f-4024-8fa1-3047605b783a","classifier":"soup-threat-label","version":"0.1","label":"information-security/information-leak/Details","confidence":100,"type":"threat"}],"topics":[{"id":"b6d95720-084b-4637-98ac-33a694d33da1","type":"domain","value":"identity.auth.abc.com","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.test.inet_location.url","offsets":[8,29]},{"element_path":"test_account.test.inet_location.domain","offsets":[0,21]}]},{"id":"2241b521-728a-49c9-a27e-5465359441da","type":"domain","value":"api.test.org","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_url","offsets":[8,24]}]},{"id":"5acdfeec-5431-4e1e-bd39-4bfc3cc1e73d","type":"sha1_hash","value":"410e44f0dc83f98f2ab600ce053cb8240e817efd","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_file.hashes.sha1","offsets":[0,40]}]},{"id":"e718f401-da26-4579-a294-c13b59453456","type":"filename","value":"DD_V92sc2f-2108mcH-HN2_startup_27-08-2024 07-09-26","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_file.filename","offsets":[0,50]}]},{"id":"880ab40b-a8bd-4ea0-a777-f286af21fc64","type":"path","value":"/login","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.test.inet_location.path","offsets":[0,6]}]},{"id":"fa330d82-83cf-4597-a266-1d1a2592d786","type":"url","value":"https://identity.auth.abc.com/login","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.test.inet_location.url","offsets":[0,35]}]},{"id":"de094b61-062b-4c5c-818a-d97315306b3c","type":"url","value":"https://api.test.org/bot6735509373/21","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_url","offsets":[0,41]}]},{"id":"a95c94f9-fcdb-4313-aa4e-3bf273e34008","type":"test_name","value":"identity.auth.abc.com","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.test.name","offsets":[0,21]}]},{"id":"0a8361e8-2912-49af-aaed-3a389bfc7019","type":"identity_name","value":"lmprashid","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.profile.identity.name","offsets":[0,10]}]},{"id":"57de115c-825e-4081-8fa6-ee2313994359","type":"sha256_hash","value":"37240d4f11dc92143540602a11283d399ac1a6793ccad3cdba3c96378b09f319","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_file.hashes.sha256","offsets":[0,64]}]},{"id":"ec6d994b-001f-4f50-ae0c-d2525a4513d3","type":"md5_hash","value":"748b6d47c5aa6ee1b4d2485a0ef3b9cf","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_file.hashes.md5","offsets":[0,32]}]}],"topic_matches":[{"topic_id":"doc_type:account_discovery","value":"account_discovery"}],"label_matches":[],"doc_matches":[{"match_path":"test_account.test.inet_location.domain","locations":[{"offsets":[0,21],"value":"identity.auth.abc.com"}]}],"tags":[],"created_at":"2024-08-27T18:18:42.83Z","updated_at":"2024-08-29T15:00:06.366Z","labels_url":"https://api.information.beta.com/v4/xyz/docs/account_discovery/de0e17f4-d113-4d39-bdc7-bf03f6344e43/labels","topics_url":"https://api.information.beta.com/v4/xyz/docs/account_discovery/de0e17f4-d113-4d39-bdc7-bf03f6344e43/topics","doc_url":"https://api.information.beta.com/v4/xyz/docs/account_discovery/de0e17f4-d113-4d39-bdc7-bf03f6344e43","status":"closed","alert_type":"Compromised Details","alert_summary":"ccmp","title":"Leaked Web test Details from \"abc.com\"","email_sent_at":"","indicator_mscore":0,"severity":"low","confidence":0.27283120687581575,"aggregated_under_id":"cqn0gb9dhn4llplv7a00","has_analysis":false,"meets_mean_policy":"policy_unset","monitor_version":1},{"id":"cr53opcbpl0lvivgq8qg","monitor_id":"cpvq246rctin8luth120","doc":{"__id":"112e3ac5-0999-49d6-8e44-1c8a17e9312f","__type":"account_discovery","ingested":"2024-08-24T19:57:35Z","test_account":{"login":"nija2001","mean":{"plain_text":"********"},"profile":{"identity":{"name":"nija2001"}},"test":{"inet_location":{"domain":"www.abconline.com","path":"/abc/login.aspx","protocol":"https","url":"https://www.abconline.com/abc/login.aspx"},"name":"www.abconline.com"}},"source":"ccmp","source_file":{"filename":"@turbo 7M Lines Part 35.txt","hashes":{"md5":"2760c8bc1d1ba3a14bbaa919d1c84a98","sha1":"dea0e8e92454a0cf77477b4620a1c3136276bac7","sha256":"79b33a26665114c25f998f6933abd98db06a535a96c6ee46a5fe8c10c69d8713"},"size":568598195},"source_url":"https://t.me/turbo","timestamp":"2024-08-18T17:53:43Z"},"labels":[{"id":"96b33901-4c8b-4c95-a6ed-5a34a6311b4a","classifier":"soup-threat-label","version":"0.1","label":"information-security/information-leak/Details","confidence":100,"type":"threat"}],"topics":[{"id":"2b696886-ccb4-4779-87a9-87b611a74bb3","type":"md5_hash","value":"2760c8bc1d1ba3a14bbaa919d1c84a98","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_file.hashes.md5","offsets":[0,32]}]},{"id":"f8cf1201-2987-471f-a452-dc48fd120833","type":"sha1_hash","value":"dea0e8e92454a0cf77477b4620a1c3136276bac7","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_file.hashes.sha1","offsets":[0,40]}]},{"id":"58175039-e25a-43e1-a6c7-b9a61b22a9f1","type":"filename","value":"@beta 7M Lines Part 35.txt","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_file.filename","offsets":[0,30]}]},{"id":"6ac763d4-0fd6-4867-a84e-271310f3ac8d","type":"path","value":"/abc/login.aspx","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.test.inet_location.path","offsets":[0,15]}]},{"id":"6c0979e6-3e90-4575-a369-f70dad6cc056","type":"url","value":"https://t.me/mooncombo","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_url","offsets":[0,22]}]},{"id":"10e8b0aa-16b9-4e40-901e-812651b6ab11","type":"url","value":"https://www.abconline.com/abc/login.aspx","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.test.inet_location.url","offsets":[0,40]}]},{"id":"161e276e-8bcd-4499-a8e7-c59497d80510","type":"domain","value":"t.me","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_url","offsets":[8,12]}]},{"id":"cc989a44-a0ae-417d-abb1-6d7c051d84f8","type":"domain","value":"www.abconline.com","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.test.inet_location.domain","offsets":[0,17]},{"element_path":"test_account.test.inet_location.url","offsets":[8,25]}]},{"id":"3c8561b8-cf49-4bfe-bc2b-4cad1c466b2e","type":"identity_name","value":"nija2001","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.profile.identity.name","offsets":[0,13]}]},{"id":"ed4fea43-fd3d-479b-9a6e-8a30e309fbb3","type":"test_name","value":"www.abconline.com","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"test_account.test.name","offsets":[0,17]}]},{"id":"4d3f3059-142f-43c2-a9e3-b19a9482ab5b","type":"sha256_hash","value":"79b33a26665114c25f998f6933abd98db06a535a96c6ee46a5fe8c10c69d8713","extractor":"xyz-ma","extractor_version":"1.0.143","entity_locations":[{"element_path":"source_file.hashes.sha256","offsets":[0,64]}]}],"topic_matches":[{"topic_id":"doc_type:account_discovery","value":"account_discovery"}],"label_matches":[],"doc_matches":[{"match_path":"test_account.test.inet_location.domain","locations":[{"offsets":[0,17],"value":"www.abconline.com"}]}],"tags":[],"created_at":"2024-08-24T20:02:45.715Z","updated_at":"2024-08-26T17:26:39.45Z","labels_url":"https://api.information.alpha.com/v4/xyz/docs/account_discovery/112e3ac5-0999-49d6-8e44-1c8a17e9312f/labels","topics_url":"https://api.information.beta.com/v4/xyz/docs/account_discovery/112e3ac5-0999-49d6-8e44-1c8a17e9312f/topics","doc_url":"https://api.information.beta.com/v4/xyz/docs/account_discovery/112e3ac5-0999-49d6-8e44-1c8a17e9312f","status":"closed","alert_type":"Test Details","alert_summary":"ccmp","title":"Leaked Web test Details from \"abconline.com\"","email_sent_at":"","indicator_mscore":0,"severity":"low","confidence":0.27283120687581575,"aggregated_under_id":"cqn0gb9dhn4llplv7a00","has_analysis":false,"meets_mean_policy":"policy_unset","monitor_version":1}]}
12

I am confused. That JSON does not contain "datatext", which is the name you are using in the mapping option of the http output. Is that really the JSON you want the http output to send?

14

I had to add it and then assign sha1 and md5 to it otherwise it fails during start up when I run debug.

[DEBUG] 2024-09-03 08:57:16.416 [[main]>worker0] headers - http-outgoing-0 << HTTP/1.1 400 Bad Request
[DEBUG] 2024-09-03 08:57:16.416 [[main]>worker0] headers - http-outgoing-0 << Content-Type: application/json
[DEBUG] 2024-09-03 08:57:16.416 [[main]>worker0] headers - http-outgoing-0 << Vary: Origin, Cookie
[DEBUG] 2024-09-03 08:57:16.416 [[main]>worker0] headers - http-outgoing-0 << Strict-Transport-Security: max-age=16000000; includeSubDomains; preload;
[DEBUG] 2024-09-03 08:57:16.416 [[main]>worker0] headers - http-outgoing-0 << Content-Security-Policy: frame-ancestors https://*.test.com;
[DEBUG] 2024-09-03 08:57:16.417 [[main]>worker0] headers - http-outgoing-0 << X-Content-Type-Options: : nosniff
[DEBUG] 2024-09-03 08:57:16.417 [[main]>worker0] headers - http-outgoing-0 << Content-Length: 34
[DEBUG] 2024-09-03 08:57:16.419 [[main]>worker0] MainClientExec - Connection can be kept alive indefinitely
[DEBUG] 2024-09-03 08:57:16.430 [[main]>worker0] wire - http-outgoing-0 << "{"message": "Datatext is missing"}"

So to get around this I added the following to my filter

mutate {
  add_field => {
        "event_data" => "%{mal_md5}, %{mal_sha1}"
       }
}

AND the following to the http output

mapping => {
                "confidence" => "10"
                "classification" => "example"
                "type" => "type_example"
                "datatext" => "%{event_data}"
                    }

So I added the datatext mapping to get around this.

15

I was able to get this working by using the following:

filter {
  # Aggregate all alerts into a single array
  if [alerts] {
    aggregate {
      task_id => "%{[@metadata][_id]}"
      code => "
        map['alerts'] ||= []
        map['alerts'] += event.get('alerts')
        event.cancel()
      "
      push_map_as_event_on_timeout => true
      timeout_task_id_field => "[@metadata][_id]"
      timeout => 5
    }
  }

  # Extract md5 and sha1 hashes into separate fields
  if [alerts] {
    ruby {
      code => "
        mal_md5_list = []
        mal_sha1_list = []

        event.get('alerts').each do |alert|
          hashes = alert.dig('doc', 'source_file', 'hashes')
          if hashes
            mal_md5_list << hashes['md5'] if hashes['md5']
            mal_sha1_list << hashes['sha1'] if hashes['sha1']
          end
        end

        event.set('event_mal_md5', mal_md5_list)
        event.set('event_mal_sha1', mal_sha1_list)
      "
    }
  }

  # Remove all fields except event_mal_md5 and event_mal_sha1
  mutate {
    remove_field => ["@version", "@timestamp", "host", "path", "message", "alerts"]
  }
}

and then assigning the

 mapping => {
                "confidence" => "90"
                "classification" => "private"
                "type" => "type_example"
                "datatext" => "%{event_mal_md5}, %{event_mal_sha1}"