Http output and curl eqvalent

Elastic Stack Logstash
1

Good Day,
I am trying to use the logstash http output command to replicate the following curl command?

curl -k PATCH https://etp.us.example.com/api/v1/policies/123455/deny -H "Content-Type: application/json" -H "x-example-api-key: <API-KEY>" -d'{"add": {"sender_ip": ["1.1.1.1", "1.1.1.2"],}}'

My logstash Output configuration for http

http {
    url => "https://etp.us.example.com/api/v1/policies/123455/deny"
    http_method => "patch"
    format => "json"
    headers => {
         "accept" => "application/json"
         "content_type" => "application/json"
         "x-example-api-key" => "<API-KEY>"
       }
    message => "%{sender_ips}"
 }

When I run logstash I actually get the servers API to send back a "HTTP/1.1 200 OK" but nothing shows up on the sever side. So it sends the sender_ip data, server accepts it but nothing shows up. If I run the curl command I get data showing up on the server side.

logstash debug shows the following:

[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "content_type: application/json[\r][\n]"
[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "accept: application/json[\r][\n]"
[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "Content-Type: application/json[\r][\n]"
[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "Content-Length: 418[\r][\n]"
[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "Host: etp.us.example.com[\r][\n]"
[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "User-Agent: Manticore 0.8.0[\r][\n]"
[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "Accept-Encoding: gzip,deflate[\r][\n]"
[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "[\r][\n]"
[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "{"sender_ips":"[\"6.3.2.5\",\"23.88.71.29\",\"37.220.6.115\",\"112.217.108.138\",\"211.192.239.232\",\"64.188.19.117\",\"142.11.209.144\",\"138.201.8.186\",\"193.29.13.167\",\"103.68.251.31\",\"103.238.226.160\",\"45.248.87.196\",\"64.44.131.109\",\"5.252.178.55\",\"185.243.115.65\",\"185.163.47.134\",\"185.117.75.156\",\"113.160.165.75\",\"202.58.104.100\",\"111.229.73.84\",\"185.221.202.35\",\"167.88.180.198\"]"}"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "HTTP/1.1 200 OK[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Date: Wed, 04 Dec 2024 21:19:05 GMT[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Content-Type: application/json[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Connection: keep-alive[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "CF-Ray: 8eced5273affce98-SJC[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "CF-Cache-Status: DYNAMIC[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Access-Control-Allow-Origin: http://example.com https://example.com http://examplegov.com https://examplegov.com http://test.com https://example.com[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Strict-Transport-Security: max-age=31536000; includeSubdomains;[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Access-Control-Allow-Credentials: true[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Access-Control-Allow-Headers: [\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Access-Control-Allow-Methods: GET, POST, PATCH, PUT, DELETE[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "X-Content-Type-Options: nosniff[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "X-Frame-Options: SAMEORIGIN[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "x-request-id: SJxZLvS0Xyx[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "X-XSS-Protection: 1; mode=block[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Vary: Accept-Encoding[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Server: cloudflare[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Content-Encoding: gzip[\r][\n]"
[DEBUG] 2024-12-04 14:19:05.407 [[main]>worker0] wire - http-outgoing-1 << "Content-Length: 12458[\r][\n]"

I am not sure why server is sending back a "HTTP/1.1 200 OK" Code but nothing shows up. It has to be the way my http output syntax is structured?

2

What do you mean by that?

3

The list of IP's for %{sender_ips} using logstash it never actually get populated when sending to the API on the server side, even though the server returns a HTTP/1.1 200 OK.
If I run the curl command to send IP's to the servers API, they are populated on the server side.
So I am trying figure out if the way I have my http Output plugin is structured how the curl command sends the data as its actually successful.

Thanks

4

The curl command you are using is different from the message logstash is sending.

In the curl command you shared you have this as the payload:

{"add": {"sender_ip": ["1.1.1.1", "1.1.1.2"]}}

But in your logstash message you have only the field %{sender_ips}, which from your debug log is being populated correctly:

[DEBUG] 2024-12-04 14:19:04.783 [[main]>worker0] wire - http-outgoing-1 >> "{"sender_ips":"["6.3.2.5","23.88.71.29","37.220.6.115","112.217.108.138","211.192.239.232","64.188.19.117","142.11.209.144","138.201.8.186","193.29.13.167","103.68.251.31","103.238.226.160","45.248.87.196","64.44.131.109","5.252.178.55","185.243.115.65","185.163.47.134","185.117.75.156","113.160.165.75","202.58.104.100","111.229.73.84","185.221.202.35","167.88.180.198"]"}"

Shouldn't this be { "add": { "sender_ip": [ list of ip addresses] }}?

Can you add an stdout output and share the output you are getting?

You will probably need to build the same payload in logstash.

6

Hi sorry that's a typo in the curl command
When it works with curl its:

-d'{"add": {"sender_ips": ["1.1.1.1", "1.1.1.2"],}}'

image
image1893×119 7.49 KB

my filer is:

filter {
  ruby {
    code => "
      sender_ips = []

      # Get the 'objects' array from the event
      objects = event.get('objects')
      if objects
        # Iterate through each object in the 'objects' array
        objects.each do |object|
          # Check if the type is 'ip'
          type = object['type']
          if type == 'ip'
            # Extract the IP address value and confidence if the type is 'ip'
            ip = object['value']
            confidence = object['source_reported_confidence']

            if ip
              # Determine the risk level based on the confidence value
              risk = case confidence
                     when 1..50 then 'Low'
                     when 51..70 then 'Medium'
                     when 71..99 then 'High'
                     when 100 then 'Critical'
                     else 'Unknown' # Default to Unknown if confidence is outside expected range
                     end

              # Only add the IP if the risk level is 'Critical'
              if risk == 'Critical'
                # Add the IP address to the sender_ips array
                sender_ips << ip
              end
            end
          end
        end
      end

      # Set the extracted IP addresses to 'sender_ips' in the event
      event.set('sender_ips', sender_ips.to_json)
    "
  }

  mutate {
    remove_field => ["objects"]
    remove_field => ["@version"]
    remove_field => ["@timestamp"]
    remove_field => ["meta"]
  }
}
7

And what is the output from logstash when using stdout? Please share as plain text, not as an image.

In your output you do not have the action, you do not have the add that you have in the curl command.

What happens if you make a curl with this payload:

-d'{"sender_ips": ["1.1.1.1", "1.1.1.2"],}'

Because this is what you are sending with logstash.

I think you need to change this:

 event.set('sender_ips', sender_ips.to_json)

Into this:

 event.set('[payload][add][sender_ips]', sender_ips.to_json)

Then use the %{payload} field in the output, but you need to test it, as you didn't a sample of the original message for this to be replicate.

Your issue seems to be that your payload from logstash is not the same from the curl command.

8

Thanks. I see what you are getting at now.