Line Break Not Working

Elastic Stack Logstash
1

Good Day,
How do you output new line with a break within Logstash. I am currently trying to do this with the following line from my filter:

event.set('sender_domain', sender_domain.join('\n'))

I want to the output of the filter to show the data as

example.com
example1.com
example2.com
example3.com

when sending to STDOUT it outputs as per below:

{
    "sender_domain" => "example.com\\example1.com\\example2.com\\example3.com"
}

When its sent via the http output plugin its sent as:

[DEBUG] 2024-11-13 13:34:04.406 [[main]>worker1] wire - http-outgoing-1 >> "example.com\example1.com\example2.com\example3.com\n"

Even when outputting to a text file as a test its comming in as:

example.com\example1.com\example2.com\example3.com\

I have the following filter:

filter {
  ruby {
    code => "
      sender_domain = []

      # Get the 'objects' array from the event
      objects = event.get('objects')
      if objects
        # Iterate through each object in the 'objects' array
        objects.each do |object|
          # Check if the type is 'domain'
          type = object['type']
          if type == 'domain'
            # Extract the IP address value and confidence if the type is 'domain'
            value = object['value']
            confidence = object['source_reported_confidence']

            if value
              # Determine the risk level based on the confidence value
              risk = case confidence
                     when 1..50 then 'Low'
                     when 51..70 then 'Medium'
                     when 71..89 then 'High'
                     when 100..100 then 'Critical'
                     else 'Unknown' # Default to Unknown if confidence is outside expected range
                     end

              # Only add the domain if the risk level is 'Critical'
              if risk == 'Critical'
                # Add the domain to the sender_domain array
                sender_domain << value
              end
            end
          end
        end
      end

      # Join the sender_domain array into a single string with new lines and set it in the event
      event.set('sender_domain', sender_domain.join('\n'))
    "
  }

  mutate {
    remove_field => ["objects"]
    remove_field => ["@version"]
    remove_field => ["@timestamp"]
    remove_field => ["meta"]
  }
}

output {
  stdout {
    codec => rubydebug
  }

file {
        path => "/usr/share/logstash/bin/domain.txt"
        codec => plain { format => "%{sender_domain}\n" }
    }

    http {
    url => "https://test.com/hx/api/v3/indicators/custom/indicator/conditions"
    http_method => "patch"
    format => "message"
    headers => {
         "accept" => "application/json, text/plain, application/json, application/json"
         "authorization" => "Basic <password>"
         "content_type" => "plain/text"
       }
    message => "%{sender_domain}\n"
 }
}

I have played around with the below but it never seems to be able to actually add the new line break .

event.set('sender_domain', sender_domain.join('\n'))

I the output to be structured as:

example.com
example1.com
example2.com
example3.com

Thanks

2

The rubydebug output and the DEBUG log message

where rubydebug shows two backslashes and log4j shows one are consistent and show that the array members have been joined using a single backslash, not newline. In ruby that would be .join("\\").

A .join("\n") in ruby would do exactly what you want it to do -- join with newline. I cannot see any way for the code you show to produce the results that you are getting.

As an aside....

mutate { join => { "sender_domain" => "\n" } }

does not do what you want, it use \n as a two character string to do the join.

3

In your example you are using a "double quote", if I try that the .conf file fails to startup, I have to use a 'single quote'.
.join("\n") has to be .join('\n') for my logstash to startup

Also I am using logstash version logstash 7.17.23 could that have anything to do with it now working when by all accounts it should be working as expected?

4

In a ruby filter you can either surround the code block with double quotes and use single quotes within it

ruby { code => "puts 'Hello, world!'" }

or surround the code block with single quotes and use double quotes within it:

ruby { code => 'puts "Hello, world!"' }

I always use this second form because it allows me to use string interpolation when I need it