Saving just the "message" to file using http input and file output plugins

Elastic Stack Logstash
1

Hello,
I am trying to store the http input "message" to file for an http call sending:

{ "index" : { "_index" : "journaling_insert","_id":"A5CC1A05-09B9-4688-9796-14BB9E8A95FC"}}
{"REMOTEIP":"1.111.1.11","CHAINCODE":"8971","EVENTID":"16","STOREATTRIBUTE3":"Passed Value","DATETIME":"2025-03-20T06:00:00.000","STOREATTRIBUTE2":"StoreDB Value","STOREATTRIBUTE4":"StoreDB Value","STOREATTRIBUTE5":"StoreDB Value","FLSECURITY":{"SID":"1111"},"FLCUSTOMER":{"FIRSTNAME":"Ben","LASTNAME":"Hunsberger"}}
{ "index" : { "_index" : "journaling_insert","_id":"057C9CC1-D224-4C98-B97B-8D7F98588CE7"}}
{"REMOTEIP":"1.111.1.11","CHAINCODE":"8971","EVENTID":"17","DRAWERIDENT":"test","DATETIME":"2025-03-20T06:30:00.000","STOREATTRIBUTE2":"StoreDB Value","STOREATTRIBUTE3":"StoreDB Value","STOREATTRIBUTE4":"StoreDB Value","STOREATTRIBUTE5":"StoreDB Value","FLTRANSACTIONATTRIBUTES":{"INVOICENUMBER":"1111"},"FLCUSTOMER":{"FIRSTNAME":"Scott","LASTNAME":"Lynn"}}

and, using what I know, I am trying to build the output file using this configuration:

input {
	http {
		port => 6043
	}
}
filter {
    mutate { gsub => [ "message", "\r", "" ] }
    mutate { split => { "message" => "
" } }
    ruby {
        code => '
            msg = event.get("message")
            if msg.is_a? Array
                while msg.length > 0 do
                    clone = event.clone
                    clone.set("message", msg.shift(1))

                    new_event_block.call(clone)
                end
            end
        '
    }
    if [message][0] {
        json { source => "[message][0]" }
    }	
	mutate {
		remove_field => "headers"
		remove_field => "host"
		remove_field => "message"
		remove_field => "@version"
	}	
}
output {
    file {
        path => "/log_streaming/my_app/log-%{+yyyy-MM-dd_HH.mm.ss.SSS}.log"	
		flush_interval => 0
		stale_cleanup_interval => 1
    }
}

and the result is almost it but I have:

{"@timestamp":"2025-03-24T16:24:39.501Z","index":{"_index":"journaling_insert","_id":"A5CC1A05-09B9-4688-9796-14BB9E8A95FC"}}
{"@timestamp":"2025-03-24T16:24:39.501Z","index":{"_index":"journaling_insert","_id":"A5CC1A05-09B9-4688-9796-14BB9E8A95FC"}}
{"CHAINCODE":"8971","STOREATTRIBUTE2":"StoreDB Value","FLSECURITY":{"SID":"1111"},"DATETIME":"2025-03-24T16:24:37.761","REMOTEIP":"1.111.1.11","STOREATTRIBUTE4":"StoreDB Value","EVENTID":"16","STOREATTRIBUTE3":"Passed Value","@timestamp":"2025-03-24T16:24:39.501Z","STOREATTRIBUTE5":"StoreDB Value","FLCUSTOMER":{"LASTNAME":"the Grey","FIRSTNAME":"Gandalf"}}
{"@timestamp":"2025-03-24T16:24:39.501Z","index":{"_index":"journaling_insert","_id":"057C9CC1-D224-4C98-B97B-8D7F98588CE7"}}
{"CHAINCODE":"8971","STOREATTRIBUTE2":"StoreDB Value","DATETIME":"2025-03-24T16:24:37.761","REMOTEIP":"1.111.1.11","DRAWERIDENT":"test","STOREATTRIBUTE4":"StoreDB Value","FLTRANSACTIONATTRIBUTES":{"INVOICENUMBER":"1111"},"EVENTID":"17","STOREATTRIBUTE3":"StoreDB Value","@timestamp":"2025-03-24T16:24:39.501Z","STOREATTRIBUTE5":"StoreDB Value","FLCUSTOMER":{"LASTNAME":"the Grey","FIRSTNAME":"Gandalf"}}

A first additional duplicated first line and a @timestamp field at every line.
How can I fix it to not have the @timestamp removed and the first line not duplicated?
But more openly, how can I save just the request body that I send with no "\r" and not other fields, like just as text?
@Badger this is probably a good question for you but anybody's help is obviously welcome gladly.

2

An [@timestamp] field is automatically added when an event is created. If you do not want it then modify your mutate to remove it

mutate { remove_field => [ "headers", "host", "message", "@version", "@timestamp" ] }

If you think about the ruby code, it splits the http response into an array of four lines, then it calls event.clone for each line. So you end up with the original event and four clones of it, so five lines in the output.

If you look at the post where you copied that code from you will see there is a call to event.cancel which deletes the original event if there are no problems. You could remove the if-else that surrounds it, but do not remove the call itself.

1 Like
3

Ah! Thanks! I am not familiar with Ruby and I didn't understand that!

@Badger , the problem I have now is that the output file I save is not able anymore to calculate the file name as

log-%{+yyyy-MM-dd_HH.mm.ss.SSS}.log

Is there a way to make work the removal of the @timestamp and the use of it in the file name building?

And in general, there is no way I can store the request body as text without all this conversion?

4

Add the following before the call the new_event_block

 clone.set("[@metadata][path]", clone.sprintf("/log_streaming/my_app/log-%{+yyyy-MM-dd_HH.mm.ss.SSS}.log"))

and change the file output to use path => "%{[@metadata][path]}"

6

It worked perfectly, great!
As final question, in general, is there no way I can store the request body as text without all this conversion?

7

Yes. Note that the only reason to start using ruby was to process the request body two lines at a time. If you don't want to do that then all it is really doing is a split.

    mutate { gsub => [ "message", "\r", "" ] }
    mutate { split => { "message" => "
" } }

    split { field => "message" }

You can set the file path using

 mutate { add_field => { "[@metadata][path]"  => "/log_streaming/my_app/log-%{+yyyy-MM-dd_HH.mm.ss.SSS}.log" } }

But ... if you just want to write the request body to a file then you can use

codec => line { format => "%{message}" }

on the file input and then you don't need to remove the [@timestamp] field, and so can let the file output do the sprintf for you.

8

Yes! That's exactly what I wanted! Thanks @Badger !