Is it possible to parse the whole data to a filter?

Elastic Stack Logstash
1

Hi all,

Is it possible to give the whole data I'm reading from rabbitmq and give it to a filter(for example as source for cipher filter so that it will encrypt all of it) ??

Thanks!!!

2

By default the whole message payload ends up in the message field which can then be given to, for example, the cipher filter.

3

Passing "source" => "message" doesn't work...

4

it looks like is trying to find a "message" field in the json which doesn't exist, so the result is just reading the logs..

5

Please show your configuration and an example message coming out from Logstash (using a stdout { codec => rubydebug } output).

6 
input{
	rabbitmq {
		key => "mykey"
		queue => "myqueue"
		host => "myhost
		exchange => "amq.topic"
		ack  => true
		auto_delete => false
		automatic_recovery => true
		codec => "json"
		durable => true
		exclusive => false
		metadata_enabled => true
		subscription_retry_interval_seconds => 5
		#type => "nginx"
		prefetch_count => 20
		port => 5672
		heartbeat => 5
		passive => false
		connection_timeout => 60000
		}
filter {

	cipher {
		algorithm => "AES-256-CBC"
		cipher_padding => 1

		# Use a static "iv"
		iv => "1234567890123456"

		# OR use a random IV per encryption
		#iv_random_length => 16

		key => "12345678987654321234567898745612"
		key_size => 32

		mode => "encrypt"
		source => "message"
		base64 => true
		max_cipher_reuse => 1
	}
	cipher {
		algorithm => "AES-256-CBC"
		cipher_padding => 1

		# Use a static "iv"
		iv => "1234567890123456"

		# OR use a random IV per encryption
		#iv_random_length => 16

		key => "12345678987654321234567898745612"
		key_size => 32

		mode => "decrypt"
		source => "message"
		target => "message_decrypted"
		base64 => true

		max_cipher_reuse => 1
	}
}

output {
	stdout {codec => rubydebug}
}

output :

 {
        "request" => {
              "user_ip" => "128.128.128.127",
                "agent" => "rdaneelolivaw@drop-pay.com",
               "georef" => "46.141496:12.205203",
               "method" => "POST|PATCH|DELETE",
            "end_point" => "aaaaa1",
         "app_instance" => "7D1aaaaaaaaaaaaa13",
             "app_name" => "Draaaaaa",
              "user_id" => "3aaaa1",
                   "aw" => "906aaaaaaaad4",
               "app_id" => "871aaaaaaaac7eb7",
           "user_agent" => "Daaaaaahaane",
             "username" => "33aaaaa7",
                   "ts" => "201aaaaaa:00"
     },
            "svc" => "daaaa",
     "@timestamp" => 2aaaaaa754Z,
       "resource" => {
          "request_attributes" => {
             "accountType" => "C|B"
         },
                   "object_id" => "BAC1923982",
         "response_attributes" => {
                                      "business_name" => "Mario Rossi",
             "user_role: `OWNER|LEGAL_REPR|DELEGATE`" => "Hello, world!",
                                       "account_type" => "C",
                                         "balance_av" => 100,
                                               "iban" => "aaaaaa",
                                               "logo" => "url logo",
                                         "balance_ef" => 100,
                                    "account_warning" => {
                    "business_fee_debit" => 0,
                 "pending_money_request" => 2,
                          "keys_version" => 2
             },
                                              "email" => "email@email.com",
                                             "status" => "ACTIVE"
         }
     },
       "@version" => "1",
          "event" => "baaadd",
          "class" => "aaa",
      "operation" => {
         "result" => {
                    "code" => "``",
             "description" => "``",
             "http_status" => "200"
         },
          "scope" => "dpapp"
     },
             "ts" => "2017-01-16T19:20:30.125+01:00"
 }
7

Don't use the json codec. Use the plain codec to get the full payload into the message field, do whatever you want with the payload, then use a json filter to deserialize the JSON blob.

8

thanks, hope to meet you someday :smiley:

9

@magnusbaeck What about if I don't want the .txt when using s3 output?

10

What about if I don't want the .txt when using s3 output?

I'm afraid I don't understand your question.

11

Using the s3 output plugin, the file are stored

ls.s3.312bc026-2f5d-49bc-ae9f-5940cf4ad9a6.2013-04-18T10.00.tag_hello.part0.txt

I don't want the .txt

My question is Can I remove it?

12

That's a totally different question. I don't think this is configurable.

13

so that means i need to change at the code level of this plugin?

14

Probably, yes.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.