# Is it possible to parse the whole data to a filter?

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475>\
**Category:** Logstash\
**Created:** [March 14, 2017, 9:14am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475 "2017-03-14T09:14:56Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vittorio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vittorio/32/18171_2.png) [@Vittorio](https://discuss.elastic.co/u/Vittorio)\
**Post date:** [March 14, 2017, 9:14am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/1 "2017-03-14T09:14:56Z")

</div>

Hi all,

Is it possible to give the whole data I'm reading from rabbitmq and give it to a filter(for example as source for cipher filter so that it will encrypt all of it) ??

Thanks!!!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2017, 9:29am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/2 "2017-03-14T09:29:46Z")

</div>

By default the whole message payload ends up in the `message` field which can then be given to, for example, the cipher filter.

---

<div class="post-metadata">

**Author:** ![Vittorio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vittorio/32/18171_2.png) [@Vittorio](https://discuss.elastic.co/u/Vittorio)\
**Post date:** [March 14, 2017, 9:44am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/3 "2017-03-14T09:44:32Z")

</div>

Passing `"source" => "message"` doesn't work...

---

<div class="post-metadata">

**Author:** ![Vittorio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vittorio/32/18171_2.png) [@Vittorio](https://discuss.elastic.co/u/Vittorio)\
**Post date:** [March 14, 2017, 9:53am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/4 "2017-03-14T09:53:46Z")

</div>

it looks like is trying to find a "message" field in the json which doesn't exist, so the result is just reading the logs..

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2017, 10:03am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/5 "2017-03-14T10:03:52Z")

</div>

Please show your configuration and an example message coming out from Logstash (using a `stdout { codec => rubydebug }` output).

---

<div class="post-metadata">

**Author:** ![Vittorio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vittorio/32/18171_2.png) [@Vittorio](https://discuss.elastic.co/u/Vittorio)\
**Post date:** [March 14, 2017, 10:15am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/6 "2017-03-14T10:15:27Z")

</div>

```
input{
	rabbitmq {
		key => "mykey"
		queue => "myqueue"
		host => "myhost
		exchange => "amq.topic"
		ack => true
		auto_delete => false
		automatic_recovery => true
		codec => "json"
		durable => true
		exclusive => false
		metadata_enabled => true
		subscription_retry_interval_seconds => 5
		#type => "nginx"
		prefetch_count => 20
		port => 5672
		heartbeat => 5
		passive => false
		connection_timeout => 60000
		}
filter {

	cipher {
		algorithm => "AES-256-CBC"
		cipher_padding => 1

		# Use a static "iv"
		iv => "1234567890123456"

		# OR use a random IV per encryption
		#iv_random_length => 16

		key => "12345678987654321234567898745612"
		key_size => 32

		mode => "encrypt"
		source => "message"
		base64 => true
		max_cipher_reuse => 1
	}
	cipher {
		algorithm => "AES-256-CBC"
		cipher_padding => 1

		# Use a static "iv"
		iv => "1234567890123456"

		# OR use a random IV per encryption
		#iv_random_length => 16

		key => "12345678987654321234567898745612"
		key_size => 32

		mode => "decrypt"
		source => "message"
		target => "message_decrypted"
		base64 => true

		max_cipher_reuse => 1
	}
}

output {
	stdout {codec => rubydebug}
}

```

output :

> ```
> {
> "request" => {
> "user_ip" => "128.128.128.127",
> "agent" => "rdaneelolivaw@drop-pay.com",
> "georef" => "46.141496:12.205203",
> "method" => "POST|PATCH|DELETE",
> "end_point" => "aaaaa1",
> "app_instance" => "7D1aaaaaaaaaaaaa13",
> "app_name" => "Draaaaaa",
> "user_id" => "3aaaa1",
> "aw" => "906aaaaaaaad4",
> "app_id" => "871aaaaaaaac7eb7",
> "user_agent" => "Daaaaaahaane",
> "username" => "33aaaaa7",
> "ts" => "201aaaaaa:00"
> },
> "svc" => "daaaa",
> "@timestamp" => 2aaaaaa754Z,
> "resource" => {
> "request_attributes" => {
> "accountType" => "C|B"
> },
> "object_id" => "BAC1923982",
> "response_attributes" => {
> "business_name" => "Mario Rossi",
> "user_role: `OWNER|LEGAL_REPR|DELEGATE`" => "Hello, world!",
> "account_type" => "C",
> "balance_av" => 100,
> "iban" => "aaaaaa",
> "logo" => "url logo",
> "balance_ef" => 100,
> "account_warning" => {
> "business_fee_debit" => 0,
> "pending_money_request" => 2,
> "keys_version" => 2
> },
> "email" => "email@email.com",
> "status" => "ACTIVE"
> }
> },
> "@version" => "1",
> "event" => "baaadd",
> "class" => "aaa",
> "operation" => {
> "result" => {
> "code" => "``",
> "description" => "``",
> "http_status" => "200"
> },
> "scope" => "dpapp"
> },
> "ts" => "2017-01-16T19:20:30.125+01:00"
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2017, 10:19am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/7 "2017-03-14T10:19:55Z")

</div>

Don't use the json codec. Use the plain codec to get the full payload into the `message` field, do whatever you want with the payload, then use a json filter to deserialize the JSON blob.

---

<div class="post-metadata">

**Author:** ![Vittorio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vittorio/32/18171_2.png) [@Vittorio](https://discuss.elastic.co/u/Vittorio)\
**Post date:** [March 14, 2017, 10:41am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/8 "2017-03-14T10:41:07Z")

</div>

> [@magnusbaeck](#):
>
> Use the plain codec to get the full payload into the message field, do whatever you want with the payload, then use a json filter to deserialize the JSON blob

thanks, hope to meet you someday 😃

---

<div class="post-metadata">

**Author:** ![Vittorio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vittorio/32/18171_2.png) [@Vittorio](https://discuss.elastic.co/u/Vittorio)\
**Post date:** [March 14, 2017, 1:49pm UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/9 "2017-03-14T13:49:46Z")

</div>

@magnusbaeck What about if I don't want the .txt when using s3 output?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2017, 2:07pm UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/10 "2017-03-14T14:07:34Z")

</div>

> What about if I don't want the .txt when using s3 output?

I'm afraid I don't understand your question.

---

<div class="post-metadata">

**Author:** ![Vittorio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vittorio/32/18171_2.png) [@Vittorio](https://discuss.elastic.co/u/Vittorio)\
**Post date:** [March 14, 2017, 2:09pm UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/11 "2017-03-14T14:09:47Z")

</div>

Using the s3 output plugin, the file are stored

> ls.s3.312bc026-2f5d-49bc-ae9f-5940cf4ad9a6.2013-04-18T10.00.tag\_hello.part0.txt

I don't want the .txt

My question is Can I remove it?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2017, 2:13pm UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/12 "2017-03-14T14:13:27Z")

</div>

That's a totally different question. I don't think this is configurable.

---

<div class="post-metadata">

**Author:** ![Vittorio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vittorio/32/18171_2.png) [@Vittorio](https://discuss.elastic.co/u/Vittorio)\
**Post date:** [March 14, 2017, 2:15pm UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/13 "2017-03-14T14:15:55Z")

</div>

so that means i need to change at the code level of this plugin?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2017, 2:24pm UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/14 "2017-03-14T14:24:29Z")

</div>

Probably, yes.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 11, 2017, 2:24pm UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-the-whole-data-to-a-filter/78475/15 "2017-04-11T14:24:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.