Delete "field" inside array

Elastic Stack Logstash
1

Hello everyone,

I'm using the "Twitter input" and I want to delete some fields that I don't need.

For example, I would like to remove (so not index in Elasticsearch), the field "indices" inside the field "[entities][hashtags]" :

   "entities": {
            "hashtags": [
              {
                "indices": [
                  10,
                  16
                ],
                "text": "sport"
              }
            ]
          },
        }

I tried to use "ruby filter" in order to loop into the "hashtags" array and delete the "indices" element but that doesn't work...

if ([entities][hashtags]) {
	ruby {
		code => "event.get('[entities][hashtags]').each {
					|k| k.delete('indices')
		}"
	}
}

Do you have any idea?

Thanks in advance for your help :slight_smile:

2

That piece of Ruby looks correct. Are you getting anything related to it in your logs?

3

Hello Magnus,

I checked my log files and I didn't see any error. The problem is that in my Elasticsearch index (ouput) I still have the field "indices" within "entities.hashtags". Do you have any idea to solve my problem?

Thank you in advance.

4

I'd try to narrow things down. Is the Ruby code being hit at all? What happens if you place event.tag('foo') at the beginning of the code block? Is the message tagged? What if you place it inside the each loop?

5

If I place event.tag('foo') at the beginning of the code block I obtain the field below in Elasticsearch, as expected :

   "tags": [
                "foo"
    ]

If I place the code here :

event.get('[entities][hashtags]').each {|k|
         event.tag('foo')
	 k.delete('indices')
}"

I also obtain the field below in Elasticsearch :

   "tags": [
                "foo"
    ]

I don't understand where is the problem...

For information, I'm using Elasticsearch 5.1.1 and Logstash 5.1.1

6

Then I don't know what's up.

7

I'm using a template for my index in Elasticsearch and in this template there is not the field "indices", do you know if it's possible to index only the fields that are in my template and ignore (do not index) the others fields ?

8

I'm not sure. I suggest you start a new topic in the Elasticsearch category.

9

It is possible to use remove_field => on my k object?

Something like this : remove_field => "[entities][hashtags][k][indices]"

One other thing, when I tried to use set() method in my k object like this :

event.get('[entities][hashtags]').each {|k|
	 event.tag('foo')
	 k.set('indices','this is a test')
}

I obtained the following error :

Ruby exception occurred: undefined method `set' for {"indices"=>[72, 80], "text"=>"data"}:Hash

Is it normal?

10

Is it normal?

Yes, that's expected.

11

So my only solution is to delete my objects in my array like this :

remove_field => "[entities][hashtags][0][indices]"
remove_field => "[entities][hashtags][1][indices]"

That works, but the problem is that I can have 0 or x objects in my array... How can I do this dynamically?

Thank you in advance.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.