Set "fields" into array - SOLVED

[adrianolimit]

Hello everyone,

I'm using the "Twitter input" and I want to put in lowercase one field, I tried some things but I didn't success ... I know that there are few changes since Logstash 5.0 (I'm using Logstash 5.1.1) with the "event API" and I don't understand how it works. This is my input data :

   "entities": {
            "hashtags": [
              {
                "indices": [
                  10,
                  16
                ],
                "text": "data"
              },...
            ]           
         },...
        }

But in my ruby filter in Logstash, the "hashtags" data seem to be "hash values" :

"entities": {
            "hashtags": [
              {
                      "indices"=> [10, 16],
                      "text" => "DATA"
               },...
            ]
         },...
        }

So, this is my ruby filter :

if ([entities][hashtags]) {
   ruby {
      code => '			
	     event.get("[entities][hashtags]").each {|hash|
	          hash.each { |key,value|
		     if key == "text"
		           event.tag(value.downcase) # Works, create an array of lowercase values in the field "tags"
			   event.set("[entities][hashtags]"[hash][key], value.downcase)  #Doesn't work, create an error
			   key= value.downcase  # Doesn't work
                           value = value.downcase  # Doesn't work			
  		      end
	          }
	       }'
          }
      }

I don't known what I have to do, I tried lot of things and I'm losing my mind.

Thank you in advance.

[magnusbaeck (Magnus Bäck)]

event.set("[entities][hashtags]"[key][k], value.downcase)  #Doesn't work, create an error

Yes, that's not surprising. Try

event.set("[entities][hashtags][#{key}][#{k}]", value.downcase)

to insert the values of key and k into the string.

[adrianolimit]

Hello Magnus,

I tried your solution :

event.set("[entities][hashtags][#{hash}][#{key}]", value.downcase)

And I obtained this error message :

[ERROR][logstash.filters.ruby ] Ruby exception occurred: For input string: "{"indices"=>"

I don't understand because in my filter I test if "k == text" before to try to set the value :

event.get("[entities][hashtags]").each {|hash|
	hash.each { |key,value|
		if key == "text"
			event.tag(value.downcase)
			event.set("[entities][hashtags][#{hash}][#{key}]", value.downcase)
		end
	}
}'

Do you have any idea?

[adrianolimit]

In the event that some people could have the same problem as me, I solved my problem with this code:

if ([entities][hashtags]) {
    ruby {
	   code => '	
	       i = 0
	       event.get("[entities][hashtags]").each {|hash|
		   event.set("[entities][hashtags][#{i}][text]",event.get("[entities][hashtags][#{i}][text]").downcase)
		   i += 1
	    }'
     }}

I could also used this code, (but for me it is slower that the first) :

if ([entities][hashtags]) {
    ruby {
	 code => '	
	      i = 0
	      event.get("[entities][hashtags]").each {|hash|
		   hash.each { |key,value|
			if key == "text"
				event.set("[entities][hashtags][#{i}][text]",value.downcase)
			end
		    }
		    i += 1
	       }'
        }
}

@magnusbaeck, In your opinion, what is the best code?