Logstash remove_field issues

Elastic Stack Logstash
1

Hello,
I am trying to remove some fields from Twitter plugin using Logstash filter.

Partial json from Kibana:

{
  "_index": "twitter",
  "_type": "tweets",
  "_id": "iedY62QB9sEBYW4Cw1c0",
  "_version": 1,
  "_score": null,
  "_source": {
    "user": {
      "followers_count": 5037,
      "profile_image_url_https": "https://pbs.twimg.com/profile_images/974284997396967430/BcsHO0dZ_normal.jpg",
      "protected": false,
      "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
      "default_profile": true,
      "translator_type": "none",
      "following": null,
      "listed_count": 3,
      "profile_use_background_image": true,
      "screen_name": "CryptoPartyBoun",
      "favourites_count": 1370,
      "profile_sidebar_border_color": "C0DEED",
      "profile_background_color": "C0DEED",
      "created_at": "Fri Dec 14 08:09:19 +0000 2012",
      "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
      "geo_enabled": true,
      "description": "#bitcoin #btc #ethereum #eth #blockchain #cryptocurrency #altcoin #ico #follow #followme #follow4follow #followback #подписка #ЧитаюВзаимно #ВзаимныйФолловинг",
      "profile_background_tile": false,
      "url": null,
      "statuses_count": 686,
      "profile_image_url": "http://pbs.twimg.com/profile_images/974284997396967430/BcsHO0dZ_normal.jpg",
      "friends_count": 4922,
      "contributors_enabled": false,
      "default_profile_image": false,
      "follow_request_sent": null,
      "id": 1010662458,
      "id_str": "1010662458",
      "utc_offset": null,
      "notifications": null,
      "location": "USA, Iowa",
      "is_translator": false,
      "profile_sidebar_fill_color": "DDEEF6",
      "profile_text_color": "333333",
      "verified": false,
      "time_zone": null,
      "name": "CryptoParty",
      "lang": "en",
      "profile_link_color": "1DA1F2"
    },

logstash filter expression:

filter {
  prune {
    whitelist_names => [  
	"^entities$",  
	"^lang$",
	"^coordinates$",
	"^retweeted_status$",
	"^text$",
	"^extended_tweet$",
	"^@timestamp$", 
	"^user$"
    ]
  }
if [user] == "null" { mutate {
	remove_field => [
	"[user][contributors_enabled]",
	"[user][created_at]",
	"[user][default_profile]",
	"[user][default_profile_image]",
	"[user][description]",
	"[user][favourites_count]",
	"[user][followers_count]",
	"[user][following]",
	"[user][follow_request_sent]",
	"[user][friends_count]",
	"[user][id]",
	"[user][is_translator]",
	"[user][listed_count]",
	"[user][location]",
	"[user][notifications]",
	"[user][profile_background_color]",
	"[user][profile_background_image_url]",
	"[user][profile_background_image_url_https]",
	"[user][profile_background_tile]",
	"[user][profile_image_url_https]",
	"[user][profile_link_color]",
	"[user][profile_sidebar_border_color]",
	"[user][profile_sidebar_fill_color]",
	"[user][profile_text_color]",
	"[user][protected]",
	"[user][screen_name]",
	"[user][statuses_count]",
	"[user][time_zone]",
	"[user][translator_type]",
	"[user][url]",
	"[user][utc_offset]",
	"[user][verified]" ] }}
}

Logstash does not throw any error and works, ingesting data, but the remove_field filter does not filter the fields at all (whitelist_names filter works as expected).
What am I doing wrong here? Should I pipe it differently?

2

You only try to remove_field if [user] is null. Did you mean that to be != perhaps?

3

You are right. Changed to !=, I still do not have this data removed.
I have also tried without if statement before, but the result was the same.

4

I don't know. It works for me.

      "user" => {
    "profile_use_background_image" => true,
                            "name" => "CryptoParty",
                            "lang" => "en",
               "profile_image_url" => "http://pbs.twimg.com/profile_images/974284997396967430/BcsHO0dZ_normal.jpg",
                          "id_str" => "1010662458",
                     "geo_enabled" => true
}
5

Whats the result? Is it a some sort or debug? To what you were outputting to?

6

That is from

output { stdout { codec => rubydebug} }

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.