Twitter Input - Logstash filter mutate remove_field - Elasticsearch

Elastic Stack Logstash
1

Hello eveybody,

Im trying to discover the fantastic world of possibilities of ELK. But i stop in a problem, and maybe someone can solve this "equation".

Im using the Twitter Input Plugin to receive Twitter data, but i dont want to use all fields that the API delivers. So im trying to remove the unwanted fields with mutate {remove_field) filter. But its not working.

My logstash.conf

input {
twitter {
# set key and token values explained steps in readme
consumer_key => "xxxxxxxxx" #API key
consumer_secret => "xxxxxxxxx" #API secret key
oauth_token =>"xxxxxxxx" #acess_token
oauth_token_secret =>"xxxxxxxx" #acess_token secret
# words to find
keywords => ["bbb23", "lula", "bolsonaro"]
languages => ["pt"]
full_tweet => true
use_proxy => false
tags => ["twitter"]
}
}

####### fields that i dont want
filter {
mutate {
remove_field => [ "entities.hashtags.indices", "display_text_range", "entities.media.additional_media_info.monetizable", "entities.media.id", "entities.media.indices", "entities.media.media_url", "entities.media.sizes.large.h", "entities.media.sizes.large.resize", "entities.media.sizes.large.w", "entities.media.sizes.medium.h", "entities.media.sizes.medium.resize", "entities.media.sizes.medium.w", "entities.media.sizes.small.h", "entities.media.sizes.small.resize", "entities.media.sizes.small.w", "entities.media.sizes.thumb.h", "entities.media.sizes.thumb.resize", "entities.media.sizes.thumb.w", "entities.media.source_status_id", "entities.media.source_user_id", "entities.urls.indices" ]
}
}

output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "logstash_internal"
password => "${LOGSTASH_INTERNAL_PASSWORD}"
index => "twitter_2"
}
}

Data example:

{
"@timestamp": [
"2023-01-31T04:02:19.000Z"
],
"@version": [
"1"
],
"@version.keyword": [
"1"
],
"created_at": [
"Tue Jan 31 04:02:19 +0000 2023"
],
"created_at.keyword": [
"Tue Jan 31 04:02:19 +0000 2023"
],
"entities.media.display_url": [
"pic.twitter.com/ohBYrGYUsz"
],
"entities.media.display_url.keyword": [
"pic.twitter.com/ohBYrGYUsz"
],
"entities.media.expanded_url": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"entities.media.expanded_url.keyword": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"entities.media.id": [
1617421607089094700
],
"entities.media.id_str": [
"1617421607089094656"
],
"entities.media.id_str.keyword": [
"1617421607089094656"
],
"entities.media.indices": [
72,
95
],
"entities.media.media_url": [
"http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"entities.media.media_url_https": [
"https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"entities.media.media_url_https.keyword": [
"https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"entities.media.media_url.keyword": [
"http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"entities.media.sizes.large.h": [
1106
],
"entities.media.sizes.large.resize": [
"fit"
],
"entities.media.sizes.large.resize.keyword": [
"fit"
],
"entities.media.sizes.large.w": [
1242
],
"entities.media.sizes.medium.h": [
1069
],
"entities.media.sizes.medium.resize": [
"fit"
],
"entities.media.sizes.medium.resize.keyword": [
"fit"
],
"entities.media.sizes.medium.w": [
1200
],
"entities.media.sizes.small.h": [
606
],
"entities.media.sizes.small.resize": [
"fit"
],
"entities.media.sizes.small.resize.keyword": [
"fit"
],
"entities.media.sizes.small.w": [
680
],
"entities.media.sizes.thumb.h": [
150
],
"entities.media.sizes.thumb.resize": [
"crop"
],
"entities.media.sizes.thumb.resize.keyword": [
"crop"
],
"entities.media.sizes.thumb.w": [
150
],
"entities.media.source_status_id": [
1617421619580014600
],
"entities.media.source_status_id_str": [
"1617421619580014592"
],
"entities.media.source_status_id_str.keyword": [
"1617421619580014592"
],
"entities.media.source_user_id": [
35801196
],
"entities.media.source_user_id_str": [
"35801196"
],
"entities.media.source_user_id_str.keyword": [
"35801196"
],
"entities.media.type": [
"photo"
],
"entities.media.type.keyword": [
"photo"
],
"entities.media.url": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"entities.media.url.keyword": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"entities.user_mentions.id": [
35801196
],
"entities.user_mentions.id_str": [
"35801196"
],
"entities.user_mentions.id_str.keyword": [
"35801196"
],
"entities.user_mentions.indices": [
3,
14
],
"entities.user_mentions.name": [
"Andrea Bella de OyĂĄ e de VĂȘnus​:small_red_triangle::female_sign:☭:zap::fist:t3::triangular_flag_on_post::heart_on_fire::taurus:"
],
"entities.user_mentions.name.keyword": [
"Andrea Bella de OyĂĄ e de VĂȘnus​:small_red_triangle::female_sign:☭:zap::fist:t3::triangular_flag_on_post::heart_on_fire::taurus:"
],
"entities.user_mentions.screen_name": [
"andreaad76"
],
"entities.user_mentions.screen_name.keyword": [
"andreaad76"
],
"extended_entities.media.display_url": [
"pic.twitter.com/ohBYrGYUsz"
],
"extended_entities.media.display_url.keyword": [
"pic.twitter.com/ohBYrGYUsz"
],
"extended_entities.media.expanded_url": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"extended_entities.media.expanded_url.keyword": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"extended_entities.media.id": [
1617421607089094700
],
"extended_entities.media.id_str": [
"1617421607089094656"
],
"extended_entities.media.id_str.keyword": [
"1617421607089094656"
],
"extended_entities.media.indices": [
72,
95
],
"extended_entities.media.media_url": [
"http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"extended_entities.media.media_url_https": [
"https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"extended_entities.media.media_url_https.keyword": [
"https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"extended_entities.media.media_url.keyword": [
"http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"extended_entities.media.sizes.large.h": [
1106
],
"extended_entities.media.sizes.large.resize": [
"fit"
],
"extended_entities.media.sizes.large.resize.keyword": [
"fit"
],
"extended_entities.media.sizes.large.w": [
1242
],
"extended_entities.media.sizes.medium.h": [
1069
],
"extended_entities.media.sizes.medium.resize": [
"fit"
],
"extended_entities.media.sizes.medium.resize.keyword": [
"fit"
],
"extended_entities.media.sizes.medium.w": [
1200
],
"extended_entities.media.sizes.small.h": [
606
],
"extended_entities.media.sizes.small.resize": [
"fit"
],
"extended_entities.media.sizes.small.resize.keyword": [
"fit"
],
"extended_entities.media.sizes.small.w": [
680
],
"extended_entities.media.sizes.thumb.h": [
150
],
"extended_entities.media.sizes.thumb.resize": [
"crop"
],
"extended_entities.media.sizes.thumb.resize.keyword": [
"crop"
],
"extended_entities.media.sizes.thumb.w": [
150
],
"extended_entities.media.source_status_id": [
1617421619580014600
],
"extended_entities.media.source_status_id_str": [
"1617421619580014592"
],
"extended_entities.media.source_status_id_str.keyword": [
"1617421619580014592"
],
"extended_entities.media.source_user_id": [
35801196
],
"extended_entities.media.source_user_id_str": [
"35801196"
],
"extended_entities.media.source_user_id_str.keyword": [
"35801196"
],
"extended_entities.media.type": [
"photo"
],
"extended_entities.media.type.keyword": [
"photo"
],
"extended_entities.media.url": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"extended_entities.media.url.keyword": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"favorite_count": [
0
],
"favorited": [
false
],
"filter_level": [
"low"
],
"filter_level.keyword": [
"low"
],
"id": [
1620271231269478400
],
"id_str": [
"1620271231269478400"
],
"id_str.keyword": [
"1620271231269478400"
],
"is_quote_status": [
false
],
"lang": [
"pt"
],
"lang.keyword": [
"pt"
],
"possibly_sensitive": [
false
],
"quote_count": [
0
],
"reply_count": [
0
],
"retweet_count": [
0
],
"retweeted": [
false
],
"retweeted_status.created_at": [
"Mon Jan 23 07:18:59 +0000 2023"
],
"retweeted_status.created_at.keyword": [
"Mon Jan 23 07:18:59 +0000 2023"
],
"retweeted_status.display_text_range": [
0,
55
],
"retweeted_status.entities.media.display_url": [
"pic.twitter.com/ohBYrGYUsz"
],
"retweeted_status.entities.media.display_url.keyword": [
"pic.twitter.com/ohBYrGYUsz"
],
"retweeted_status.entities.media.expanded_url": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.entities.media.expanded_url.keyword": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.entities.media.id": [
1617421607089094700
],
"retweeted_status.entities.media.id_str": [
"1617421607089094656"
],
"retweeted_status.entities.media.id_str.keyword": [
"1617421607089094656"
],
"retweeted_status.entities.media.indices": [
56,
79
],
"retweeted_status.entities.media.media_url": [
"http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"retweeted_status.entities.media.media_url_https": [
"https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"retweeted_status.entities.media.media_url_https.keyword": [
"https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"retweeted_status.entities.media.media_url.keyword": [
"http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"retweeted_status.entities.media.sizes.large.h": [
1106
],
"retweeted_status.entities.media.sizes.large.resize": [
"fit"
],
"retweeted_status.entities.media.sizes.large.resize.keyword": [
"fit"
],
"retweeted_status.entities.media.sizes.large.w": [
1242
],
"retweeted_status.entities.media.sizes.medium.h": [
1069
],
"retweeted_status.entities.media.sizes.medium.resize": [
"fit"
],
"retweeted_status.entities.media.sizes.medium.resize.keyword": [
"fit"
],
"retweeted_status.entities.media.sizes.medium.w": [
1200
],
"retweeted_status.entities.media.sizes.small.h": [
606
],
"retweeted_status.entities.media.sizes.small.resize": [
"fit"
],
"retweeted_status.entities.media.sizes.small.resize.keyword": [
"fit"
],
"retweeted_status.entities.media.sizes.small.w": [
680
],
"retweeted_status.entities.media.sizes.thumb.h": [
150
],
"retweeted_status.entities.media.sizes.thumb.resize": [
"crop"
],
"retweeted_status.entities.media.sizes.thumb.resize.keyword": [
"crop"
],
"retweeted_status.entities.media.sizes.thumb.w": [
150
],
"retweeted_status.entities.media.type": [
"photo"
],
"retweeted_status.entities.media.type.keyword": [
"photo"
],
"retweeted_status.entities.media.url": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.entities.media.url.keyword": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.extended_entities.media.display_url": [
"pic.twitter.com/ohBYrGYUsz"
],
"retweeted_status.extended_entities.media.display_url.keyword": [
"pic.twitter.com/ohBYrGYUsz"
],
"retweeted_status.extended_entities.media.expanded_url": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.extended_entities.media.expanded_url.keyword": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.extended_entities.media.id": [
1617421607089094700
],
"retweeted_status.extended_entities.media.id_str": [
"1617421607089094656"
],
"retweeted_status.extended_entities.media.id_str.keyword": [
"1617421607089094656"
],
"retweeted_status.extended_entities.media.indices": [
56,
79
],
"retweeted_status.extended_entities.media.media_url": [
"http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"retweeted_status.extended_entities.media.media_url_https": [
"https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"retweeted_status.extended_entities.media.media_url_https.keyword": [
"https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"retweeted_status.extended_entities.media.media_url.keyword": [
"http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg"
],
"retweeted_status.extended_entities.media.sizes.large.h": [
1106
],
"retweeted_status.extended_entities.media.sizes.large.resize": [
"fit"
],
"retweeted_status.extended_entities.media.sizes.large.resize.keyword": [
"fit"
],
"retweeted_status.extended_entities.media.sizes.large.w": [
1242
],
"retweeted_status.extended_entities.media.sizes.medium.h": [
1069
],
"retweeted_status.extended_entities.media.sizes.medium.resize": [
"fit"
],
"retweeted_status.extended_entities.media.sizes.medium.resize.keyword": [
"fit"
],
"retweeted_status.extended_entities.media.sizes.medium.w": [
1200
],
"retweeted_status.extended_entities.media.sizes.small.h": [
606
],
"retweeted_status.extended_entities.media.sizes.small.resize": [
"fit"
],
"retweeted_status.extended_entities.media.sizes.small.resize.keyword": [
"fit"
],
"retweeted_status.extended_entities.media.sizes.small.w": [
680
],
"retweeted_status.extended_entities.media.sizes.thumb.h": [
150
],
"retweeted_status.extended_entities.media.sizes.thumb.resize": [
"crop"
],
"retweeted_status.extended_entities.media.sizes.thumb.resize.keyword": [
"crop"
],
"retweeted_status.extended_entities.media.sizes.thumb.w": [
150
],
"retweeted_status.extended_entities.media.type": [
"photo"
],
"retweeted_status.extended_entities.media.type.keyword": [
"photo"
],
"retweeted_status.extended_entities.media.url": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.extended_entities.media.url.keyword": [
"https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.favorite_count": [
1408
],
"retweeted_status.favorited": [
false
],
"retweeted_status.filter_level": [
"low"
],
"retweeted_status.filter_level.keyword": [
"low"
],
"retweeted_status.id": [
1617421619580014600
],
"retweeted_status.id_str": [
"1617421619580014592"
],
"retweeted_status.id_str.keyword": [
"1617421619580014592"
],
"retweeted_status.is_quote_status": [
false
],
"retweeted_status.lang": [
"pt"
],
"retweeted_status.lang.keyword": [
"pt"
],
"retweeted_status.possibly_sensitive": [
false
],
"retweeted_status.quote_count": [
19
],
"retweeted_status.reply_count": [
83
],
"retweeted_status.retweet_count": [
321
],
"retweeted_status.retweeted": [
false
],
"retweeted_status.source": [
"<a href="http://twitter.com/download/iphone\" rel="nofollow">Twitter for iPhone"
],
"retweeted_status.source.keyword": [
"<a href="http://twitter.com/download/iphone\" rel="nofollow">Twitter for iPhone"
],
"retweeted_status.text": [
"BOLSONARO GENOCIDA \nBOLSONARO NA CADEIA\nBOLSONARO PRESO https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.text.keyword": [
"BOLSONARO GENOCIDA \nBOLSONARO NA CADEIA\nBOLSONARO PRESO https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"retweeted_status.truncated": [
false
],
"retweeted_status.user.contributors_enabled": [
false
],
"retweeted_status.user.created_at": [
"Mon Apr 27 17:29:21 +0000 2009"
],
"retweeted_status.user.created_at.keyword": [
"Mon Apr 27 17:29:21 +0000 2009"
],
"retweeted_status.user.default_profile": [
false
],
"retweeted_status.user.default_profile_image": [
false
],
"retweeted_status.user.description": [
"VIVER É UM ATO POLÍTICO​:anatomical_heart:”A massa nĂŁo Ă© apenas objeto da ação revolucionĂĄria;Ă© sobretudo sujeito.”Rosa Luxemburgo :small_red_triangle:de BH :black_heart: Sou GALO​:zap:Filha de OyĂĄ :no_entry_sign:DM :no_entry_sign:Porn"
],
"retweeted_status.user.description.keyword": [
"VIVER É UM ATO POLÍTICO​:anatomical_heart:”A massa nĂŁo Ă© apenas objeto da ação revolucionĂĄria;Ă© sobretudo sujeito.”Rosa Luxemburgo :small_red_triangle:de BH :black_heart: Sou GALO​:zap:Filha de OyĂĄ :no_entry_sign:DM :no_entry_sign:Porn"
],
"retweeted_status.user.favourites_count": [
44317
],
"retweeted_status.user.followers_count": [
16527
],
"retweeted_status.user.friends_count": [
13553
],
"retweeted_status.user.geo_enabled": [
true
],
"retweeted_status.user.id": [
35801196
],
"retweeted_status.user.id_str": [
"35801196"
],
"retweeted_status.user.id_str.keyword": [
"35801196"
],
"retweeted_status.user.is_translator": [
false
],
"retweeted_status.user.listed_count": [
11
],
"retweeted_status.user.location": [
"Curitiba, PR, Brasil"
],
"retweeted_status.user.location.keyword": [
"Curitiba, PR, Brasil"
],
"retweeted_status.user.name": [
"Andrea Bella de OyĂĄ e de VĂȘnus​:small_red_triangle::female_sign:☭:zap::fist:t3::triangular_flag_on_post::heart_on_fire::taurus:"
],
"retweeted_status.user.name.keyword": [
"Andrea Bella de OyĂĄ e de VĂȘnus​:small_red_triangle::female_sign:☭:zap::fist:t3::triangular_flag_on_post::heart_on_fire::taurus:"
],
"retweeted_status.user.profile_background_color": [
"CF5F7B"
],
"retweeted_status.user.profile_background_color.keyword": [
"CF5F7B"
],
"retweeted_status.user.profile_background_image_url": [
"http://abs.twimg.com/images/themes/theme7/bg.gif"
],
"retweeted_status.user.profile_background_image_url_https": [
"https://abs.twimg.com/images/themes/theme7/bg.gif"
],
"retweeted_status.user.profile_background_image_url_https.keyword": [
"https://abs.twimg.com/images/themes/theme7/bg.gif"
],
"retweeted_status.user.profile_background_image_url.keyword": [
"http://abs.twimg.com/images/themes/theme7/bg.gif"
],
"retweeted_status.user.profile_background_tile": [
true
],
"retweeted_status.user.profile_banner_url": [
"https://pbs.twimg.com/profile_banners/35801196/1674688206"
],
"retweeted_status.user.profile_banner_url.keyword": [
"https://pbs.twimg.com/profile_banners/35801196/1674688206"
],
"retweeted_status.user.profile_image_url": [
"http://pbs.twimg.com/profile_images/1616583126800547841/vO8qOoUu_normal.jpg"
],
"retweeted_status.user.profile_image_url_https": [
"https://pbs.twimg.com/profile_images/1616583126800547841/vO8qOoUu_normal.jpg"
],
"retweeted_status.user.profile_image_url_https.keyword": [
"https://pbs.twimg.com/profile_images/1616583126800547841/vO8qOoUu_normal.jpg"
],
"retweeted_status.user.profile_image_url.keyword": [
"http://pbs.twimg.com/profile_images/1616583126800547841/vO8qOoUu_normal.jpg"
],
"retweeted_status.user.profile_link_color": [
"990000"
],
"retweeted_status.user.profile_link_color.keyword": [
"990000"
],
"retweeted_status.user.profile_sidebar_border_color": [
"D141D1"
],
"retweeted_status.user.profile_sidebar_border_color.keyword": [
"D141D1"
],
"retweeted_status.user.profile_sidebar_fill_color": [
"9E839E"
],
"retweeted_status.user.profile_sidebar_fill_color.keyword": [
"9E839E"
],
"retweeted_status.user.profile_text_color": [
"4A104A"
],
"retweeted_status.user.profile_text_color.keyword": [
"4A104A"
],
"retweeted_status.user.profile_use_background_image": [
true
],
"retweeted_status.user.protected": [
false
],
"retweeted_status.user.screen_name": [
"andreaad76"
],
"retweeted_status.user.screen_name.keyword": [
"andreaad76"
],
"retweeted_status.user.statuses_count": [
21472
],
"retweeted_status.user.translator_type": [
"none"
],
"retweeted_status.user.translator_type.keyword": [
"none"
],
"retweeted_status.user.url": [
"https://twitter.com/i/status/1586918733238263808"
],
"retweeted_status.user.url.keyword": [
"https://twitter.com/i/status/1586918733238263808"
],
"retweeted_status.user.verified": [
false
],
"retweeted_status.user.verified_type": [
"blue"
],
"retweeted_status.user.verified_type.keyword": [
"blue"
],
"source": [
"<a href="https://mobile.twitter.com" rel="nofollow">Twitter Web App"
],
"source.keyword": [
"<a href="https://mobile.twitter.com" rel="nofollow">Twitter Web App"
],
"tags": [
"twitter"
],
"tags.keyword": [
"twitter"
],
"text": [
"RT @andreaad76: BOLSONARO GENOCIDA \nBOLSONARO NA CADEIA\nBOLSONARO PRESO https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"text.keyword": [
"RT @andreaad76: BOLSONARO GENOCIDA \nBOLSONARO NA CADEIA\nBOLSONARO PRESO https://twitter.com/andreaad76/status/1617421619580014592/photo/1"
],
"timestamp_ms": [
"1675137739375"
],
"timestamp_ms.keyword": [
"1675137739375"
],
"truncated": [
false
],
"user.contributors_enabled": [
false
],
"user.created_at": [
"Tue Sep 19 17:02:07 +0000 2017"
],
"user.created_at.keyword": [
"Tue Sep 19 17:02:07 +0000 2017"
],
"user.default_profile": [
true
],
"user.default_profile_image": [
false
],
"user.description": [
"Games Entretenimento Cinema Geek DiversĂŁo Humor"
],
"user.description.keyword": [
"Games Entretenimento Cinema Geek DiversĂŁo Humor"
],
"user.favourites_count": [
2114
],
"user.followers_count": [
15
],
"user.friends_count": [
208
],
"user.geo_enabled": [
false
],
"user.id": [
910187258388648000
],
"user.id_str": [
"910187258388647936"
],
"user.id_str.keyword": [
"910187258388647936"
],
"user.is_translator": [
false
],
"user.listed_count": [
0
],
"user.location": [
"Rio de Janeiro, Brasil"
],
"user.location.keyword": [
"Rio de Janeiro, Brasil"
],
"user.name": [
"DigCorreia"
],
"user.name.keyword": [
"DigCorreia"
],
"user.profile_background_color": [
"F5F8FA"
],
"user.profile_background_color.keyword": [
"F5F8FA"
],
"user.profile_background_image_url": [
""
],
"user.profile_background_image_url_https": [
""
],
"user.profile_background_image_url_https.keyword": [
""
],
"user.profile_background_image_url.keyword": [
""
],
"user.profile_background_tile": [
false
],
"user.profile_banner_url": [
"https://pbs.twimg.com/profile_banners/910187258388647936/1505951259"
],
"user.profile_banner_url.keyword": [
"https://pbs.twimg.com/profile_banners/910187258388647936/1505951259"
],
"user.profile_image_url": [
"http://pbs.twimg.com/profile_images/1161447549926019078/HF0THyNh_normal.jpg"
],
"user.profile_image_url_https": [
"https://pbs.twimg.com/profile_images/1161447549926019078/HF0THyNh_normal.jpg"
],
"user.profile_image_url_https.keyword": [
"https://pbs.twimg.com/profile_images/1161447549926019078/HF0THyNh_normal.jpg"
],
"user.profile_image_url.keyword": [
"http://pbs.twimg.com/profile_images/1161447549926019078/HF0THyNh_normal.jpg"
],
"user.profile_link_color": [
"1DA1F2"
],
"user.profile_link_color.keyword": [
"1DA1F2"
],
"user.profile_sidebar_border_color": [
"C0DEED"
],
"user.profile_sidebar_border_color.keyword": [
"C0DEED"
],
"user.profile_sidebar_fill_color": [
"DDEEF6"
],
"user.profile_sidebar_fill_color.keyword": [
"DDEEF6"
],
"user.profile_text_color": [
"333333"
],
"user.profile_text_color.keyword": [
"333333"
],
"user.profile_use_background_image": [
true
],
"user.protected": [
false
],
"user.screen_name": [
"DigCorreia"
],
"user.screen_name.keyword": [
"DigCorreia"
],
"user.statuses_count": [
1057
],
"user.translator_type": [
"none"
],
"user.translator_type.keyword": [
"none"
],
"user.verified": [
false
],
"user.verified_type": [
"blue"
],
"user.verified_type.keyword": [
"blue"
],
"_id": "87P9BYYBzTlWHRjvfmrL",
"_index": "twitter_2",
"_score": null
}

2

In JSON a field can be contain other fields (e.g. {"a": {"x":1, "y":2}} or just contain data {"a": 7}.

I'll give you 10 to 1 that elasticsearch, logstash, and kibana were developed by different teams and they realised that combining them was a huge win.

However, the logstash configuration language treats [a][b] as an [b] field within an [a] object, and [a.x] as a field with a stop in it. Kibana and Elasticsearch treat [a.b] as an [b] field inside an [a] object. It just does not parse it the same way.
Elasticsearch did stop allowing stops in field names until they improved the disambiguation engine.

What Kibana and Elasticsearch call user.verified Logstash would call

[user][verified]

Note that you can blow away the whole of [user] using a single entry in the remove_field option if you do not want any of it.

3

Hello Badger,

Thanks for explanation! But just to undestand...
Kibana and Elasticsearch understand: entities.media.additional_media_info.monetizable
Logstash understand: [entites][media][additional_media_info][monetizable]

And the right filter syntax is:

mutate {
remove_field => [
"display_text_range",
"[entities][media][additional_media_info][monetizable]",
"[entities][media][id]" ]
}

4

I try to use:

Option 01:

filter { if "twitter" in [tags]{
mutate {
remove_field => [ "[entities][media][additional_media_info]", "entities.media.additional_media_info.monetizable" , "[entities][media][additional_media_info][monetizable]" ]
}
}
}

Option 2:

filter {
mutate {
remove_field => [ "[entities][media][additional_media_info]", "entities.media.additional_media_info.monetizable" , "[entities][media][additional_media_info][monetizable]" ]
}
}

But none of them works. The field always appears.

5

Can you share the json of the document? Maybe there is an array on one of the levels, but to confirm it you need to share the json of the document.

1 Like
6

JSON example

{
    "lang": "pt",
    "is_quote_status": false,
    "tags": [
        "twitter"
    ],
    "timestamp_ms": "1675191730741",
    "reply_count": 0,
    "truncated": false,
    "favorite_count": 0,
    "entities": {
        "symbols": [],
        "media": [
            {
                "source_user_id_str": "3991142595",
                "display_url": "pic.twitter.com/oX9zyz4bKo",
                "type": "photo",
                "source_status_id": 1620230684739764225,
                "id_str": "1620230600165580802",
                "sizes": {
                    "large": {
                        "w": 1080,
                        "resize": "fit",
                        "h": 1080
                    },
                    "medium": {
                        "w": 1080,
                        "resize": "fit",
                        "h": 1080
                    },
                    "small": {
                        "w": 680,
                        "resize": "fit",
                        "h": 680
                    },
                    "thumb": {
                        "w": 150,
                        "resize": "crop",
                        "h": 150
                    }
                },
                "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                "url": "https://t.co/oX9zyz4bKo",
                "indices": [
                    112,
                    135
                ],
                "source_status_id_str": "1620230684739764225",
                "source_user_id": 3991142595,
                "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                "expanded_url": "https://twitter.com/sergiorvitoria/status/1620230684739764225/video/1",
                "id": 1620230600165580802,
                "additional_media_info": {
                    "monetizable": false
                }
            }
        ],
        "hashtags": [],
        "urls": [],
        "user_mentions": [
            {
                "id_str": "3991142595",
                "indices": [
                    3,
                    18
                ],
                "screen_name": "sergiorvitoria",
                "name": "SĂ©rgio VitĂłria",
                "id": 3991142595
            },
            {
                "id_str": "1523748277",
                "indices": [
                    20,
                    36
                ],
                "screen_name": "conexaopolitica",
                "name": "ConexĂŁo PolĂ­tica",
                "id": 1523748277
            },
            {
                "id_str": "1587148141001932804",
                "indices": [
                    37,
                    51
                ],
                "screen_name": "CarlineNagano",
                "name": "Carline Nagano đŸ‡§đŸ‡·đŸŒ»đŸ’šđŸ’›",
                "id": 1587148141001932804
            }
        ]
    },
    "possibly_sensitive": false,
    "coordinates": null,
    "geo": null,
    "place": null,
    "in_reply_to_user_id_str": null,
    "source": "<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>",
    "in_reply_to_user_id": null,
    "retweeted_status": {
        "lang": "pt",
        "display_text_range": [
            32,
            91
        ],
        "is_quote_status": false,
        "reply_count": 2,
        "truncated": false,
        "favorite_count": 62,
        "entities": {
            "symbols": [],
            "media": [
                {
                    "display_url": "pic.twitter.com/oX9zyz4bKo",
                    "type": "photo",
                    "id_str": "1620230600165580802",
                    "sizes": {
                        "large": {
                            "w": 1080,
                            "resize": "fit",
                            "h": 1080
                        },
                        "medium": {
                            "w": 1080,
                            "resize": "fit",
                            "h": 1080
                        },
                        "small": {
                            "w": 680,
                            "resize": "fit",
                            "h": 680
                        },
                        "thumb": {
                            "w": 150,
                            "resize": "crop",
                            "h": 150
                        }
                    },
                    "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                    "url": "https://t.co/oX9zyz4bKo",
                    "indices": [
                        92,
                        115
                    ],
                    "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                    "expanded_url": "https://twitter.com/sergiorvitoria/status/1620230684739764225/video/1",
                    "id": 1620230600165580802,
                    "additional_media_info": {
                        "monetizable": false
                    }
                }
            ],
            "hashtags": [],
            "urls": [],
            "user_mentions": [
                {
                    "id_str": "1523748277",
                    "indices": [
                        0,
                        16
                    ],
                    "screen_name": "conexaopolitica",
                    "name": "ConexĂŁo PolĂ­tica",
                    "id": 1523748277
                },
                {
                    "id_str": "1587148141001932804",
                    "indices": [
                        17,
                        31
                    ],
                    "screen_name": "CarlineNagano",
                    "name": "Carline Nagano đŸ‡§đŸ‡·đŸŒ»đŸ’šđŸ’›",
                    "id": 1587148141001932804
                }
            ]
        },
        "possibly_sensitive": false,
        "coordinates": null,
        "geo": null,
        "place": null,
        "in_reply_to_user_id_str": "1523748277",
        "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
        "in_reply_to_user_id": 1523748277,
        "retweeted": false,
        "created_at": "Tue Jan 31 01:21:12 +0000 2023",
        "id_str": "1620230684739764225",
        "retweet_count": 15,
        "in_reply_to_status_id": 1620175660865163264,
        "extended_entities": {
            "media": [
                {
                    "display_url": "pic.twitter.com/oX9zyz4bKo",
                    "type": "video",
                    "id_str": "1620230600165580802",
                    "sizes": {
                        "large": {
                            "w": 1080,
                            "resize": "fit",
                            "h": 1080
                        },
                        "medium": {
                            "w": 1080,
                            "resize": "fit",
                            "h": 1080
                        },
                        "small": {
                            "w": 680,
                            "resize": "fit",
                            "h": 680
                        },
                        "thumb": {
                            "w": 150,
                            "resize": "crop",
                            "h": 150
                        }
                    },
                    "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                    "url": "https://t.co/oX9zyz4bKo",
                    "indices": [
                        92,
                        115
                    ],
                    "video_info": {
                        "duration_millis": 35035,
                        "variants": [
                            {
                                "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/540x540/CdT6qOtm5mqqeUCr.mp4?tag=12",
                                "bitrate": 832000,
                                "content_type": "video/mp4"
                            },
                            {
                                "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/720x720/KDXJ0YCDpW5ggJqV.mp4?tag=12",
                                "bitrate": 1280000,
                                "content_type": "video/mp4"
                            },
                            {
                                "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/pl/F0TiM3lc0qKfgOHN.m3u8?tag=12&container=fmp4",
                                "content_type": "application/x-mpegURL"
                            },
                            {
                                "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/320x320/nFRwoSMqjhdtu2ru.mp4?tag=12",
                                "bitrate": 432000,
                                "content_type": "video/mp4"
                            }
                        ],
                        "aspect_ratio": [
                            1,
                            1
                        ]
                    },
                    "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                    "expanded_url": "https://twitter.com/sergiorvitoria/status/1620230684739764225/video/1",
                    "id": 1620230600165580802,
                    "additional_media_info": {
                        "monetizable": false
                    }
                }
            ]
        },
        "in_reply_to_screen_name": "conexaopolitica",
        "text": "@conexaopolitica @CarlineNagano Pacheco representa a continuação do desrespeito ditatorial! https://t.co/oX9zyz4bKo",
        "quote_count": 2,
        "filter_level": "low",
        "user": {
            "profile_background_color": "C0DEED",
            "profile_sidebar_border_color": "C0DEED",
            "profile_image_url_https": "https://pbs.twimg.com/profile_images/1588327555345666053/H_sPO9qS_normal.jpg",
            "verified": false,
            "profile_banner_url": "https://pbs.twimg.com/profile_banners/3991142595/1594481781",
            "default_profile_image": false,
            "notifications": null,
            "is_translator": false,
            "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
            "withheld_in_countries": [],
            "friends_count": 31129,
            "statuses_count": 62042,
            "name": "SĂ©rgio VitĂłria",
            "listed_count": 20,
            "profile_text_color": "333333",
            "followers_count": 54724,
            "following": null,
            "location": "Porto Alegre, Brasil",
            "time_zone": null,
            "profile_use_background_image": true,
            "favourites_count": 79993,
            "created_at": "Mon Oct 19 01:58:02 +0000 2015",
            "default_profile": true,
            "description": "Arquiteto đŸ‡§đŸ‡· \nA favor da liberdade de expressĂŁo e contra a bandidagem!\nO Brasil Ă© uma Republiqueta de Bananas!",
            "id_str": "3991142595",
            "verified_type": "blue",
            "utc_offset": null,
            "profile_image_url": "http://pbs.twimg.com/profile_images/1588327555345666053/H_sPO9qS_normal.jpg",
            "follow_request_sent": null,
            "translator_type": "none",
            "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
            "url": null,
            "protected": false,
            "profile_background_tile": false,
            "contributors_enabled": false,
            "profile_sidebar_fill_color": "DDEEF6",
            "geo_enabled": true,
            "id": 3991142595,
            "screen_name": "sergiorvitoria",
            "lang": null,
            "profile_link_color": "1DA1F2"
        },
        "id": 1620230684739764225,
        "contributors": null,
        "favorited": false,
        "in_reply_to_status_id_str": "1620175660865163264"
    },
    "retweeted": false,
    "created_at": "Tue Jan 31 19:02:10 +0000 2023",
    "type": "tweets",
    "id_str": "1620497687471882240",
    "retweet_count": 0,
    "in_reply_to_status_id": null,
    "extended_entities": {
        "media": [
            {
                "source_user_id_str": "3991142595",
                "display_url": "pic.twitter.com/oX9zyz4bKo",
                "type": "video",
                "source_status_id": 1620230684739764225,
                "id_str": "1620230600165580802",
                "sizes": {
                    "large": {
                        "w": 1080,
                        "resize": "fit",
                        "h": 1080
                    },
                    "medium": {
                        "w": 1080,
                        "resize": "fit",
                        "h": 1080
                    },
                    "small": {
                        "w": 680,
                        "resize": "fit",
                        "h": 680
                    },
                    "thumb": {
                        "w": 150,
                        "resize": "crop",
                        "h": 150
                    }
                },
                "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                "url": "https://t.co/oX9zyz4bKo",
                "indices": [
                    112,
                    135
                ],
                "source_status_id_str": "1620230684739764225",
                "source_user_id": 3991142595,
                "video_info": {
                    "duration_millis": 35035,
                    "variants": [
                        {
                            "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/540x540/CdT6qOtm5mqqeUCr.mp4?tag=12",
                            "bitrate": 832000,
                            "content_type": "video/mp4"
                        },
                        {
                            "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/720x720/KDXJ0YCDpW5ggJqV.mp4?tag=12",
                            "bitrate": 1280000,
                            "content_type": "video/mp4"
                        },
                        {
                            "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/pl/F0TiM3lc0qKfgOHN.m3u8?tag=12&container=fmp4",
                            "content_type": "application/x-mpegURL"
                        },
                        {
                            "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/320x320/nFRwoSMqjhdtu2ru.mp4?tag=12",
                            "bitrate": 432000,
                            "content_type": "video/mp4"
                        }
                    ],
                    "aspect_ratio": [
                        1,
                        1
                    ]
                },
                "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                "expanded_url": "https://twitter.com/sergiorvitoria/status/1620230684739764225/video/1",
                "id": 1620230600165580802,
                "additional_media_info": {
                    "monetizable": false
                }
            }
        ]
    },
    "@timestamp": "2023-01-31T19:02:10.000Z",
    "in_reply_to_screen_name": null,
    "text": "RT @sergiorvitoria: @conexaopolitica @CarlineNagano Pacheco representa a continuação do desrespeito ditatorial! https://t.co/oX9zyz4bKo",
    "quote_count": 0,
    "filter_level": "low",
    "@version": "1",
    "user": {
        "profile_background_color": "F5F8FA",
        "profile_sidebar_border_color": "C0DEED",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/1618418636686856192/wmyyOJ9l_normal.jpg",
        "verified": false,
        "profile_banner_url": "https://pbs.twimg.com/profile_banners/1458812188961947649/1667018148",
        "default_profile_image": false,
        "notifications": null,
        "is_translator": false,
        "profile_background_image_url": "",
        "withheld_in_countries": [],
        "friends_count": 320,
        "statuses_count": 1672,
        "name": "EstudandoDireito",
        "listed_count": 0,
        "profile_text_color": "333333",
        "followers_count": 10,
        "following": null,
        "location": "TrĂȘs coraçÔes MG",
        "time_zone": null,
        "profile_use_background_image": true,
        "favourites_count": 15,
        "created_at": "Thu Nov 11 15:03:14 +0000 2021",
        "default_profile": true,
        "description": "http://unincor.br",
        "id_str": "1458812188961947649",
        "verified_type": "blue",
        "utc_offset": null,
        "profile_image_url": "http://pbs.twimg.com/profile_images/1618418636686856192/wmyyOJ9l_normal.jpg",
        "follow_request_sent": null,
        "translator_type": "none",
        "profile_background_image_url_https": "",
        "url": "http://www.google.com",
        "protected": false,
        "profile_background_tile": false,
        "contributors_enabled": false,
        "profile_sidebar_fill_color": "DDEEF6",
        "geo_enabled": false,
        "id": 1458812188961947649,
        "screen_name": "MaxcuelS",
        "lang": null,
        "profile_link_color": "1DA1F2"
    },
    "id": 1620497687471882240,
    "contributors": null,
    "favorited": false,
    "in_reply_to_status_id_str": null
}
7

The "media" is an array, starts with [

Try with:
remove_field => [ "[entities][media][0][additional_media_info]" ]

1 Like
8

Rios,
Thanks a lot for the insight! Now I understood!

9

Is it working? I haven't tested :blush: :innocent:

1 Like
10

Works!!!! But still a question, if have more than one image, i need to put [1] [2]...? Or there some way to "auto identify" the array and remove "additional_media_info"?

11

I think no "auto identify" except if you have 2-3 array members, you can put:
remove_field => [ "[entities][media][0][additional_media_info]", ]"[entities][media][1][additional_media_info]", "[entities][media][2][additional_media_info]" ]
The mutate shouldn't report an error if field is missing.
If you don't know max number additional_media_info members, then use loop in ruby.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.