# Twitter Input - Logstash filter mutate remove\_field - Elasticsearch

**URL:** <https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290>\
**Category:** Logstash\
**Created:** [January 31, 2023, 4:06am UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290 "2023-01-31T04:06:35Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![xalmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xalmer/32/117685_2.png) [@xalmer](https://discuss.elastic.co/u/xalmer)\
**Post date:** [January 31, 2023, 4:06am UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/1 "2023-01-31T04:06:35Z")

</div>

Hello eveybody,

Im trying to discover the fantastic world of possibilities of ELK. But i stop in a problem, and maybe someone can solve this "equation".

Im using the Twitter Input Plugin to receive Twitter data, but i dont want to use all fields that the API delivers. So im trying to remove the unwanted fields with mutate {remove\_field) filter. But its not working.

**My logstash.conf**

input {  
twitter {  
# set key and token values explained steps in readme  
consumer\_key =\> "xxxxxxxxx" #API key  
consumer\_secret =\> "xxxxxxxxx" #API secret key  
oauth\_token =\>"xxxxxxxx" #acess\_token  
oauth\_token\_secret =\>"xxxxxxxx" #acess\_token secret  
# words to find  
keywords =\> ["bbb23", "lula", "bolsonaro"]  
languages =\> ["pt"]  
full\_tweet =\> true  
use\_proxy =\> false  
tags =\> ["twitter"]  
}  
}

####### fields that i dont want  
filter {  
mutate {  
remove\_field =\> ["entities.hashtags.indices", "display\_text\_range", "entities.media.additional\_media\_info.monetizable", "entities.media.id", "entities.media.indices", "entities.media.media\_url", "entities.media.sizes.large.h", "entities.media.sizes.large.resize", "entities.media.sizes.large.w", "entities.media.sizes.medium.h", "entities.media.sizes.medium.resize", "entities.media.sizes.medium.w", "entities.media.sizes.small.h", "entities.media.sizes.small.resize", "entities.media.sizes.small.w", "entities.media.sizes.thumb.h", "entities.media.sizes.thumb.resize", "entities.media.sizes.thumb.w", "entities.media.source\_status\_id", "entities.media.source\_user\_id", "entities.urls.indices"]  
}  
}

output {  
elasticsearch {  
hosts =\> "elasticsearch:9200"  
user =\> "logstash\_internal"  
password =\> "${LOGSTASH\_INTERNAL\_PASSWORD}"  
index =\> "twitter\_2"  
}  
}

**Data example:**

{  
"@timestamp": [  
"2023-01-31T04:02:19.000Z"  
],  
"@version": [  
"1"  
],  
"@version.keyword": [  
"1"  
],  
"created\_at": [  
"Tue Jan 31 04:02:19 +0000 2023"  
],  
"created\_at.keyword": [  
"Tue Jan 31 04:02:19 +0000 2023"  
],  
"entities.media.display\_url": [  
"pic.twitter.com/ohBYrGYUsz"  
],  
"entities.media.display\_url.keyword": [  
"pic.twitter.com/ohBYrGYUsz"  
],  
"entities.media.expanded\_url": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://twitter.com/andreaad76/status/1617421619580014592/photo/1)"  
],  
"entities.media.expanded\_url.keyword": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://twitter.com/andreaad76/status/1617421619580014592/photo/1)"  
],  
"entities.media.id": [  
1617421607089094700  
],  
"entities.media.id\_str": [  
"1617421607089094656"  
],  
"entities.media.id\_str.keyword": [  
"1617421607089094656"  
],  
"entities.media.indices": [  
72,  
95  
],  
"entities.media.media\_url": [  
"[http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"entities.media.media\_url\_https": [  
"[https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"entities.media.media\_url\_https.keyword": [  
"[https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"entities.media.media\_url.keyword": [  
"[http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"entities.media.sizes.large.h": [  
1106  
],  
"entities.media.sizes.large.resize": [  
"fit"  
],  
"entities.media.sizes.large.resize.keyword": [  
"fit"  
],  
"entities.media.sizes.large.w": [  
1242  
],  
"entities.media.sizes.medium.h": [  
1069  
],  
"entities.media.sizes.medium.resize": [  
"fit"  
],  
"entities.media.sizes.medium.resize.keyword": [  
"fit"  
],  
"entities.media.sizes.medium.w": [  
1200  
],  
"entities.media.sizes.small.h": [  
606  
],  
"entities.media.sizes.small.resize": [  
"fit"  
],  
"entities.media.sizes.small.resize.keyword": [  
"fit"  
],  
"entities.media.sizes.small.w": [  
680  
],  
"entities.media.sizes.thumb.h": [  
150  
],  
"entities.media.sizes.thumb.resize": [  
"crop"  
],  
"entities.media.sizes.thumb.resize.keyword": [  
"crop"  
],  
"entities.media.sizes.thumb.w": [  
150  
],  
"entities.media.source\_status\_id": [  
1617421619580014600  
],  
"entities.media.source\_status\_id\_str": [  
"1617421619580014592"  
],  
"entities.media.source\_status\_id\_str.keyword": [  
"1617421619580014592"  
],  
"entities.media.source\_user\_id": [  
35801196  
],  
"entities.media.source\_user\_id\_str": [  
"35801196"  
],  
"entities.media.source\_user\_id\_str.keyword": [  
"35801196"  
],  
"entities.media.type": [  
"photo"  
],  
"entities.media.type.keyword": [  
"photo"  
],  
"entities.media.url": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"entities.media.url.keyword": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"entities.user\_mentions.id": [  
35801196  
],  
"entities.user\_mentions.id\_str": [  
"35801196"  
],  
"entities.user\_mentions.id\_str.keyword": [  
"35801196"  
],  
"entities.user\_mentions.indices": [  
3,  
14  
],  
"entities.user\_mentions.name": [  
"Andrea Bella de Oyá e de Vênus​:small\_red\_triangle:♀☭:zap:✊🏼🚩❤️‍🔥♉"  
],  
"entities.user\_mentions.name.keyword": [  
"Andrea Bella de Oyá e de Vênus​:small\_red\_triangle:♀☭:zap:✊🏼🚩❤️‍🔥♉"  
],  
"entities.user\_mentions.screen\_name": [  
"andreaad76"  
],  
"entities.user\_mentions.screen\_name.keyword": [  
"andreaad76"  
],  
"extended\_entities.media.display\_url": [  
"pic.twitter.com/ohBYrGYUsz"  
],  
"extended\_entities.media.display\_url.keyword": [  
"pic.twitter.com/ohBYrGYUsz"  
],  
"extended\_entities.media.expanded\_url": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://twitter.com/andreaad76/status/1617421619580014592/photo/1)"  
],  
"extended\_entities.media.expanded\_url.keyword": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://twitter.com/andreaad76/status/1617421619580014592/photo/1)"  
],  
"extended\_entities.media.id": [  
1617421607089094700  
],  
"extended\_entities.media.id\_str": [  
"1617421607089094656"  
],  
"extended\_entities.media.id\_str.keyword": [  
"1617421607089094656"  
],  
"extended\_entities.media.indices": [  
72,  
95  
],  
"extended\_entities.media.media\_url": [  
"[http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"extended\_entities.media.media\_url\_https": [  
"[https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"extended\_entities.media.media\_url\_https.keyword": [  
"[https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"extended\_entities.media.media\_url.keyword": [  
"[http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"extended\_entities.media.sizes.large.h": [  
1106  
],  
"extended\_entities.media.sizes.large.resize": [  
"fit"  
],  
"extended\_entities.media.sizes.large.resize.keyword": [  
"fit"  
],  
"extended\_entities.media.sizes.large.w": [  
1242  
],  
"extended\_entities.media.sizes.medium.h": [  
1069  
],  
"extended\_entities.media.sizes.medium.resize": [  
"fit"  
],  
"extended\_entities.media.sizes.medium.resize.keyword": [  
"fit"  
],  
"extended\_entities.media.sizes.medium.w": [  
1200  
],  
"extended\_entities.media.sizes.small.h": [  
606  
],  
"extended\_entities.media.sizes.small.resize": [  
"fit"  
],  
"extended\_entities.media.sizes.small.resize.keyword": [  
"fit"  
],  
"extended\_entities.media.sizes.small.w": [  
680  
],  
"extended\_entities.media.sizes.thumb.h": [  
150  
],  
"extended\_entities.media.sizes.thumb.resize": [  
"crop"  
],  
"extended\_entities.media.sizes.thumb.resize.keyword": [  
"crop"  
],  
"extended\_entities.media.sizes.thumb.w": [  
150  
],  
"extended\_entities.media.source\_status\_id": [  
1617421619580014600  
],  
"extended\_entities.media.source\_status\_id\_str": [  
"1617421619580014592"  
],  
"extended\_entities.media.source\_status\_id\_str.keyword": [  
"1617421619580014592"  
],  
"extended\_entities.media.source\_user\_id": [  
35801196  
],  
"extended\_entities.media.source\_user\_id\_str": [  
"35801196"  
],  
"extended\_entities.media.source\_user\_id\_str.keyword": [  
"35801196"  
],  
"extended\_entities.media.type": [  
"photo"  
],  
"extended\_entities.media.type.keyword": [  
"photo"  
],  
"extended\_entities.media.url": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"extended\_entities.media.url.keyword": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"favorite\_count": [  
0  
],  
"favorited": [  
false  
],  
"filter\_level": [  
"low"  
],  
"filter\_level.keyword": [  
"low"  
],  
"id": [  
1620271231269478400  
],  
"id\_str": [  
"1620271231269478400"  
],  
"id\_str.keyword": [  
"1620271231269478400"  
],  
"is\_quote\_status": [  
false  
],  
"lang": [  
"pt"  
],  
"lang.keyword": [  
"pt"  
],  
"possibly\_sensitive": [  
false  
],  
"quote\_count": [  
0  
],  
"reply\_count": [  
0  
],  
"retweet\_count": [  
0  
],  
"retweeted": [  
false  
],  
"retweeted\_status.created\_at": [  
"Mon Jan 23 07:18:59 +0000 2023"  
],  
"retweeted\_status.created\_at.keyword": [  
"Mon Jan 23 07:18:59 +0000 2023"  
],  
"retweeted\_status.display\_text\_range": [  
0,  
55  
],  
"retweeted\_status.entities.media.display\_url": [  
"pic.twitter.com/ohBYrGYUsz"  
],  
"retweeted\_status.entities.media.display\_url.keyword": [  
"pic.twitter.com/ohBYrGYUsz"  
],  
"retweeted\_status.entities.media.expanded\_url": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://twitter.com/andreaad76/status/1617421619580014592/photo/1)"  
],  
"retweeted\_status.entities.media.expanded\_url.keyword": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://twitter.com/andreaad76/status/1617421619580014592/photo/1)"  
],  
"retweeted\_status.entities.media.id": [  
1617421607089094700  
],  
"retweeted\_status.entities.media.id\_str": [  
"1617421607089094656"  
],  
"retweeted\_status.entities.media.id\_str.keyword": [  
"1617421607089094656"  
],  
"retweeted\_status.entities.media.indices": [  
56,  
79  
],  
"retweeted\_status.entities.media.media\_url": [  
"[http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"retweeted\_status.entities.media.media\_url\_https": [  
"[https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"retweeted\_status.entities.media.media\_url\_https.keyword": [  
"[https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"retweeted\_status.entities.media.media\_url.keyword": [  
"[http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"retweeted\_status.entities.media.sizes.large.h": [  
1106  
],  
"retweeted\_status.entities.media.sizes.large.resize": [  
"fit"  
],  
"retweeted\_status.entities.media.sizes.large.resize.keyword": [  
"fit"  
],  
"retweeted\_status.entities.media.sizes.large.w": [  
1242  
],  
"retweeted\_status.entities.media.sizes.medium.h": [  
1069  
],  
"retweeted\_status.entities.media.sizes.medium.resize": [  
"fit"  
],  
"retweeted\_status.entities.media.sizes.medium.resize.keyword": [  
"fit"  
],  
"retweeted\_status.entities.media.sizes.medium.w": [  
1200  
],  
"retweeted\_status.entities.media.sizes.small.h": [  
606  
],  
"retweeted\_status.entities.media.sizes.small.resize": [  
"fit"  
],  
"retweeted\_status.entities.media.sizes.small.resize.keyword": [  
"fit"  
],  
"retweeted\_status.entities.media.sizes.small.w": [  
680  
],  
"retweeted\_status.entities.media.sizes.thumb.h": [  
150  
],  
"retweeted\_status.entities.media.sizes.thumb.resize": [  
"crop"  
],  
"retweeted\_status.entities.media.sizes.thumb.resize.keyword": [  
"crop"  
],  
"retweeted\_status.entities.media.sizes.thumb.w": [  
150  
],  
"retweeted\_status.entities.media.type": [  
"photo"  
],  
"retweeted\_status.entities.media.type.keyword": [  
"photo"  
],  
"retweeted\_status.entities.media.url": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"retweeted\_status.entities.media.url.keyword": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"retweeted\_status.extended\_entities.media.display\_url": [  
"pic.twitter.com/ohBYrGYUsz"  
],  
"retweeted\_status.extended\_entities.media.display\_url.keyword": [  
"pic.twitter.com/ohBYrGYUsz"  
],  
"retweeted\_status.extended\_entities.media.expanded\_url": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://twitter.com/andreaad76/status/1617421619580014592/photo/1)"  
],  
"retweeted\_status.extended\_entities.media.expanded\_url.keyword": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://twitter.com/andreaad76/status/1617421619580014592/photo/1)"  
],  
"retweeted\_status.extended\_entities.media.id": [  
1617421607089094700  
],  
"retweeted\_status.extended\_entities.media.id\_str": [  
"1617421607089094656"  
],  
"retweeted\_status.extended\_entities.media.id\_str.keyword": [  
"1617421607089094656"  
],  
"retweeted\_status.extended\_entities.media.indices": [  
56,  
79  
],  
"retweeted\_status.extended\_entities.media.media\_url": [  
"[http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"retweeted\_status.extended\_entities.media.media\_url\_https": [  
"[https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"retweeted\_status.extended\_entities.media.media\_url\_https.keyword": [  
"[https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](https://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"retweeted\_status.extended\_entities.media.media\_url.keyword": [  
"[http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg](http://pbs.twimg.com/media/FnI8YdaWQAAr0at.jpg)"  
],  
"retweeted\_status.extended\_entities.media.sizes.large.h": [  
1106  
],  
"retweeted\_status.extended\_entities.media.sizes.large.resize": [  
"fit"  
],  
"retweeted\_status.extended\_entities.media.sizes.large.resize.keyword": [  
"fit"  
],  
"retweeted\_status.extended\_entities.media.sizes.large.w": [  
1242  
],  
"retweeted\_status.extended\_entities.media.sizes.medium.h": [  
1069  
],  
"retweeted\_status.extended\_entities.media.sizes.medium.resize": [  
"fit"  
],  
"retweeted\_status.extended\_entities.media.sizes.medium.resize.keyword": [  
"fit"  
],  
"retweeted\_status.extended\_entities.media.sizes.medium.w": [  
1200  
],  
"retweeted\_status.extended\_entities.media.sizes.small.h": [  
606  
],  
"retweeted\_status.extended\_entities.media.sizes.small.resize": [  
"fit"  
],  
"retweeted\_status.extended\_entities.media.sizes.small.resize.keyword": [  
"fit"  
],  
"retweeted\_status.extended\_entities.media.sizes.small.w": [  
680  
],  
"retweeted\_status.extended\_entities.media.sizes.thumb.h": [  
150  
],  
"retweeted\_status.extended\_entities.media.sizes.thumb.resize": [  
"crop"  
],  
"retweeted\_status.extended\_entities.media.sizes.thumb.resize.keyword": [  
"crop"  
],  
"retweeted\_status.extended\_entities.media.sizes.thumb.w": [  
150  
],  
"retweeted\_status.extended\_entities.media.type": [  
"photo"  
],  
"retweeted\_status.extended\_entities.media.type.keyword": [  
"photo"  
],  
"retweeted\_status.extended\_entities.media.url": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"retweeted\_status.extended\_entities.media.url.keyword": [  
"[https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"retweeted\_status.favorite\_count": [  
1408  
],  
"retweeted\_status.favorited": [  
false  
],  
"retweeted\_status.filter\_level": [  
"low"  
],  
"retweeted\_status.filter\_level.keyword": [  
"low"  
],  
"retweeted\_status.id": [  
1617421619580014600  
],  
"retweeted\_status.id\_str": [  
"1617421619580014592"  
],  
"retweeted\_status.id\_str.keyword": [  
"1617421619580014592"  
],  
"retweeted\_status.is\_quote\_status": [  
false  
],  
"retweeted\_status.lang": [  
"pt"  
],  
"retweeted\_status.lang.keyword": [  
"pt"  
],  
"retweeted\_status.possibly\_sensitive": [  
false  
],  
"retweeted\_status.quote\_count": [  
19  
],  
"retweeted\_status.reply\_count": [  
83  
],  
"retweeted\_status.retweet\_count": [  
321  
],  
"retweeted\_status.retweeted": [  
false  
],  
"retweeted\_status.source": [  
"\<a href="[http://twitter.com/download/iphone\](http://twitter.com/download/iphone%5C)" rel="nofollow"\>Twitter for iPhone"  
],  
"retweeted\_status.source.keyword": [  
"\<a href="[http://twitter.com/download/iphone\](http://twitter.com/download/iphone%5C)" rel="nofollow"\>Twitter for iPhone"  
],  
"retweeted\_status.text": [  
"BOLSONARO GENOCIDA \nBOLSONARO NA CADEIA\nBOLSONARO PRESO [https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"retweeted\_status.text.keyword": [  
"BOLSONARO GENOCIDA \nBOLSONARO NA CADEIA\nBOLSONARO PRESO [https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"retweeted\_status.truncated": [  
false  
],  
"retweeted\_status.user.contributors\_enabled": [  
false  
],  
"retweeted\_status.user.created\_at": [  
"Mon Apr 27 17:29:21 +0000 2009"  
],  
"retweeted\_status.user.created\_at.keyword": [  
"Mon Apr 27 17:29:21 +0000 2009"  
],  
"retweeted\_status.user.default\_profile": [  
false  
],  
"retweeted\_status.user.default\_profile\_image": [  
false  
],  
"retweeted\_status.user.description": [  
"VIVER É UM ATO POLÍTICO​:anatomical\_heart:”A massa não é apenas objeto da ação revolucionária;é sobretudo sujeito.”Rosa Luxemburgo 🔺de BH 🖤 Sou GALO​:zap:Filha de Oyá 🚫DM 🚫Porn"  
],  
"retweeted\_status.user.description.keyword": [  
"VIVER É UM ATO POLÍTICO​:anatomical\_heart:”A massa não é apenas objeto da ação revolucionária;é sobretudo sujeito.”Rosa Luxemburgo 🔺de BH 🖤 Sou GALO​:zap:Filha de Oyá 🚫DM 🚫Porn"  
],  
"retweeted\_status.user.favourites\_count": [  
44317  
],  
"retweeted\_status.user.followers\_count": [  
16527  
],  
"retweeted\_status.user.friends\_count": [  
13553  
],  
"retweeted\_status.user.geo\_enabled": [  
true  
],  
"retweeted\_status.user.id": [  
35801196  
],  
"retweeted\_status.user.id\_str": [  
"35801196"  
],  
"retweeted\_status.user.id\_str.keyword": [  
"35801196"  
],  
"retweeted\_status.user.is\_translator": [  
false  
],  
"retweeted\_status.user.listed\_count": [  
11  
],  
"retweeted\_status.user.location": [  
"Curitiba, PR, Brasil"  
],  
"retweeted\_status.user.location.keyword": [  
"Curitiba, PR, Brasil"  
],  
"retweeted\_status.user.name": [  
"Andrea Bella de Oyá e de Vênus​:small\_red\_triangle:♀☭:zap:✊🏼🚩❤️‍🔥♉"  
],  
"retweeted\_status.user.name.keyword": [  
"Andrea Bella de Oyá e de Vênus​:small\_red\_triangle:♀☭:zap:✊🏼🚩❤️‍🔥♉"  
],  
"retweeted\_status.user.profile\_background\_color": [  
"CF5F7B"  
],  
"retweeted\_status.user.profile\_background\_color.keyword": [  
"CF5F7B"  
],  
"retweeted\_status.user.profile\_background\_image\_url": [  
"[http://abs.twimg.com/images/themes/theme7/bg.gif](http://abs.twimg.com/images/themes/theme7/bg.gif)"  
],  
"retweeted\_status.user.profile\_background\_image\_url\_https": [  
"[https://abs.twimg.com/images/themes/theme7/bg.gif](https://abs.twimg.com/images/themes/theme7/bg.gif)"  
],  
"retweeted\_status.user.profile\_background\_image\_url\_https.keyword": [  
"[https://abs.twimg.com/images/themes/theme7/bg.gif](https://abs.twimg.com/images/themes/theme7/bg.gif)"  
],  
"retweeted\_status.user.profile\_background\_image\_url.keyword": [  
"[http://abs.twimg.com/images/themes/theme7/bg.gif](http://abs.twimg.com/images/themes/theme7/bg.gif)"  
],  
"retweeted\_status.user.profile\_background\_tile": [  
true  
],  
"retweeted\_status.user.profile\_banner\_url": [  
"[https://pbs.twimg.com/profile\_banners/35801196/1674688206](https://pbs.twimg.com/profile_banners/35801196/1674688206)"  
],  
"retweeted\_status.user.profile\_banner\_url.keyword": [  
"[https://pbs.twimg.com/profile\_banners/35801196/1674688206](https://pbs.twimg.com/profile_banners/35801196/1674688206)"  
],  
"retweeted\_status.user.profile\_image\_url": [  
"[http://pbs.twimg.com/profile\_images/1616583126800547841/vO8qOoUu\_normal.jpg](http://pbs.twimg.com/profile_images/1616583126800547841/vO8qOoUu_normal.jpg)"  
],  
"retweeted\_status.user.profile\_image\_url\_https": [  
"[https://pbs.twimg.com/profile\_images/1616583126800547841/vO8qOoUu\_normal.jpg](https://pbs.twimg.com/profile_images/1616583126800547841/vO8qOoUu_normal.jpg)"  
],  
"retweeted\_status.user.profile\_image\_url\_https.keyword": [  
"[https://pbs.twimg.com/profile\_images/1616583126800547841/vO8qOoUu\_normal.jpg](https://pbs.twimg.com/profile_images/1616583126800547841/vO8qOoUu_normal.jpg)"  
],  
"retweeted\_status.user.profile\_image\_url.keyword": [  
"[http://pbs.twimg.com/profile\_images/1616583126800547841/vO8qOoUu\_normal.jpg](http://pbs.twimg.com/profile_images/1616583126800547841/vO8qOoUu_normal.jpg)"  
],  
"retweeted\_status.user.profile\_link\_color": [  
"990000"  
],  
"retweeted\_status.user.profile\_link\_color.keyword": [  
"990000"  
],  
"retweeted\_status.user.profile\_sidebar\_border\_color": [  
"D141D1"  
],  
"retweeted\_status.user.profile\_sidebar\_border\_color.keyword": [  
"D141D1"  
],  
"retweeted\_status.user.profile\_sidebar\_fill\_color": [  
"9E839E"  
],  
"retweeted\_status.user.profile\_sidebar\_fill\_color.keyword": [  
"9E839E"  
],  
"retweeted\_status.user.profile\_text\_color": [  
"4A104A"  
],  
"retweeted\_status.user.profile\_text\_color.keyword": [  
"4A104A"  
],  
"retweeted\_status.user.profile\_use\_background\_image": [  
true  
],  
"retweeted\_status.user.protected": [  
false  
],  
"retweeted\_status.user.screen\_name": [  
"andreaad76"  
],  
"retweeted\_status.user.screen\_name.keyword": [  
"andreaad76"  
],  
"retweeted\_status.user.statuses\_count": [  
21472  
],  
"retweeted\_status.user.translator\_type": [  
"none"  
],  
"retweeted\_status.user.translator\_type.keyword": [  
"none"  
],  
"retweeted\_status.user.url": [  
"[https://twitter.com/i/status/1586918733238263808](https://twitter.com/i/status/1586918733238263808)"  
],  
"retweeted\_status.user.url.keyword": [  
"[https://twitter.com/i/status/1586918733238263808](https://twitter.com/i/status/1586918733238263808)"  
],  
"retweeted\_status.user.verified": [  
false  
],  
"retweeted\_status.user.verified\_type": [  
"blue"  
],  
"retweeted\_status.user.verified\_type.keyword": [  
"blue"  
],  
"source": [  
"\<a href="[https://mobile.twitter.com](https://mobile.twitter.com)" rel="nofollow"\>Twitter Web App"  
],  
"source.keyword": [  
"\<a href="[https://mobile.twitter.com](https://mobile.twitter.com)" rel="nofollow"\>Twitter Web App"  
],  
"tags": [  
"twitter"  
],  
"tags.keyword": [  
"twitter"  
],  
"text": [  
"RT @andreaad76: BOLSONARO GENOCIDA \nBOLSONARO NA CADEIA\nBOLSONARO PRESO [https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"text.keyword": [  
"RT @andreaad76: BOLSONARO GENOCIDA \nBOLSONARO NA CADEIA\nBOLSONARO PRESO [https://twitter.com/andreaad76/status/1617421619580014592/photo/1](https://t.co/ohBYrGYUsz)"  
],  
"timestamp\_ms": [  
"1675137739375"  
],  
"timestamp\_ms.keyword": [  
"1675137739375"  
],  
"truncated": [  
false  
],  
"user.contributors\_enabled": [  
false  
],  
"user.created\_at": [  
"Tue Sep 19 17:02:07 +0000 2017"  
],  
"user.created\_at.keyword": [  
"Tue Sep 19 17:02:07 +0000 2017"  
],  
"user.default\_profile": [  
true  
],  
"user.default\_profile\_image": [  
false  
],  
"user.description": [  
"Games Entretenimento Cinema Geek Diversão Humor"  
],  
"user.description.keyword": [  
"Games Entretenimento Cinema Geek Diversão Humor"  
],  
"user.favourites\_count": [  
2114  
],  
"user.followers\_count": [  
15  
],  
"user.friends\_count": [  
208  
],  
"user.geo\_enabled": [  
false  
],  
"user.id": [  
910187258388648000  
],  
"user.id\_str": [  
"910187258388647936"  
],  
"user.id\_str.keyword": [  
"910187258388647936"  
],  
"user.is\_translator": [  
false  
],  
"user.listed\_count": [  
0  
],  
"user.location": [  
"Rio de Janeiro, Brasil"  
],  
"user.location.keyword": [  
"Rio de Janeiro, Brasil"  
],  
"user.name": [  
"DigCorreia"  
],  
"user.name.keyword": [  
"DigCorreia"  
],  
"user.profile\_background\_color": [  
"F5F8FA"  
],  
"user.profile\_background\_color.keyword": [  
"F5F8FA"  
],  
"user.profile\_background\_image\_url": [  
""  
],  
"user.profile\_background\_image\_url\_https": [  
""  
],  
"user.profile\_background\_image\_url\_https.keyword": [  
""  
],  
"user.profile\_background\_image\_url.keyword": [  
""  
],  
"user.profile\_background\_tile": [  
false  
],  
"user.profile\_banner\_url": [  
"[https://pbs.twimg.com/profile\_banners/910187258388647936/1505951259](https://pbs.twimg.com/profile_banners/910187258388647936/1505951259)"  
],  
"user.profile\_banner\_url.keyword": [  
"[https://pbs.twimg.com/profile\_banners/910187258388647936/1505951259](https://pbs.twimg.com/profile_banners/910187258388647936/1505951259)"  
],  
"user.profile\_image\_url": [  
"[http://pbs.twimg.com/profile\_images/1161447549926019078/HF0THyNh\_normal.jpg](http://pbs.twimg.com/profile_images/1161447549926019078/HF0THyNh_normal.jpg)"  
],  
"user.profile\_image\_url\_https": [  
"[https://pbs.twimg.com/profile\_images/1161447549926019078/HF0THyNh\_normal.jpg](https://pbs.twimg.com/profile_images/1161447549926019078/HF0THyNh_normal.jpg)"  
],  
"user.profile\_image\_url\_https.keyword": [  
"[https://pbs.twimg.com/profile\_images/1161447549926019078/HF0THyNh\_normal.jpg](https://pbs.twimg.com/profile_images/1161447549926019078/HF0THyNh_normal.jpg)"  
],  
"user.profile\_image\_url.keyword": [  
"[http://pbs.twimg.com/profile\_images/1161447549926019078/HF0THyNh\_normal.jpg](http://pbs.twimg.com/profile_images/1161447549926019078/HF0THyNh_normal.jpg)"  
],  
"user.profile\_link\_color": [  
"1DA1F2"  
],  
"user.profile\_link\_color.keyword": [  
"1DA1F2"  
],  
"user.profile\_sidebar\_border\_color": [  
"C0DEED"  
],  
"user.profile\_sidebar\_border\_color.keyword": [  
"C0DEED"  
],  
"user.profile\_sidebar\_fill\_color": [  
"DDEEF6"  
],  
"user.profile\_sidebar\_fill\_color.keyword": [  
"DDEEF6"  
],  
"user.profile\_text\_color": [  
"333333"  
],  
"user.profile\_text\_color.keyword": [  
"333333"  
],  
"user.profile\_use\_background\_image": [  
true  
],  
"user.protected": [  
false  
],  
"user.screen\_name": [  
"DigCorreia"  
],  
"user.screen\_name.keyword": [  
"DigCorreia"  
],  
"user.statuses\_count": [  
1057  
],  
"user.translator\_type": [  
"none"  
],  
"user.translator\_type.keyword": [  
"none"  
],  
"user.verified": [  
false  
],  
"user.verified\_type": [  
"blue"  
],  
"user.verified\_type.keyword": [  
"blue"  
],  
"\_id": "87P9BYYBzTlWHRjvfmrL",  
"\_index": "twitter\_2",  
"\_score": null  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 31, 2023, 4:47am UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/2 "2023-01-31T04:47:30Z")

</div>

> [@xalmer](#):
>
> So im trying to remove the unwanted fields with mutate {remove\_field) filter. But its not working.

In JSON a field can be contain other fields (e.g. `{"a": {"x":1, "y":2}}` or just contain data `{"a": 7}`.

I'll give you 10 to 1 that elasticsearch, logstash, and kibana were developed by different teams and they realised that combining them was a huge win.

However, the logstash configuration language treats [a][b] as an [b] field within an [a] object, and [a.x] as a field with a stop in it. Kibana and Elasticsearch treat [a.b] as an [b] field inside an [a] object. It just does not parse it the same way.  
Elasticsearch did stop allowing stops in field names until they improved the disambiguation engine.

What Kibana and Elasticsearch call user.verified Logstash would call

```
[user][verified]

```

Note that you can blow away the whole of [user] using a single entry in the remove\_field option if you do not want any of it.

---

<div class="post-metadata">

**Author:** ![xalmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xalmer/32/117685_2.png) [@xalmer](https://discuss.elastic.co/u/xalmer)\
**Post date:** [January 31, 2023, 4:57am UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/3 "2023-01-31T04:57:20Z")

</div>

Hello Badger,

Thanks for explanation! But just to undestand...  
Kibana and Elasticsearch understand: entities.media.additional\_media\_info.monetizable  
Logstash understand: [entites][media][additional\_media\_info][monetizable]

And the right filter syntax is:

mutate {  
remove\_field =\> [  
"display\_text\_range",  
"[entities][media][additional\_media\_info][monetizable]",  
"[entities][media][id]" ]  
}

---

<div class="post-metadata">

**Author:** ![xalmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xalmer/32/117685_2.png) [@xalmer](https://discuss.elastic.co/u/xalmer)\
**Post date:** [January 31, 2023, 4:48pm UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/4 "2023-01-31T16:48:25Z")

</div>

I try to use:

**Option 01:**

filter { if "twitter" in [tags]{  
mutate {  
remove\_field =\> ["[entities][media][additional\_media\_info]", "entities.media.additional\_media\_info.monetizable" , "[entities][media][additional\_media\_info][monetizable]" ]  
}  
}  
}

**Option 2:**

filter {  
mutate {  
remove\_field =\> ["[entities][media][additional\_media\_info]", "entities.media.additional\_media\_info.monetizable" , "[entities][media][additional\_media\_info][monetizable]" ]  
}  
}

But none of them works. The field always appears.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 31, 2023, 4:55pm UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/5 "2023-01-31T16:55:58Z")

</div>

> [@xalmer](#):
>
> But none of them works. The field always appears.

Can you share the json of the document? Maybe there is an array on one of the levels, but to confirm it you need to share the json of the document.

---

<div class="post-metadata">

**Author:** ![xalmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xalmer/32/117685_2.png) [@xalmer](https://discuss.elastic.co/u/xalmer)\
**Post date:** [January 31, 2023, 7:23pm UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/6 "2023-01-31T19:23:51Z")

</div>

JSON example

```auto
{
    "lang": "pt",
    "is_quote_status": false,
    "tags": [
        "twitter"
    ],
    "timestamp_ms": "1675191730741",
    "reply_count": 0,
    "truncated": false,
    "favorite_count": 0,
    "entities": {
        "symbols": [],
        "media": [
            {
                "source_user_id_str": "3991142595",
                "display_url": "pic.twitter.com/oX9zyz4bKo",
                "type": "photo",
                "source_status_id": 1620230684739764225,
                "id_str": "1620230600165580802",
                "sizes": {
                    "large": {
                        "w": 1080,
                        "resize": "fit",
                        "h": 1080
                    },
                    "medium": {
                        "w": 1080,
                        "resize": "fit",
                        "h": 1080
                    },
                    "small": {
                        "w": 680,
                        "resize": "fit",
                        "h": 680
                    },
                    "thumb": {
                        "w": 150,
                        "resize": "crop",
                        "h": 150
                    }
                },
                "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                "url": "https://t.co/oX9zyz4bKo",
                "indices": [
                    112,
                    135
                ],
                "source_status_id_str": "1620230684739764225",
                "source_user_id": 3991142595,
                "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                "expanded_url": "https://twitter.com/sergiorvitoria/status/1620230684739764225/video/1",
                "id": 1620230600165580802,
                "additional_media_info": {
                    "monetizable": false
                }
            }
        ],
        "hashtags": [],
        "urls": [],
        "user_mentions": [
            {
                "id_str": "3991142595",
                "indices": [
                    3,
                    18
                ],
                "screen_name": "sergiorvitoria",
                "name": "Sérgio Vitória",
                "id": 3991142595
            },
            {
                "id_str": "1523748277",
                "indices": [
                    20,
                    36
                ],
                "screen_name": "conexaopolitica",
                "name": "Conexão Política",
                "id": 1523748277
            },
            {
                "id_str": "1587148141001932804",
                "indices": [
                    37,
                    51
                ],
                "screen_name": "CarlineNagano",
                "name": "Carline Nagano 🇧🇷🌻💚💛",
                "id": 1587148141001932804
            }
        ]
    },
    "possibly_sensitive": false,
    "coordinates": null,
    "geo": null,
    "place": null,
    "in_reply_to_user_id_str": null,
    "source": "<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>",
    "in_reply_to_user_id": null,
    "retweeted_status": {
        "lang": "pt",
        "display_text_range": [
            32,
            91
        ],
        "is_quote_status": false,
        "reply_count": 2,
        "truncated": false,
        "favorite_count": 62,
        "entities": {
            "symbols": [],
            "media": [
                {
                    "display_url": "pic.twitter.com/oX9zyz4bKo",
                    "type": "photo",
                    "id_str": "1620230600165580802",
                    "sizes": {
                        "large": {
                            "w": 1080,
                            "resize": "fit",
                            "h": 1080
                        },
                        "medium": {
                            "w": 1080,
                            "resize": "fit",
                            "h": 1080
                        },
                        "small": {
                            "w": 680,
                            "resize": "fit",
                            "h": 680
                        },
                        "thumb": {
                            "w": 150,
                            "resize": "crop",
                            "h": 150
                        }
                    },
                    "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                    "url": "https://t.co/oX9zyz4bKo",
                    "indices": [
                        92,
                        115
                    ],
                    "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                    "expanded_url": "https://twitter.com/sergiorvitoria/status/1620230684739764225/video/1",
                    "id": 1620230600165580802,
                    "additional_media_info": {
                        "monetizable": false
                    }
                }
            ],
            "hashtags": [],
            "urls": [],
            "user_mentions": [
                {
                    "id_str": "1523748277",
                    "indices": [
                        0,
                        16
                    ],
                    "screen_name": "conexaopolitica",
                    "name": "Conexão Política",
                    "id": 1523748277
                },
                {
                    "id_str": "1587148141001932804",
                    "indices": [
                        17,
                        31
                    ],
                    "screen_name": "CarlineNagano",
                    "name": "Carline Nagano 🇧🇷🌻💚💛",
                    "id": 1587148141001932804
                }
            ]
        },
        "possibly_sensitive": false,
        "coordinates": null,
        "geo": null,
        "place": null,
        "in_reply_to_user_id_str": "1523748277",
        "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
        "in_reply_to_user_id": 1523748277,
        "retweeted": false,
        "created_at": "Tue Jan 31 01:21:12 +0000 2023",
        "id_str": "1620230684739764225",
        "retweet_count": 15,
        "in_reply_to_status_id": 1620175660865163264,
        "extended_entities": {
            "media": [
                {
                    "display_url": "pic.twitter.com/oX9zyz4bKo",
                    "type": "video",
                    "id_str": "1620230600165580802",
                    "sizes": {
                        "large": {
                            "w": 1080,
                            "resize": "fit",
                            "h": 1080
                        },
                        "medium": {
                            "w": 1080,
                            "resize": "fit",
                            "h": 1080
                        },
                        "small": {
                            "w": 680,
                            "resize": "fit",
                            "h": 680
                        },
                        "thumb": {
                            "w": 150,
                            "resize": "crop",
                            "h": 150
                        }
                    },
                    "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                    "url": "https://t.co/oX9zyz4bKo",
                    "indices": [
                        92,
                        115
                    ],
                    "video_info": {
                        "duration_millis": 35035,
                        "variants": [
                            {
                                "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/540x540/CdT6qOtm5mqqeUCr.mp4?tag=12",
                                "bitrate": 832000,
                                "content_type": "video/mp4"
                            },
                            {
                                "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/720x720/KDXJ0YCDpW5ggJqV.mp4?tag=12",
                                "bitrate": 1280000,
                                "content_type": "video/mp4"
                            },
                            {
                                "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/pl/F0TiM3lc0qKfgOHN.m3u8?tag=12&container=fmp4",
                                "content_type": "application/x-mpegURL"
                            },
                            {
                                "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/320x320/nFRwoSMqjhdtu2ru.mp4?tag=12",
                                "bitrate": 432000,
                                "content_type": "video/mp4"
                            }
                        ],
                        "aspect_ratio": [
                            1,
                            1
                        ]
                    },
                    "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                    "expanded_url": "https://twitter.com/sergiorvitoria/status/1620230684739764225/video/1",
                    "id": 1620230600165580802,
                    "additional_media_info": {
                        "monetizable": false
                    }
                }
            ]
        },
        "in_reply_to_screen_name": "conexaopolitica",
        "text": "@conexaopolitica @CarlineNagano Pacheco representa a continuação do desrespeito ditatorial! https://t.co/oX9zyz4bKo",
        "quote_count": 2,
        "filter_level": "low",
        "user": {
            "profile_background_color": "C0DEED",
            "profile_sidebar_border_color": "C0DEED",
            "profile_image_url_https": "https://pbs.twimg.com/profile_images/1588327555345666053/H_sPO9qS_normal.jpg",
            "verified": false,
            "profile_banner_url": "https://pbs.twimg.com/profile_banners/3991142595/1594481781",
            "default_profile_image": false,
            "notifications": null,
            "is_translator": false,
            "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
            "withheld_in_countries": [],
            "friends_count": 31129,
            "statuses_count": 62042,
            "name": "Sérgio Vitória",
            "listed_count": 20,
            "profile_text_color": "333333",
            "followers_count": 54724,
            "following": null,
            "location": "Porto Alegre, Brasil",
            "time_zone": null,
            "profile_use_background_image": true,
            "favourites_count": 79993,
            "created_at": "Mon Oct 19 01:58:02 +0000 2015",
            "default_profile": true,
            "description": "Arquiteto 🇧🇷 \nA favor da liberdade de expressão e contra a bandidagem!\nO Brasil é uma Republiqueta de Bananas!",
            "id_str": "3991142595",
            "verified_type": "blue",
            "utc_offset": null,
            "profile_image_url": "http://pbs.twimg.com/profile_images/1588327555345666053/H_sPO9qS_normal.jpg",
            "follow_request_sent": null,
            "translator_type": "none",
            "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
            "url": null,
            "protected": false,
            "profile_background_tile": false,
            "contributors_enabled": false,
            "profile_sidebar_fill_color": "DDEEF6",
            "geo_enabled": true,
            "id": 3991142595,
            "screen_name": "sergiorvitoria",
            "lang": null,
            "profile_link_color": "1DA1F2"
        },
        "id": 1620230684739764225,
        "contributors": null,
        "favorited": false,
        "in_reply_to_status_id_str": "1620175660865163264"
    },
    "retweeted": false,
    "created_at": "Tue Jan 31 19:02:10 +0000 2023",
    "type": "tweets",
    "id_str": "1620497687471882240",
    "retweet_count": 0,
    "in_reply_to_status_id": null,
    "extended_entities": {
        "media": [
            {
                "source_user_id_str": "3991142595",
                "display_url": "pic.twitter.com/oX9zyz4bKo",
                "type": "video",
                "source_status_id": 1620230684739764225,
                "id_str": "1620230600165580802",
                "sizes": {
                    "large": {
                        "w": 1080,
                        "resize": "fit",
                        "h": 1080
                    },
                    "medium": {
                        "w": 1080,
                        "resize": "fit",
                        "h": 1080
                    },
                    "small": {
                        "w": 680,
                        "resize": "fit",
                        "h": 680
                    },
                    "thumb": {
                        "w": 150,
                        "resize": "crop",
                        "h": 150
                    }
                },
                "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                "url": "https://t.co/oX9zyz4bKo",
                "indices": [
                    112,
                    135
                ],
                "source_status_id_str": "1620230684739764225",
                "source_user_id": 3991142595,
                "video_info": {
                    "duration_millis": 35035,
                    "variants": [
                        {
                            "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/540x540/CdT6qOtm5mqqeUCr.mp4?tag=12",
                            "bitrate": 832000,
                            "content_type": "video/mp4"
                        },
                        {
                            "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/720x720/KDXJ0YCDpW5ggJqV.mp4?tag=12",
                            "bitrate": 1280000,
                            "content_type": "video/mp4"
                        },
                        {
                            "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/pl/F0TiM3lc0qKfgOHN.m3u8?tag=12&container=fmp4",
                            "content_type": "application/x-mpegURL"
                        },
                        {
                            "url": "https://video.twimg.com/ext_tw_video/1620230600165580802/pu/vid/320x320/nFRwoSMqjhdtu2ru.mp4?tag=12",
                            "bitrate": 432000,
                            "content_type": "video/mp4"
                        }
                    ],
                    "aspect_ratio": [
                        1,
                        1
                    ]
                },
                "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/1620230600165580802/pu/img/doCcUOoOvSCOfixC.jpg",
                "expanded_url": "https://twitter.com/sergiorvitoria/status/1620230684739764225/video/1",
                "id": 1620230600165580802,
                "additional_media_info": {
                    "monetizable": false
                }
            }
        ]
    },
    "@timestamp": "2023-01-31T19:02:10.000Z",
    "in_reply_to_screen_name": null,
    "text": "RT @sergiorvitoria: @conexaopolitica @CarlineNagano Pacheco representa a continuação do desrespeito ditatorial! https://t.co/oX9zyz4bKo",
    "quote_count": 0,
    "filter_level": "low",
    "@version": "1",
    "user": {
        "profile_background_color": "F5F8FA",
        "profile_sidebar_border_color": "C0DEED",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/1618418636686856192/wmyyOJ9l_normal.jpg",
        "verified": false,
        "profile_banner_url": "https://pbs.twimg.com/profile_banners/1458812188961947649/1667018148",
        "default_profile_image": false,
        "notifications": null,
        "is_translator": false,
        "profile_background_image_url": "",
        "withheld_in_countries": [],
        "friends_count": 320,
        "statuses_count": 1672,
        "name": "EstudandoDireito",
        "listed_count": 0,
        "profile_text_color": "333333",
        "followers_count": 10,
        "following": null,
        "location": "Três corações MG",
        "time_zone": null,
        "profile_use_background_image": true,
        "favourites_count": 15,
        "created_at": "Thu Nov 11 15:03:14 +0000 2021",
        "default_profile": true,
        "description": "http://unincor.br",
        "id_str": "1458812188961947649",
        "verified_type": "blue",
        "utc_offset": null,
        "profile_image_url": "http://pbs.twimg.com/profile_images/1618418636686856192/wmyyOJ9l_normal.jpg",
        "follow_request_sent": null,
        "translator_type": "none",
        "profile_background_image_url_https": "",
        "url": "http://www.google.com",
        "protected": false,
        "profile_background_tile": false,
        "contributors_enabled": false,
        "profile_sidebar_fill_color": "DDEEF6",
        "geo_enabled": false,
        "id": 1458812188961947649,
        "screen_name": "MaxcuelS",
        "lang": null,
        "profile_link_color": "1DA1F2"
    },
    "id": 1620497687471882240,
    "contributors": null,
    "favorited": false,
    "in_reply_to_status_id_str": null
}

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [January 31, 2023, 8:15pm UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/7 "2023-01-31T20:15:04Z")

</div>

The "media" is an array, starts with [

Try with:  
`remove_field => ["[entities][media][0][additional_media_info]" ]`

---

<div class="post-metadata">

**Author:** ![xalmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xalmer/32/117685_2.png) [@xalmer](https://discuss.elastic.co/u/xalmer)\
**Post date:** [January 31, 2023, 8:25pm UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/8 "2023-01-31T20:25:44Z")

</div>

Rios,  
Thanks a lot for the insight! Now I understood!

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [January 31, 2023, 8:26pm UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/9 "2023-01-31T20:26:29Z")

</div>

Is it working? I haven't tested 😊 😇

---

<div class="post-metadata">

**Author:** ![xalmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xalmer/32/117685_2.png) [@xalmer](https://discuss.elastic.co/u/xalmer)\
**Post date:** [January 31, 2023, 8:32pm UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/10 "2023-01-31T20:32:07Z")

</div>

Works!!!! But still a question, if have more than one image, i need to put [1] [2]...? Or there some way to "auto identify" the array and remove "additional\_media\_info"?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [January 31, 2023, 8:38pm UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/11 "2023-01-31T20:38:45Z")

</div>

I think no "auto identify" except if you have 2-3 array members, you can put:  
`remove_field => ["[entities][media][0][additional_media_info]", ]"[entities][media][1][additional_media_info]", "[entities][media][2][additional_media_info]" ]`  
The mutate shouldn't report an error if field is missing.  
If you don't know max number additional\_media\_info members, then use loop in ruby.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2023, 8:39pm UTC](https://discuss.elastic.co/t/twitter-input-logstash-filter-mutate-remove-field-elasticsearch/324290/12 "2023-02-28T20:39:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.