# Removing fields from Twitter pipeline via conf file

**URL:** <https://discuss.elastic.co/t/removing-fields-from-twitter-pipeline-via-conf-file/103976>\
**Category:** Logstash\
**Created:** [October 14, 2017, 6:09am UTC](https://discuss.elastic.co/t/removing-fields-from-twitter-pipeline-via-conf-file/103976 "2017-10-14T06:09:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sharon1](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@sharon1](https://discuss.elastic.co/u/sharon1)\
**Post date:** [October 14, 2017, 6:09am UTC](https://discuss.elastic.co/t/removing-fields-from-twitter-pipeline-via-conf-file/103976/1 "2017-10-14T06:09:41Z")

</div>

Good day to you all,

Recently I installed ES on my laptop and I have been playing around with it.

At the moment I have got a Twitter pipeline running. All Twitter fields are being loaded into ES.  
I would like to remove some of the fields and tried via different settings in the conf file.

For example, how do I remove these fields from loading into ES?

"\_id" is not part of an array  
"in\_reply\_to\_status\_id\_str" is part of the array "\_source"  
"urls" is part of the arrays "\_source" and array "entities"

Can someone tell me what code to use to filter out these three fields?  
Thanks!  
Sharon

```
    "_id": "AV8DKinuBGKL_Y3fmyHR",
    "_score": 5.068332,
    "_source": {
      "in_reply_to_status_id_str": null,
      "in_reply_to_status_id": null,
      "created_at": "Mon Oct 09 22:03:35 +0000 2017",
      "in_reply_to_user_id_str": null,

      "entities": {
        "urls": [],
        "hashtags": [],
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 16, 2017, 7:56pm UTC](https://discuss.elastic.co/t/removing-fields-from-twitter-pipeline-via-conf-file/103976/2 "2017-10-16T19:56:59Z")

</div>

> "\_id" is not part of an array

It's not even part of the document. `_source` points to the contents of the event sent by Logstash (and it's not an array).

> "in\_reply\_to\_status\_id\_str" is part of the array "\_source"

Use a mutate filter's `remove_field` option to remove that field.

> "urls" is part of the arrays "\_source" and array "entities"

Use a mutate filter just like above, but keep in mind that it's a nested field so you need to use the `[entitites][urls]` notation to reference it. See [Accessing event data and fields | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references).

---

<div class="post-metadata">

**Author:** ![sharon1](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@sharon1](https://discuss.elastic.co/u/sharon1)\
**Post date:** [October 17, 2017, 6:00pm UTC](https://discuss.elastic.co/t/removing-fields-from-twitter-pipeline-via-conf-file/103976/3 "2017-10-17T18:00:14Z")

</div>

Dear Magnus,  
Thanks for your reply. Am going to try tomorrow.  
Thanks again,Sharon

> > magnusbaeck Magnus Bäck Logstash Plugins Community Maintainer  
> > October 16 |

"\_id" is not part of an array

It's not even part of the document. \_source points to the contents of the event sent by Logstash (and it's not an array).

"in\_reply\_to\_status\_id\_str" is part of the array "\_source"

Use a mutate filter's remove\_field option to remove that field.

"urls" is part of the arrays "\_source" and array "entities"

Use a mutate filter just like above, but keep in mind that it's a nested field so you need to use the [entitites][urls] notation to reference it. See [Accessing event data and fields | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references).

Visit Topic or reply to this email to respond.

To unsubscribe from these emails, click here.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 14, 2017, 6:00pm UTC](https://discuss.elastic.co/t/removing-fields-from-twitter-pipeline-via-conf-file/103976/4 "2017-11-14T18:00:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.