Removing fields from Twitter pipeline via conf file

Elastic Stack Logstash
1

Good day to you all,

Recently I installed ES on my laptop and I have been playing around with it.

At the moment I have got a Twitter pipeline running. All Twitter fields are being loaded into ES.
I would like to remove some of the fields and tried via different settings in the conf file.

For example, how do I remove these fields from loading into ES?

"_id" is not part of an array
"in_reply_to_status_id_str" is part of the array "_source"
"urls" is part of the arrays "_source" and array "entities"

Can someone tell me what code to use to filter out these three fields?
Thanks!
Sharon

    "_id": "AV8DKinuBGKL_Y3fmyHR",
    "_score": 5.068332,
    "_source": {
      "in_reply_to_status_id_str": null,
      "in_reply_to_status_id": null,
      "created_at": "Mon Oct 09 22:03:35 +0000 2017",
      "in_reply_to_user_id_str": null,

      "entities": {
        "urls": [],
        "hashtags": [],
2

"_id" is not part of an array

It's not even part of the document. _source points to the contents of the event sent by Logstash (and it's not an array).

"in_reply_to_status_id_str" is part of the array "_source"

Use a mutate filter's remove_field option to remove that field.

"urls" is part of the arrays "_source" and array "entities"

Use a mutate filter just like above, but keep in mind that it's a nested field so you need to use the [entitites][urls] notation to reference it. See Accessing event data and fields | Logstash Reference [8.11] | Elastic.

3

Dear Magnus,
Thanks for your reply. Am going to try tomorrow.
Thanks again,Sharon

magnusbaeck Magnus Bäck Logstash Plugins Community Maintainer
October 16 |

"_id" is not part of an array

It's not even part of the document. _source points to the contents of the event sent by Logstash (and it's not an array).

"in_reply_to_status_id_str" is part of the array "_source"

Use a mutate filter's remove_field option to remove that field.

"urls" is part of the arrays "_source" and array "entities"

Use a mutate filter just like above, but keep in mind that it's a nested field so you need to use the [entitites][urls] notation to reference it. See Accessing event data and fields | Logstash Reference [8.11] | Elastic.

Visit Topic or reply to this email to respond.

To unsubscribe from these emails, click here.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.