# Isolating Hashtags in Twitter Feed

**URL:** <https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532>\
**Category:** Logstash\
**Created:** [March 26, 2018, 2:37am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532 "2018-03-26T02:37:14Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [March 26, 2018, 2:37am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532/1 "2018-03-26T02:37:14Z")

</div>

Maybe my best bet is to start with my end goal and then someone can tell me if this is possible. I'd like to build Kibana visuals that show top mentions of hashtags as they appear in the `[extended_tweet][full_text]` field. It seem like the first step is to pull the 'hashtagged' words out of this field and stick them as an array into their own field. Is it then possible for visualizations to search and aggregate separate values in a field?

How would I isolate data in a text field and move it into another?  
Can Kibana visualizations aggregate and visualize values that are part of an array in a field?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 26, 2018, 2:52am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532/2 "2018-03-26T02:52:03Z")

</div>

We do this via a ruby filter;

```auto
if [message] {
  ruby {
    code => "event.set('hashtags', event.get('message').scan(/\#[a-z]*/i))"
  }
}

```

It could probably be folded back into the plugin, but this came via someone smarter than me and I am not sure on how to submit it back as code 😉

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [March 26, 2018, 2:53am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532/3 "2018-03-26T02:53:18Z")

</div>

If I turn my head sideways and squint really hard....I can almost...nope, I still can't tell wtf that code is, lol.

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [March 26, 2018, 3:05am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532/4 "2018-03-26T03:05:08Z")

</div>

Huzzah, it works! Thanks again, Mark.

```
  if [extended_tweet][full_text] {
    ruby {
      code => "event.set('hashtags', event.get('[extended_tweet][full_text]').scan(/\#[a-z]*/i))"
    }
  } else {
    ruby {
      code => "event.set('hashtags', event.get('text').scan(/\#[a-z]*/i))"
    }
  }
```

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [March 26, 2018, 3:16am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532/5 "2018-03-26T03:16:14Z")

</div>

OH OH! Bonus question!!

Say I want to do the same thing...but with hyperlinks...would this be how you configure the "scan" parameters?

`"event.set('links', event.get('[extended_tweet][full_text]').scan(/\http[a-z]*/i))"`

OOOO.....and is it possible to instruct Kibana to make them clickable hyperlinks?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 26, 2018, 3:55am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532/6 "2018-03-26T03:55:02Z")

</div>

> [@wwalker](#):
>
> Say I want to do the same thing...but with hyperlinks...would this be how you configure the "scan" parameters?

It looks ok, but I am not that good at regex or ruby.

> [@wwalker](#):
>
> OOOO.....and is it possible to instruct Kibana to make them clickable hyperlinks?

It is yeah;

 ![31](https://us1.discourse-cdn.com/elastic/original/3X/4/9/4937033b09ed248d862ba25ca12e4e4d2eb764f0.png)

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [March 26, 2018, 4:33am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532/7 "2018-03-26T04:33:53Z")

</div>

Looks to be a little more complicated...so far I've come up with:  
`code => "event.set('links', event.get('text').scan(/(https:\/\/|http:\/\/)[a-z].*/i))` but all that's being captured in the links field is `http://` or `https://`. I also don't think this will work completely as it is looking to include everything after the http://, I need a delimiter to separate multiple links.

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [March 26, 2018, 5:11am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532/8 "2018-03-26T05:11:26Z")

</div>

hmm....found an online regex builder and the builder seemed to like this but it's still only putting http:// or https:// in the field... `code => "event.set('links', event.get('[extended_tweet][full_text]').scan(/(https:\/\/|http:\/\/)\S*/i))"`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 23, 2018, 5:11am UTC](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532/9 "2018-04-23T05:11:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.