# Create arrays based on certain text in a field

**URL:** <https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811>\
**Category:** Logstash\
**Created:** [August 28, 2020, 6:57pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811 "2020-08-28T18:57:27Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [August 28, 2020, 6:57pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/1 "2020-08-28T18:57:28Z")

</div>

I am parsing chat messages and I want to create three arrays for tag cloud purposes.

One for normal chat words, one for user tags, and one for chat emotes.

For example, take this chat message:  
`WE LOVE YOU @CHANDLER, YOUR GREAT AT EVERYTHING!! <3 <3 VirtualHug VirtualHug :) :WE LOVE YOU CHANDLER, YOUR GREAT AT EVERYTHING!! [emote=<3] <3 [emote=VirtualHug] VirtualHug [emote=:)] :)`

Based on this message, I'd like to peel out the following :

- user tag(s) : **@CHANDLER**
- emotes : **[emote=\<3]**, **[emote=VirtualHug]**, **[emote=:)]**

I have tried various examples I have found on these forums but nothing seems to work and always brings all of Logstash down.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 28, 2020, 7:29pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/2 "2020-08-28T19:29:38Z")

</div>

I explained how to do that in a [reply](https://discuss.elastic.co/t/multiple-match-array/245817/2) to another of your posts. Did you have an issue with that solution? If so, what is the issue?

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [August 28, 2020, 7:47pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/3 "2020-08-28T19:47:26Z")

</div>

Can you better explain how to use this code?  
`ruby { code => 'event.set("matches", event.get("message").scan(/@\w+/))' }`

I am reading a field called : `channel_message`. I am wanting to create fields : `tag_cloud_emote`, `tag_cloud_chat`, and `tag_cloud_tags`.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 28, 2020, 8:41pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/4 "2020-08-28T20:41:15Z")

</div>

If you have two things you want to scan for I would slightly change that code:

```
    ruby {
        code => '
            msg = event.get("channel_message")
            if msg
                event.set("tag_cloud_tags", msg.scan(/@\w+/))
                event.set("tag_cloud_emote", msg.scan(/\[emote=[^]]*\]/))
            end
        '
    }

```

which will get you

```
 "tag_cloud_tags" => [
    [0] "@CHANDLER"
],
"tag_cloud_emote" => [
    [0] "[emote=<3]",
    [1] "[emote=VirtualHug]",
    [2] "[emote=:)]"
],

```

Just insert that ruby filter into the filter section of your configuration file.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [August 28, 2020, 8:57pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/5 "2020-08-28T20:57:27Z")

</div>

Thank you sir.

How do I account for just the chat words (that are not tags or emotes) ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 28, 2020, 9:07pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/6 "2020-08-28T21:07:54Z")

</div>

You could try something like

```
mutate { add_field => { "tag_cloud_chat" => "%{channel_message}" } }
mutate {
    gsub => [
        "tag_cloud_chat", "\[emote=[^]]*\]", "",
        "tag_cloud_chat", "@\w+", "",
        "tag_cloud_chat", " ", " "
    ]
}

```

Removing the extra spaces with a third gsub is just easier than trying to add spaces to the other gsubs and handling corner cases where those spaces do not exist.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [September 3, 2020, 3:08pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/7 "2020-09-03T15:08:39Z")

</div>

Badger this appears to be working great.

How can I, while in the Ruby filter, remove the `[emote=`(capture\_group)`]` from around the capture group?

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [September 3, 2020, 3:34pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/8 "2020-09-03T15:34:57Z")

</div>

Also, here is the current set of code I am using in Logstash. I had some field changes per other app requirements.

```
##########
# PARSE TAG CLOUD [CHAT]
##########
if [tag][cloud][chat] =~ /^.+$/ {
  mutate { gsub => ["[tag][cloud][chat]", "(\[emote=\S+\])", "" ] }
  mutate { gsub => ["[tag][cloud][chat]", "(@\S+)", "" ] }
  mutate { split => { "[tag][cloud][chat]" => " " } }
}

##########
# PARSE TAG CLOUD [EMOTE]
##########
if [channel][msg][text] =~ /(?i)\[emote=\S+\]/ {
  ruby {
    code => '
      msg = event.get("[channel][msg][text]")
      if msg
        event.set("[tag][cloud][emote]", msg.scan(/\[emote=[^]]*\]/))
      end
    '
  }
  if [tag][cloud][emote] =~ /(?i)\[emote=\S+\]/ {
    mutate { gsub => ["[tag][cloud][emote]", "\[emote=(\S+)\]", "\1" ] }
  }
}

##########
# PARSE TAG CLOUD [TAG]
##########
if [channel][msg][text] =~ /(?i)@\w+/ {
  ruby {
    code => '
      msg = event.get("[channel][msg][text]")
      if msg
        event.set("[tag][cloud][tag]", msg.scan(/@\w+/))
      end
    '
  }
}

```

What are you thoughts of this code?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 3, 2020, 3:46pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/9 "2020-09-03T15:46:34Z")

</div>

Change the second scan to be

```
event.set("tag_cloud_emote", msg.scan(/\[emote=([^]]*)\]/).flatten)
```

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [September 3, 2020, 3:54pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/10 "2020-09-03T15:54:53Z")

</div>

Thank you sir.

Just noticed something a little odd with the output in a Kibana visualization.

```
Tag Count 
@TimTheTatman 255
@Asmongold 214
@timthetatman 105
@DrLupo 84
@NICKMERCS 75
@nickmercs 52
@drlupo 50

```

I guess I need to have the ruby filter force all matches to lowercase? If so, how?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 3, 2020, 4:09pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/11 "2020-09-03T16:09:15Z")

</div>

You can use mutate+lowercase to do it. If a field is an array it will iterate over the members.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [September 3, 2020, 4:29pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/12 "2020-09-03T16:29:51Z")

</div>

That did the trick. Thank you again sir!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 1, 2020, 4:29pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811/13 "2020-10-01T16:29:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.