Splitting a string to json like results

Elastic Stack Logstash
1

Hey there, I have my own logs coming into the logstash that are being created from my python script.

The format for the logs is: %{DATA:Sender}|%{DATA:Recipient}|%{DATA:Subject}|(%{DATA:Links})?$ So far so good.

Problem is I am getting multiple (unknown amount of links) in the last field. I would like to split these links so the end result will look something along the lines of:

links:{
http://link1.com
http://link2.com
http://link3,com
}

I am really new to Elastic so help is greatly appreciated. I need to know how do I exactly split this field.
This is top priority for me as tomorrow evening I need to show this to my client

2

I am attempting to use the split method that should in turn split my links to different values but for some reason the links is still arriving as an unknown field and not as a string even though I am mutating it.

What is wrong here?

filter {
grok {
match => {
"message" => '%{DATA:Sender}|%{DATA:Recipient}|%{DATA:Subject}|(%{DATA:links})?$'
}
}
mutate {
convert => { "links" => "string"}
}
split{
field => "links"
terminator => "\s"
}

}

3

If you use

output { stdout { codec => rubydebug } }

what does the links field look like?

4

Hey there Badger, thank you for the quick reply.

 "Recipient" => "something@gmail.com",
   "Subject" => "Bring Your Story to Life as a Creative Writer with These Courses",
  "@version" => "1",
"@timestamp" => 2020-01-17T19:21:33.785Z,
      "host" => {
    "name" => "XXX"
},
      "tags" => [
    [0] "beats_input_codec_plain_applied"
],
     "links" => "https://stacksocial.us2.list-manage.com/track/click?u3D820c8324c8d125df5a73a0bb3&id3Df81e9884ff&e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/track/click?u3D820c8324c8d125df5a73a0bb3&id3D4ffcd003e3&e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/track/click?u3D820c8324c8d125df5a73a0bb3&id3De55bdb63ec&e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/track/click?u3D820c8324c8d125df5a73a0bb3&id3D4c652e0f68&e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/track/click?u3D820c8324c8d125df5a73a0bb3&id3D3c72e77f27&e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/track/click?u3D820c8324c8d125df5a73a0bb3&id3D4a02f80752&e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/track/click?u3D820c8324c8d125df5a73a0bb3&id3D075ecee663&e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/track/click?u3D820c8324c8d125df5a73a0bb3&id3Daa90e866d4&e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/track/click?u3D820c8324c8d125df5a73a0bb3&id3Dd0bb50405d&e3D50b9b0ed06 https://mailchi.mp/7eb4471c3c7c/good-websites-start-with-the-front-end-1434353?e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/profile?u3D820c8324c8d125df5a73a0bb3&id3D9dafc8b11e&e3D50b9b0ed06 https://stacksocial.us2.list-manage.com/unsubscribe?u3D820c8324c8d125df5a73a0bb3&id3D9dafc8b11e&e3D50b9b0ed06&c3Dc03a8cbf44 $"

I am hoping this is what you are reffering to

5

Update:
So I figured out my problem was with the "\s"
seems that logstash didn't like it that much. So I replaced it and now I am getting the splitting done right in the CMD but in the Kibana it doesn't show that its being split as it should..
Any ideas?

image
image1075×169 2.54 KB

image
image1268×202 57.7 KB

I will note that on the Json format it does show right:

image
image987×261 59.9 KB

6

The format is right in logstash and elasticsearch but not in kinana? Is the JSON from the JSON tab in kibana or from elasticsearch?

7

The json is from the Json tab in kibana so it should be sending the json right all the way.
The problem is the screen pic above it which shows that the field (Which is unknown for some reason) is showing all the data at once and not separated as I thought it will.

8

Does refreshing the index in kibana help?

9

Sadly no..
Now the field is recognized as string but still it shows it all as if its one string and not separate entries

image
image1273×639 204 KB

Is my requested output even possible in Kibana?

10

Not sure, I do not run kibana. Perhaps ask in the kibana forum.

11

I will. Thanks for that!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.