# Logstash Split Plugin

**URL:** <https://discuss.elastic.co/t/logstash-split-plugin/74322>\
**Category:** Logstash\
**Created:** [February 8, 2017, 6:07am UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322 "2017-02-08T06:07:51Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shrawan\_Bhagwat](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\
**Post date:** [February 8, 2017, 6:07am UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/1 "2017-02-08T06:07:52Z")

</div>

Hi All,

We do have JSON data in the form  
{  
"data": [

```
	{
		"appName": "DemoApp",
		"appVersion": "1.1",
		"deviceId": "1234567",
		"deviceName": "moto e",
		"deviceOSVersion": "5.1",
		"packageName": "com.abc.DemoApp",
		"message": "testing null pointer exception",
		"errorLog": "null pointer exception"
	},

	{
		"appName": "DemoApp",
		"appVersion": "1.1",
		"deviceId": "1234567",
		"deviceName": "moto e",
		"deviceOSVersion": "5.1",
		"packageName": "com.abc.DemoApp",
		"message": "testing illegal state exception",
		"errorLog": "illegal state exception"
	}
]

```

}

We want to split data into separate messages and different fields like appName, appVersion etc.  
My queries:  
i) We use Split filter to split it into different fields and we are getting "data\_appName" as field name instead of "appName"! how to change this field to "appName" etc without data\_ prefix  
ii) Message field contains both data not single data. how to split message field?  
iii) We are getting %host in source field. I have tried to rename this using mutate plugin but it doesnt work. How can i rename field value in this??

filter in config file:

filter{  
json {  
source =\> "message"  
}

```
   mutate { gsub => ["message", "},", "shr"] }
    split {
         terminator => "shr"
          field => "data"
   }

```

Please guide for both queries.

Regards,  
Shrawan

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 8, 2017, 6:46am UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/2 "2017-02-08T06:46:01Z")

</div>

> We use Split filter to split it into different fields and we are getting "data\_appName" as field name instead of "appName"! how to change this field to "appName" etc without data\_ prefix

I can't reproduce.

```plaintext
$ cat test.config 
input { stdin { codec => json } }
output { stdout { codec => rubydebug } }
filter {
  split {
    field => "data"
  }
}
$ cat data 
{ "data": [{ "appName": "DemoApp", "appVersion": "1.1", "deviceId": "1234567", "deviceName": "moto e", "deviceOSVersion": "5.1", "packageName": "com.abc.DemoApp", "message": "testing null pointer exception", "errorLog": "null pointer exception" }, { "appName": "DemoApp", "appVersion": "1.1", "deviceId": "1234567", "deviceName": "moto e", "deviceOSVersion": "5.1", "packageName": "com.abc.DemoApp", "message": "testing illegal state exception", "errorLog": "illegal state exception" }] }
$ cat data | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
{
          "data" => {
                "appName" => "DemoApp",
             "appVersion" => "1.1",
               "deviceId" => "1234567",
             "deviceName" => "moto e",
        "deviceOSVersion" => "5.1",
            "packageName" => "com.abc.DemoApp",
                "message" => "testing null pointer exception",
               "errorLog" => "null pointer exception"
    },
      "@version" => "1",
    "@timestamp" => "2017-02-08T06:42:43.941Z",
          "host" => "lnxolofon"
}
{
          "data" => {
                "appName" => "DemoApp",
             "appVersion" => "1.1",
               "deviceId" => "1234567",
             "deviceName" => "moto e",
        "deviceOSVersion" => "5.1",
            "packageName" => "com.abc.DemoApp",
                "message" => "testing illegal state exception",
               "errorLog" => "illegal state exception"
    },
      "@version" => "1",
    "@timestamp" => "2017-02-08T06:42:43.941Z",
          "host" => "lnxolofon"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

```

> Message field contains both data not single data. how to split message field?

I don't understand.

> We are getting %host in source field. I have tried to rename this using mutate plugin but it doesnt work. How can i rename field value in this??

Who or what populates the `source` field? What inputs do you have?

Instead of describing what you get, show it. Use a `stdout { codec => rubydebug }` output. Even better if you provide a reproducibility recipe like I have above where the input, the configuration, and the results are clearly visible.

---

<div class="post-metadata">

**Author:** ![Shrawan\_Bhagwat](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\
**Post date:** [February 8, 2017, 7:12am UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/3 "2017-02-08T07:12:27Z")

</div>

![](https://us1.discourse-cdn.com/elastic/original/2X/5/5ba34cfadc80d44336950c97d7945a6648193d36.png)

Hi Magnus, 🙂

I have attached screen shot of output that we are getting on Graylog UI.  
In this output as you can see below points:

1. when we are splitting "field" on basis of "data", the field name are getting prefixed by "data\_". What we can do to avoid this prefix or to replace this prefix with original field name?
2. When we are splitting it on basis of field, "full\_message" or "message" field contains complete message instead of individual messages like  
output1 -  
{  
"appName": "DemoApp",  
"appVersion": "1.1",  
"deviceId": "1234567",  
"deviceName": "moto e",  
"deviceOSVersion": "5.1",  
"packageName": "com.abc.DemoApp",  
"message": "testing null pointer exception",  
"errorLog": "null pointer exception"  
},

output 2:  
{  
"appName": "DemoApp",  
"appVersion": "1.1",  
"deviceId": "1234567",  
"deviceName": "moto e",  
"deviceOSVersion": "5.1",  
"packageName": "com.abc.DemoApp",  
"message": "testing null pointer exception",  
"errorLog": "null pointer exception"  
},

1. we know that source field contains value of source from which we are accessing this. but in our case these are service calls and therefore the source party is not having any name or so. So is there any way to rename field value of source field and facility field?

Regards,  
shrawan

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 8, 2017, 7:28am UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/4 "2017-02-08T07:28:05Z")

</div>

> when we are splitting "field" on basis of "data", the field name are getting prefixed by "data\_". What we can do to avoid this prefix or to replace this prefix with original field name?

Okay. The data\_ prefix is probably a Graylog thing. If you move the subfields to the `data` field into the root of the message that'll probably fix itself.

I don't think you can change the current behavior of split where it places the fields as subfields of `data`, but you can use a mutate filter's rename option to move the subfields to the root of the message. If you won't know exactly what fields you'll have you need to use a ruby filter for moving the fields.

> When we are splitting it on basis of field, "full\_message" or "message" field contains complete message instead of individual messages like

This'll also fix itself when you address the previous point.

> we know that source field contains value of source from which we are accessing this. but in our case these are service calls and therefore the source party is not having any name or so. So is there any way to rename field value of source field and facility field?

So I ask again: What inputs do you have?

---

<div class="post-metadata">

**Author:** ![Shrawan\_Bhagwat](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\
**Post date:** [February 8, 2017, 7:45am UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/5 "2017-02-08T07:45:43Z")

</div>

Hi Magnus,

for point#1: we have tried using following ruby filter to rename all dynamic fields that start with data\_ ,but it didn't work.  
ruby {  
code =\> "  
event.to\_hash.keys.each { |k|  
if k.start\_with?('data\_')  
[k.tr](http://k.tr)('data\_',' ')  
end  
}  
"  
}

Can you please guide us with this?

for point #3,  
We do have input of the following form:  
input {

```
    http {
    codec => "plain"
    }

```

}

Regards,  
Shrawan

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 8, 2017, 8:43am UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/6 "2017-02-08T08:43:17Z")

</div>

Your Ruby snippet has at least two problems:

- It tries to modify the key name in place which won't work.
- You're ignoring what I said about the data\_ prefix being specific to Graylog. As the example I showed previously the split filter doesn't produce any field prefixed with data\_.

(You're potentially also misunderstanding with tr does, but in this case it happens to do what you want anyway.)

Have a look at [http://stackoverflow.com/a/28368575/414355](http://stackoverflow.com/a/28368575/414355). If you run Logstash 2.4 or later you need to switch to the new event API so the answer won't be usable out of the box.

Regarding the `source` field I don't know what's going on. AFAICT the http input doesn't add such a field.

---

<div class="post-metadata">

**Author:** ![Shrawan\_Bhagwat](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\
**Post date:** [February 8, 2017, 11:25am UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/7 "2017-02-08T11:25:12Z")

</div>

Thanks Magnus for your help 🙂 I will let you once i will try that.

---

<div class="post-metadata">

**Author:** ![Shrawan\_Bhagwat](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\
**Post date:** [February 8, 2017, 12:25pm UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/8 "2017-02-08T12:25:17Z")

</div>

Hi Magnus,

Can you please help me with add\_field feature of mutate plugin:

I am trying to add new field name appName and i want it to contain dynamic value of appName. i have tried below mentioned code:  
mutate {  
add\_field =\> {"AppName"=\> "%{appName}"}  
}

but it's printing %{appName} in field value instead of dynamic value.

Please guide.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 8, 2017, 12:31pm UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/9 "2017-02-08T12:31:03Z")

</div>

That indicates that the event didn't contain an `appName` field.

---

<div class="post-metadata">

**Author:** ![Shrawan\_Bhagwat](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\
**Post date:** [February 8, 2017, 12:40pm UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/10 "2017-02-08T12:40:32Z")

</div>

How can i add new field on basis of appName i m getting in DATA?  
can you please help. 🙂

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 8, 2017, 12:44pm UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/11 "2017-02-08T12:44:34Z")

</div>

What do your events currently look like? What do you want them to look like instead? What does your Logstash configuration look like? Please provide as much information as possible. I don't have time to play 20 questions every time people here need help.

---

<div class="post-metadata">

**Author:** ![Shrawan\_Bhagwat](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\
**Post date:** [February 8, 2017, 12:48pm UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/12 "2017-02-08T12:48:26Z")

</div>

here is my config file:  
input {

```
    http {
    codec => "plain"
    }

```

}

filter{

```
   mutate { gsub => ["message", "},", "shr"] }
    split {
         terminator => "shr"
          }

```

mutate {  
add\_field =\> {"AppName"=\> "%{appName}"}  
}  
}

here is the data that we are receiving through http plugin:  
{  
"data": [

```
	{
		"appName": "DemoApp",
		"appVersion": "1.1",
		"deviceId": "1234567",
		"deviceName": "moto e",
		"deviceOSVersion": "5.1",
		"packageName": "com.abc.DemoApp",
		"message": "testing null pointer exception",
		"errorLog": "null pointer exception"
	},

	{
		"appName": "DemoApp",
		"appVersion": "1.1",
		"deviceId": "1234567",
		"deviceName": "moto e",
		"deviceOSVersion": "5.1",
		"packageName": "com.abc.DemoApp",
		"message": "testing illegal state exception",
		"errorLog": "illegal state exception"
	}
]

```

}

we want to separate all data like appName, deviceID etc into separate field.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 8, 2017, 12:54pm UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/13 "2017-02-08T12:54:38Z")

</div>

I've already answered your question with a reference to a ruby filter that you can use, but now you're back to asking the same question again. Sorry, I can't help here anymore.

---

<div class="post-metadata">

**Author:** ![Shrawan\_Bhagwat](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\
**Post date:** [February 8, 2017, 12:55pm UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/14 "2017-02-08T12:55:39Z")

</div>

thanks magnus 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 8, 2017, 12:56pm UTC](https://discuss.elastic.co/t/logstash-split-plugin/74322/15 "2017-03-08T12:56:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.