# How to change a message into multiple key and value in logstash filter

**URL:** <https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199>\
**Category:** Logstash\
**Created:** [August 16, 2017, 7:48am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199 "2017-08-16T07:48:10Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Robin\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_guo/32/42297_2.png) [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\
**Post date:** [August 16, 2017, 7:48am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199/1 "2017-08-16T07:48:10Z")

</div>

**original data :**

![image](https://us1.discourse-cdn.com/elastic/original/3X/c/1/c1d579303f76b69f293331b9ad94dabbf8f07aa7.png)

**except field in kibana : (I also want to remove message field)**

```
agent： metricbeat
env: production
dc: fr4  
pgpool_activite: 20
pgpool_idle:8
pgpool_waiting:100 
...
...
....
```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 16, 2017, 8:39am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199/2 "2017-08-16T08:39:14Z")

</div>

What does your Logstash config look like? Are you using a son filter or codec?

---

<div class="post-metadata">

**Author:** ![Robin\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_guo/32/42297_2.png) [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\
**Post date:** [August 16, 2017, 9:42am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199/3 "2017-08-16T09:42:40Z")

</div>

I just sent a jason based data to logstash.

#json data

```
./pgpool.sh >/tmp/test

cat /tmp/test
{"tags":"tcpbeat","agent":"metricbeat","env":"production","dc":"fr4","os":"Linux","service":"pgpool","beat.name:"robinguo-HP-Z210-Workstation","pgpool_active":20,"pgpool_idle":80,"pgpool_waiting":100}
170816173601 root@robinguo-HP-Z210-Workstation python # nc logstashtest01.tls.ad 5050 < /tmp/test

```

#logstash pipline  
170816173806 root@ conf.d # pwd  
/etc/logstash/conf.d

```
cat tcpbeat.conf
input {
  tcp {
    port => 5050
  }
}

output {
    file {
    path => "/tmp/logstash-5050"
    }
}
```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 16, 2017, 9:45am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199/4 "2017-08-16T09:45:45Z")

</div>

Try adding a [json filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) to the config in order to parse the JSON object.

---

<div class="post-metadata">

**Author:** ![Robin\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_guo/32/42297_2.png) [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\
**Post date:** [August 16, 2017, 9:50am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199/5 "2017-08-16T09:50:53Z")

</div>

Would you please walk me through an example to do this cuz I'm a newbie to logstash?

Thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 16, 2017, 10:08am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199/6 "2017-08-16T10:08:54Z")

</div>

Add the following (taken from the example in the docs I linked to) ahead of the output section and see how it works:

```auto
filter {
  json {
    source => "message"
  }
}

```

---

<div class="post-metadata">

**Author:** ![Robin\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_guo/32/42297_2.png) [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\
**Post date:** [August 16, 2017, 10:38am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199/7 "2017-08-16T10:38:53Z")

</div>

**seems it's not working well.**

```
#logstash for tcpbeat
input {
  tcp {
    port => 5050
  }
}

filter {
  json {
    source => "message"
  }
}

output {
    file {
    path => "/tmp/logstash-5050"
    }
}

```

**the results:**  
`{"@timestamp":"2017-08-16T10:37:02.669Z","port":46369,"@version":"1","host":"10.65.186.63","message":"{\"tags\":\"tcpbeat\",\"agent\":\"metricbeat\",\"env\":\"production\",\"dc\":\"fr4\",\"os\":\"Linux\",\"service\":\"pgpool\",\"beat.name:\"robinguo-HP-Z210-Workstation\",\"pgpool_active\":20,\"pgpool_idle\":80,\"pgpool_waiting\":100}","tags":["_jsonparsefailure"]}`

---

<div class="post-metadata">

**Author:** ![Robin\_Guo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_guo/32/42297_2.png) [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\
**Post date:** [August 18, 2017, 4:32am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199/8 "2017-08-18T04:32:05Z")

</div>

hi @Christian_Dahlqvist,  
Finally, It works. it's because of bad JSON format strings.

one more request, I want to remove the unnecessary key from logstash pipeline which is I want to remove message {} or content {}, but still need to keep the contents of in it, is there a way can do it?

**#logstash pipline conf**

```
input {
  tcp {
    port => 5050
  }
}

filter {
  json {
    source => "message"
    target => "content"
  }
  mutate {
         remove_field => ["message"]  
  }
}

output {
     file {
         path =>"/tmp/logstash-5050"
     }
}

```

**the original output**

`{"@timestamp":"2017-08-18T04:07:44.125Z","port":36600,"@version":"1","host":"10.65.186.63","content":{"pgpool_waiting":100,"agent":"metricbeat","beat_name":"robinguo-HP-Z210-Workstation","os":"Linux","service":"pgpool","pgpool_idle":80,"env":"production","dc":"fr4","pgpool_active":20}}`

**the results I need to output**

`{"@timestamp":"2017-08-18T04:07:44.125Z","port":36600,"@version":"1","host":"10.65.186.63","pgpool_waiting":100,"agent":"metricbeat","beat_name":"robinguo-HP-Z210-Workstation","os":"Linux","service":"pgpool","pgpool_idle":80,"env":"production","dc":"fr4","pgpool_active":20}`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 15, 2017, 4:32am UTC](https://discuss.elastic.co/t/how-to-change-a-message-into-multiple-key-and-value-in-logstash-filter/97199/9 "2017-09-15T04:32:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.