# How to remove "message" field added by Logstash

**URL:** https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899
**Category:** Logstash
**Created:** [February 25, 2020, 5:02pm UTC](https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899 "2020-02-25T17:02:45Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![welin](https://avatars.discourse-cdn.com/v4/letter/w/ed8c4c/32.png) [@welin](https://discuss.elastic.co/u/welin)
#### Post date: [February 25, 2020, 5:02pm UTC](https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899/1 "2020-02-25T17:02:45Z")

</div>

I want to store json object {'customerId': '123', 'customerName': 'abc'} to Elastisearch via Logstash, but the json object is stored as message="{'customerId': '123', 'customerName': 'abc'}" in Elasticsearch. How can I **remove "message" field** and make my own fields be direct children of "\_source"?

**Node.js code using UDP socket to send json:**  
\> var dgram = require('dgram');

> ```
> var client = dgram.createSocket('udp4');
> var message = new Buffer("{'customerId': '123', 'customerName': 'abc'}"); // I tried to remove double quote here, and the object will not be stored to Elasticsearch
> client.send(message, 0, message.length, port, host, function(err, bytes) {
> if (err) throw err;
> console.log('UDP client message sent');
> client.close();
> });
> 
> ```

**Logstash configuration:**

> input {  
> udp {  
> port =\> 8515  
> }  
> }
> 
> filter{  
> json{  
> source =\> "message"  
> }  
> }
> 
> output {  
> elasticsearch {  
> hosts =\> ["[https://172.25.75.103:443](https://172.25.75.103:443)"]  
> user =\> "myUser"  
> password =\> "myPassword"  
> index =\> "myIndex"  
> }  
> }

**Current output:**

> ```
> {
> "_index": "myIndex",
> "_type": "doc",
> "_id": "N_w8fXABocJFtplFKRoK",
> "_version": 1,
> "_score": null,
> "_source": {
> "@version": "1",
> "host": "172.19.132.126",
> "@timestamp": "2020-02-25T16:44:10.891Z",
> "message": "{'customerId': '123', 'customerName': 'abc'}"
> }
> 
> ```

**Expected Output:**

> ```
> {
> "_index": "myIndex",
> "_type": "doc",
> "_id": "N_w8fXABocJFtplFKRoK",
> "_version": 1,
> "_score": null,
> "_source": {
> "@version": "1",
> "host": "172.19.132.126",
> "@timestamp": "2020-02-25T16:44:10.891Z",
> "customerId": "123",
> "customerName" : "abc"
> }
> 
> ```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 25, 2020, 5:24pm UTC](https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899/2 "2020-02-25T17:24:30Z")

</div>

I would expect you to get a \_jsonparsefailure with that input because a json filter requires names and objects to be surrounded by double quotes, not single quotes.

---

<div class="post-metadata">

### Author: ![welin](https://avatars.discourse-cdn.com/v4/letter/w/ed8c4c/32.png) [@welin](https://discuss.elastic.co/u/welin)
#### Post date: [February 25, 2020, 5:38pm UTC](https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899/3 "2020-02-25T17:38:51Z")

</div>

I tried following 2 format, both are posted

message = new Buffer("{'test1':'test1'}");  
message = new Buffer('{"test2":"test2"}');

"\_source": {  
"message": "{'test1':'test1'}",  
}

"\_source": {  
"message": "{"test2":"test2"}",  
}

---

<div class="post-metadata">

### Author: ![welin](https://avatars.discourse-cdn.com/v4/letter/w/ed8c4c/32.png) [@welin](https://discuss.elastic.co/u/welin)
#### Post date: [February 25, 2020, 9:37pm UTC](https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899/4 "2020-02-25T21:37:17Z")

</div>

I changed UDP to HTTP POST and am able to post a json object with all its attributes be direct children of "\_source".  
Is it true that because applicaiton/json is a HTTP level content type so Logstash only accepts JSON through HTTP and if protocol is TCP/UDP then content has to be decoded as a String and assigned to "message"? No way to workaround it under UDP then?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 25, 2020, 9:47pm UTC](https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899/5 "2020-02-25T21:47:04Z")

</div>

I can think of no reason why a json filter would not work with a UDP filter. But then again, I can see no way that documents like the ones you show could possibly be created by the logstash configuration you show.

---

<div class="post-metadata">

### Author: ![welin](https://avatars.discourse-cdn.com/v4/letter/w/ed8c4c/32.png) [@welin](https://discuss.elastic.co/u/welin)
#### Post date: [February 25, 2020, 9:54pm UTC](https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899/6 "2020-02-25T21:54:36Z")

</div>

> But then again, I can see no way that documents like the ones you show could possibly be created by the logstash configuration you show.

Are you referring to UDP or HTTP?  
The code and configuration in original post is for UDP. We have another application written in Scala using logback to post message to Logstash via UDP, the message itself is assigned to "message" field too after being posted in Elasticsearch

With HTTP, I removed json filter:  
input {  
http {  
port =\> 8515  
}  
}  
output {  
elasticsearch {  
hosts =\> ["[https://172.25.75.103:443](https://172.25.75.103/)"]  
user =\> "myUser"  
password =\> "myPassword"  
index =\> "myIndex"  
}  
}

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 25, 2020, 11:23pm UTC](https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899/7 "2020-02-25T23:23:29Z")

</div>

> [@welin](#):
>
> Are you referring to UDP or HTTP?

UDP.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 24, 2020, 11:23pm UTC](https://discuss.elastic.co/t/how-to-remove-message-field-added-by-logstash/220899/8 "2020-03-24T23:23:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
