How to remove "message" field added by Logstash

Elastic Stack Logstash
1

I want to store json object {'customerId': '123', 'customerName': 'abc'} to Elastisearch via Logstash, but the json object is stored as message="{'customerId': '123', 'customerName': 'abc'}" in Elasticsearch. How can I remove "message" field and make my own fields be direct children of "_source"?

Node.js code using UDP socket to send json:
> var dgram = require('dgram');

        var client = dgram.createSocket('udp4');
        var message = new Buffer("{'customerId': '123', 'customerName': 'abc'}"); // I tried to remove double quote here, and the object will not be stored to Elasticsearch
        client.send(message, 0, message.length, port, host, function(err, bytes) {
                if (err) throw err;
                console.log('UDP client message sent');
                client.close();
            });

Logstash configuration:

input {
udp {
port => 8515
}
}

filter{
json{
source => "message"
}
}

output {
elasticsearch {
hosts => ["https://172.25.75.103:443"]
user => "myUser"
password => "myPassword"
index => "myIndex"
}
}

Current output:

{
      "_index": "myIndex",
      "_type": "doc",
      "_id": "N_w8fXABocJFtplFKRoK",
      "_version": 1,
      "_score": null,
      "_source": {
          "@version": "1",
          "host": "172.19.132.126",
          "@timestamp": "2020-02-25T16:44:10.891Z",
          "message": "{'customerId': '123', 'customerName': 'abc'}"
    }

Expected Output:

{
          "_index": "myIndex",
          "_type": "doc",
          "_id": "N_w8fXABocJFtplFKRoK",
          "_version": 1,
          "_score": null,
          "_source": {
              "@version": "1",
              "host": "172.19.132.126",
              "@timestamp": "2020-02-25T16:44:10.891Z",
              "customerId": "123",
              "customerName" : "abc"
}
2

I would expect you to get a _jsonparsefailure with that input because a json filter requires names and objects to be surrounded by double quotes, not single quotes.

3

I tried following 2 format, both are posted

message = new Buffer("{'test1':'test1'}");
message = new Buffer('{"test2":"test2"}');

"_source": {
"message": "{'test1':'test1'}",
}

"_source": {
"message": "{"test2":"test2"}",
}

4

I changed UDP to HTTP POST and am able to post a json object with all its attributes be direct children of "_source".
Is it true that because applicaiton/json is a HTTP level content type so Logstash only accepts JSON through HTTP and if protocol is TCP/UDP then content has to be decoded as a String and assigned to "message"? No way to workaround it under UDP then?

5

I can think of no reason why a json filter would not work with a UDP filter. But then again, I can see no way that documents like the ones you show could possibly be created by the logstash configuration you show.

6

But then again, I can see no way that documents like the ones you show could possibly be created by the logstash configuration you show.

Are you referring to UDP or HTTP?
The code and configuration in original post is for UDP. We have another application written in Scala using logback to post message to Logstash via UDP, the message itself is assigned to "message" field too after being posted in Elasticsearch

With HTTP, I removed json filter:
input {
http {
port => 8515
}
}
output {
elasticsearch {
hosts => ["https://172.25.75.103:443"]
user => "myUser"
password => "myPassword"
index => "myIndex"
}
}

7

UDP.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.