How to send JSON directly to elasticsearch w/o any parsing in logstash

Elastic Stack Logstash
1

hey folks,

My log files are already in JSON format and I have full control of how they look. When I use logstash, looks like I have to specify the input as JSON either in File plugin (use codec) or Filter (use json). I want to make logstash consume less CPU resources, say without any parsing, just read and send.

Here are my configurations
logstash 2.1

input file
{"@timestamp":"2018-03-08T22:15:44,267", "className":"myClassName","logLevel":"WARN","threadName":"main","requestId":"","message":"some messages"}

input {
   file {
      path => "my.json*"
      exclude => "*.gz"
      sincedb_path => "file.sincedb"
      type => "some_tpye"
   }
}
filter {
   #json {
   #  source => "message"
   #}
   mutate {
      remove_field => ["@version", "path"]
   }
}
output {
   stdout{ codec => rubydebug { metadata => true }}
}

The output

{
    "message" => "{\"@timestamp\":\"2018-03-08T22:15:44,267\", \"className\":\"myClassName\",\"logLevel\":\"WARN\",\"threadName\":\"main\",\"requestId\":\"\",\"message\":\"some messages\"}",
    "type" => "some_tpye",
    "@metadata" => {
      "path" => "file_path"
    }
}

My question is

  1. how to make logstash read json directly and send to elasticsearch w/o parsing, current file input will set each line as message, which is not needed.

I know JSON filter works, but I want to use less host resources. As far as I know, JSON filter will still validate the input, which may hurt when we have large input.

Thanks!

2

Unfortunately this isn't possible.

The intermediate representation of an event in a Logstash pipeline is called Event, which is a fully-contextualised object with fields and metadata.

The logstash-output-elasticsearch plugin knows how to inject this Event into a bulk request against the Elasticsearch API, and uses individual attributes of the Event to formulate each entry in the bulk insert request (such as which index to insert to, the document's id, etc.); it cannot use individual attributes unless it first parses the event.

That said, you may be interested in using a Filebeat, and sending the logs directly to Elasticsearch (docs, compatibility matrix); Beats are extremely light-weight log- and metric-shippers that are useful for capturing logs and metrics on edge machines.

3

Hey Ry, sounds like filebeat suits my case better. Thanks for the this!

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.