Using logstash filters to convert stdin message to be the json written to elasticsearch

Elastic Stack Logstash
1

Newbie alert. Today is my first day reading through elasticsearch, logstash and kibana.

Goal: post a chunked stream of json strings to an http server that converts each json chunk to be the message inserted into elasticsearch. Would really like a persistent connection (a.k.a) streaming interface.

Maybe this can be done using existing logstash input/filter/output/codec config.

Maybe, I'll have to write a Ruby input plugin that does this if not possible with existing filters.

Why: The existing code base is C++. I want to send custom messages to the ELK stack from my C++ program. I could probably use flume as the source, but would prefer not to. I may eventually move to pushing messages into Kafka instead of using flume. I suppose I could skip logstash and write directly to elasticsearch, but the roadmap for logstash seems to promise some durability (like writing to disk if the connection to the elasticsearch server is down) and other goodies.

I tried this config:

input { stdin {} }

filter {
json {
source => "string"
target => "json"
}

   json_encode {
               source => "json"
               add_field => {
                      "new_field" => "new_static_value"
                      "alex_agent" => "%{agent}"
               }
   }

}

output { stdout { codec => "rubydebug"} }

With this session:

logstash-2.0.0]$ bin/logstash -f stdin.conf
Default settings used: Filter workers: 2
Logstash startup completed
{"agent" : "tsung", "host" : "arrakis"}
{
"message" => "{"agent" : "tsung", "host" : "arrakis"}",
"@version" => "1",
"@timestamp" => "2015-11-04T23:46:37.862Z",
"host" => "localhost.localdomain",
"json" => "null",
"new_field" => "new_static_value",
"alex_agent" => "%{agent}"
}
Logstash shutdown completed
[anelson@localhost logstash-2.0.0]$

I'm obviously missing something basic which is a common disease for newbies.

2

Hi Alex,
text-based input like stdin, file always write to the message field of the event, so your json filter source should be source => "message" instead of "string"

Or you can try to configure your stdin input to use the json codec directly

Cheers.

3

Thank you @wiibaa.
All of the inputs, outputs and filters have a complete list of supported controls, but what is missing for me are examples.