Newbie alert. Today is my first day reading through elasticsearch, logstash and kibana.
Goal: post a chunked stream of json strings to an http server that converts each json chunk to be the message inserted into elasticsearch. Would really like a persistent connection (a.k.a) streaming interface.
Maybe this can be done using existing logstash input/filter/output/codec config.
Maybe, I'll have to write a Ruby input plugin that does this if not possible with existing filters.
Why: The existing code base is C++. I want to send custom messages to the ELK stack from my C++ program. I could probably use flume as the source, but would prefer not to. I may eventually move to pushing messages into Kafka instead of using flume. I suppose I could skip logstash and write directly to elasticsearch, but the roadmap for logstash seems to promise some durability (like writing to disk if the connection to the elasticsearch server is down) and other goodies.
I tried this config:
input { stdin {} }
filter {
json {
source => "string"
target => "json"
}
json_encode {
source => "json"
add_field => {
"new_field" => "new_static_value"
"alex_agent" => "%{agent}"
}
}
}
output { stdout { codec => "rubydebug"} }
With this session:
logstash-2.0.0]$ bin/logstash -f stdin.conf
Default settings used: Filter workers: 2
Logstash startup completed
{"agent" : "tsung", "host" : "arrakis"}
{
"message" => "{"agent" : "tsung", "host" : "arrakis"}",
"@version" => "1",
"@timestamp" => "2015-11-04T23:46:37.862Z",
"host" => "localhost.localdomain",
"json" => "null",
"new_field" => "new_static_value",
"alex_agent" => "%{agent}"
}
Logstash shutdown completed
[anelson@localhost logstash-2.0.0]$
I'm obviously missing something basic which is a common disease for newbies.