Json parsing

[Nikolay_Petrov (raba)]

Hi folks,
could you please help to understand 'json' data processing eg.:
json data file:

ssl.json:

[
{
"certname": "host1",
"environment": "nk",
"name": "ruby",
"value": {
"platform": "x86_64-linux",
"sitedir": "/path/site",
"version": "2.1.9"
}
},
{
"certname": "host2",
"environment": "staging",
"name": "memorysize_mb",
"value": 7986.953125
}
]

I'm trying to store the new incoming json data(see above) in separate index with fields namae grabbing by json, but my fault most likely is not enough experience on this matter and I have as a result that all data is stored in one big message field like:

To get it, i setup a http server:

input {
http {
host => "192.168.1.2"
port => 8088
}
}
I'm not sure that message are going through this filter because (see below how data is posting to elastic):
filter {
if [headers][request_path] == "/ssl-events" {
json {
source => "message"
}
}
mutate {
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/ {
json { source => message }
}
}
output {
if [headers][request_path] == "/ssl-events" {
elasticsearch {
codec => json
hosts => "192.168.1.2:9200"
manage_template => false
index => "ssl-events-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}

posting data to elastic

curl -XPOST http://192.168.1.2:8088/ssl-events -d '@ssl.json'

The question is: how to setup in the configuration to be able to store the data in elastic field like in json file:
[item]: [value] :

"certname": "host1"
"environment": "nk"
"name": "ruby"
"platform": "x86_64-linux"
"sitedir": "/path/site"
"version": "2.1.9"
"certname": "host2"
"environment": "staging"
"name": "memorysize_mb"
"value": 7986.953125

[magnusbaeck (Magnus Bäck)]

Your configuration looks reasonable. For now, replace the elasticsearch output with a stdout { codec => rubydebug } output. What do you get in the Logstash log when you post ssl.json to Logstash?

replace => [ "message", "%{message}" ]

This doesn't make any sense.

codec => json

Remove.

[Nikolay_Petrov (raba)]

Thanks for your reply!
I was change configuration based on your recommendations.
My json file has ~3million lines(the example from above was just in short form).
When i post it to the elastic logstash crashed with:

[2017-10-27T07:29:02,883][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NameError: undefined local variable or method dotfile' for #<AwesomePrint::Inspector:0x8dfd0b>>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/awesome_print-1.8.0/lib/awesome_print/inspector.rb:163:in merge_custom_defaults!'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/awesome_print-1.8.0/lib/awesome_print/inspector.rb:50:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/awesome_print-1.8.0/lib/awesome_print/core_ext/kernel.rb:9:in ai'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-rubydebug-3.0.4/lib/logstash/codecs/rubydebug.rb:39:in encode_default'", "org/jruby/RubyMethod.java:120:in call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-rubydebug-3.0.4/lib/logstash/codecs/rubydebug.rb:35:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/base.rb:50:in multi_encode'", "org/jruby/RubyArray.java:1613:in each'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/base.rb:50:in multi_encode'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:90:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/single.rb:15:in multi_receive'", "org/jruby/ext/thread/Mutex.java:149:in synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:436:in output_batch'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:435:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:381:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in start_workers'"]}

and I'm getting expected behaviour from curl that server is down:

curl: (52) Empty reply from server

and after that logstash is auto restarting by initctl and return to normal state.

[magnusbaeck (Magnus Bäck)]

Appears to be a known bug (https://github.com/logstash-plugins/logstash-output-stdout/issues/11). You can use a json_lines codec instead.

[Nikolay_Petrov (raba)]

Fixed for now.
Thanks a lot.
Let mark this case fixed with suggested solution(it works if disable codec rubydebug).
Just in case I will re-open or rise another one