I am running a very simple config, which should index a json file. It works fine when the json is small (4 or 5 lines) but then
when it gets any longer, I get an error and it only indexes the last few lines. To test. To try different approaches (codecs, etc),
I run DELETE index in Kibana, change the file slightly by adding a new line, and try again. Its very consistent--if the json is
only a few lines its fine, but when it gets longer, no good. I will paste my config, my json, and my error below. I am using
Logstash 7.3.2:
CONFIG
input {
file {
type => "json"
#codec => "plain"
path => ["/home/yaz/app/logstash/logstash-7.3.2/logs_to_process/testlog.log"]
start_position => beginning
#codec => multiline{
# pattern => "^["
# negate => true
# what => "previous"
#}
}
}
filter{
json{
source => "message"
}
}
output {
elasticsearch {
hosts => ["http://myserver.com:9200"]
index => "tst_idx_data"
}
stdout {
codec => rubydebug
}
}
JSON
*(Note that there are no commas. From what I can see, proper json should have after each closing bracket, but I get a parsing error
every time. I have also tried around the entire json, with and without commas. cpu_seconds are sequential to make it easier to
track.)*
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":1}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":2}
{"bus":"bbb","ds":"ccc","freq":"mon","busdsfreq":"stcccmon","cpu_seconds":3}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":4}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":5}
{"bus":"bbb","ds":"ccc","freq":"mon","busdsfreq":"stcccmon","cpu_seconds":6}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":7}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":8}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":9}
{"bus":"bbb","ds":"ccc","freq":"mon","busdsfreq":"stcccmon","cpu_seconds":10}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":11}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":12}
{"bus":"bbb","ds":"ccc","freq":"mon","busdsfreq":"stcccmon","cpu_seconds":13}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":14}
{"bus":"bbb","ds":"ccc","freq":"mon","busdsfreq":"stcccmon","cpu_seconds":15}
{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":16}
ERROR
*(notice only the line with cpu_seconds=16 is indexed. All others are left out due to a parsing error. If I were to delete all
lines except cpu_seconds <4 or so, all would be fine)*
[2020-02-07T15:25:35,566][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"","busdsfreq":
"stcccmon","cpu_seconds":15}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'busdsfreq': was expecting
('true', 'false' or 'null')
at [Source: (byte)"","busdsfreq":"stcccmon","cpu_seconds":15}"; line: 1, column: 14]>}
/home/yaz/app/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/awesome_print-
1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"message" => "","busdsfreq":"stcccmon","cpu_seconds":15}",
"path" => "/home/yaz/app/logstash/logstash-7.3.2/logs_to_process/testlog.log",
"host" => "myserver.com",
"@version" => "1",
"@timestamp" => 2020-02-07T20:25:32.944Z,
"tags" => [
[0] "_jsonparsefailure"
],
"type" => "json"
}
{
"freq" => "mon",
"busdsfreq" => "aaacccmon",
"message" => "{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":16}",
"path" => "/home/yaz/app/logstash/logstash-7.3.2/logs_to_process/testlog.log",
"host" => "myserver.com",
"type" => "json",
"bus" => "aaa",
"cpu_seconds" => 16,
"ds" => "ccc",
"@version" => "1",
"@timestamp" => 2020-02-07T20:25:32.987Z
}