Elasticsearch / Logstash isnt indexing all lines of json file

I am running a very simple config, which should index a json file. It works fine when the json is small (4 or 5 lines) but then

when it gets any longer, I get an error and it only indexes the last few lines. To test. To try different approaches (codecs, etc),

I run DELETE index in Kibana, change the file slightly by adding a new line, and try again. Its very consistent--if the json is

only a few lines its fine, but when it gets longer, no good. I will paste my config, my json, and my error below. I am using

Logstash 7.3.2:

input {
file {
type => "json"
#codec => "plain"
path => ["/home/yaz/app/logstash/logstash-7.3.2/logs_to_process/testlog.log"]
start_position => beginning
#codec => multiline{
# pattern => "^["
# negate => true
# what => "previous"

source => "message"

output {
elasticsearch {
hosts => ["http://myserver.com:9200"]
index => "tst_idx_data"
stdout {
codec => rubydebug

*(Note that there are no commas. From what I can see, proper json should have after each closing bracket, but I get a parsing error

every time. I have also tried around the entire json, with and without commas. cpu_seconds are sequential to make it easier to



*(notice only the line with cpu_seconds=16 is indexed. All others are left out due to a parsing error. If I were to delete all

lines except cpu_seconds <4 or so, all would be fine)*

[2020-02-07T15:25:35,566][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"","busdsfreq":

"stcccmon","cpu_seconds":15}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'busdsfreq': was expecting

('true', 'false' or 'null')
at [Source: (byte)"","busdsfreq":"stcccmon","cpu_seconds":15}"; line: 1, column: 14]>}

1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
"message" => "","busdsfreq":"stcccmon","cpu_seconds":15}",
"path" => "/home/yaz/app/logstash/logstash-7.3.2/logs_to_process/testlog.log",
"host" => "myserver.com",
"@version" => "1",
"@timestamp" => 2020-02-07T20:25:32.944Z,
"tags" => [
[0] "_jsonparsefailure"
"type" => "json"
"freq" => "mon",
"busdsfreq" => "aaacccmon",
"message" => "{"bus":"aaa","ds":"ccc","freq":"mon","busdsfreq":"aaacccmon","cpu_seconds":16}",
"path" => "/home/yaz/app/logstash/logstash-7.3.2/logs_to_process/testlog.log",
"host" => "myserver.com",
"type" => "json",
"bus" => "aaa",
"cpu_seconds" => 16,
"ds" => "ccc",
"@version" => "1",
"@timestamp" => 2020-02-07T20:25:32.987Z

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.