I found that logstash complains about syntax errors in even "simple" JSON files when a JSON "record" becomes large. By large, I mean a few hundred key/value pairs.
The example file is too large to paste in here, so created a github repository with the exmaple file and the config file:
I tried to read the file directly with logstash and also pipe it in through filebeat. The problem occurs in both cases.
The example has just one record with 617 key/value pairs in a map. Parsing it fails with the following error:
[2017-10-16T14:49:01,155][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"\"some_col_500_key\":\"some value 500\",\n\"some_col_501_key\":\"some value 501\",\n\"some_col_502_key\":\"some value 502\",\n\"some_col_503_key\":\"some value 503\",\n\"some_col_504_key\":\"some value 504\",\n\"some_col_505_key\":\"some value 505\",\n\"some_col_506_key\":\"some value 506\",\n\"some_col_507_key\":\"some value 507\",\n\"some_col_508_key\":\"some value 508\",\n\"some_col_509_key\":\"some value 509\",\n\"some_col_510_key\":\"some value 510\",\n\"some_col_511_key\":\"some value 511\",\n\"some_col_512_key\":\"some value 512\",\n\"some_col_513_key\":\"some value 513\",\n\"some_col_514_key\":\"some value 514\",\n\"some_col_515_key\":\"some value 515\",\n\"some_col_516_key\":\"some value 516\",\n\"some_col_517_key\":\"some value 517\",\n\"some_col_518_key\":\"some value 518\",\n\"some_col_519_key\":\"some value 519\",\n\"some_col_520_key\":\"some value 520\",\n\"some_col_521_key\":\"some value 521\",\n\"some_col_522_key\":\"some value 522\",\n\"some_col_523_key\":\"some value 523\",\n\"some_col_524_key\":\"some value 524\",\n\"some_col_525_key\":\"some value 525\",\n\"some_col_526_key\":\"some value 526\",\n\"some_col_527_key\":\"some value 527\",\n\"some_col_528_key\":\"some value 528\",\n\"some_col_529_key\":\"some value 529\",\n\"some_col_530_key\":\"some value 530\",\n\"some_col_531_key\":\"some value 531\",\n\"some_col_532_key\":\"some value 532\",\n\"some_col_533_key\":\"some value 533\",\n\"some_col_534_key\":\"some value 534\",\n\"some_col_535_key\":\"some value 535\",\n\"some_col_536_key\":\"some value 536\",\n\"some_col_537_key\":\"some value 537\",\n\"some_col_538_key\":\"some value 538\",\n\"some_col_539_key\":\"some value 539\",\n\"some_col_540_key\":\"some value 540\",\n\"some_col_541_key\":\"some value 541\",\n\"some_col_542_key\":\"some value 542\",\n\"some_col_543_key\":\"some value 543\",\n\"some_col_544_key\":\"some value 544\",\n\"some_col_545_key\":\"some value 545\",\n\"some_col_546_key\":\"some value 546\",\n\"some_col_547_key\":\"some value 547\",\n\"some_col_548_key\":\"some value 548\",\n\"some_col_549_key\":\"some value 549\",\n\"some_col_550_key\":\"some value 550\",\n\"some_col_551_key\":\"some value 551\",\n\"some_col_552_key\":\"some value 552\",\n\"some_col_553_key\":\"some value 553\",\n\"some_col_554_key\":\"some value 554\",\n\"some_col_555_key\":\"some value 555\",\n\"some_col_556_key\":\"some value 556\",\n\"some_col_557_key\":\"some value 557\",\n\"some_col_558_key\":\"some value 558\",\n\"some_col_559_key\":\"some value 559\",\n\"some_col_560_key\":\"some value 560\",\n\"some_col_561_key\":\"some value 561\",\n\"some_col_562_key\":\"some value 562\",\n\"some_col_563_key\":\"some value 563\",\n\"some_col_564_key\":\"some value 564\",\n\"some_col_565_key\":\"some value 565\",\n\"some_col_566_key\":\"some value 566\",\n\"some_col_567_key\":\"some value 567\",\n\"some_col_568_key\":\"some value 568\",\n\"some_col_569_key\":\"some value 569\",\n\"some_col_570_key\":\"some value 570\",\n\"some_col_571_key\":\"some value 571\",\n\"some_col_572_key\":\"some value 572\",\n\"some_col_573_key\":\"some value 573\",\n\"some_col_574_key\":\"some value 574\",\n\"some_col_575_key\":\"some value 575\",\n\"some_col_576_key\":\"some value 576\",\n\"some_col_577_key\":\"some value 577\",\n\"some_col_578_key\":\"some value 578\",\n\"some_col_579_key\":\"some value 579\",\n\"some_col_580_key\":\"some value 580\",\n\"some_col_581_key\":\"some value 581\",\n\"some_col_582_key\":\"some value 582\",\n\"some_col_583_key\":\"some value 583\",\n\"some_col_584_key\":\"some value 584\",\n\"some_col_585_key\":\"some value 585\",\n\"some_col_586_key\":\"some value 586\",\n\"some_col_587_key\":\"some value 587\",\n\"some_col_588_key\":\"some value 588\",\n\"some_col_589_key\":\"some value 589\",\n\"some_col_590_key\":\"some value 590\",\n\"some_col_591_key\":\"some value 591\",\n\"some_col_592_key\":\"some value 592\",\n\"some_col_593_key\":\"some value 593\",\n\"some_col_594_key\":\"some value 594\",\n\"some_col_595_key\":\"some value 595\",\n\"some_col_596_key\":\"some value 596\",\n\"some_col_597_key\":\"some value 597\",\n\"some_col_598_key\":\"some value 598\",\n\"some_col_599_key\":\"some value 599\",\n\"some_col_600_key\":\"some value 600\",\n\"some_col_601_key\":\"some value 601\",\n\"some_col_602_key\":\"some value 602\",\n\"some_col_603_key\":\"some value 603\",\n\"some_col_604_key\":\"some value 604\",\n\"some_col_605_key\":\"some value 605\",\n\"some_col_606_key\":\"some value 606\",\n\"some_col_607_key\":\"some value 607\",\n\"some_col_608_key\":\"some value 608\",\n\"some_col_609_key\":\"some value 609\",\n\"some_col_610_key\":\"some value 610\",\n\"some_col_611_key\":\"some value 611\",\n\"some_col_612_key\":\"some value 612\",\n\"some_col_613_key\":\"some value 613\",\n\"some_col_614_key\":\"some value 614\",\n\"some_col_615_key\":\"some value 615\",\n\"some_col_616_key\":\"some value 616\"\n}", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: (byte[])""some_col_500_key":"some value 500",
"some_col_501_key":"some value 501",
"some_col_502_key":"some value 502",
"some_col_503_key":"some value 503",
"some_col_504_key":"some value 504",
"some_col_505_key":"some value 505",
"some_col_506_key":"some value 506",
"some_col_507_key":"some value 507",
"some_col_508_key":"some value 508",
"some_col_509_key":"some value 509",
"some_col_510_key":"some value 510",
"some_col_511_key":"some value 511",
"some_col_512_key":"some value 512",
"some_col_513_key":"[truncated 3829 bytes]; line: 1, column: 20]>}
I have observed this in 5.5.1 and am still seeing this in 5.6.3
Is there a way to avoid this without having to shorten the content?
Can I report this as a bug if I am just a community user?
Thank you!