# Logstash codecs (multiline,json) ParseError: illegal character

**URL:** <https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292>\
**Category:** Logstash\
**Created:** [January 12, 2018, 12:35pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292 "2018-01-12T12:35:22Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ged/32/65761_2.png) [@Ged](https://discuss.elastic.co/u/Ged)\
**Post date:** [January 12, 2018, 12:35pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/1 "2018-01-12T12:35:23Z")

</div>

Hi,

Elasticsearch 5.5.0  
Logstash 5.5.0  
logstash-codec-multiline 3.0.5  
logstash-codec-json 3.0.3.

Another issue when reading file (NFS) containing JSON lines using above.

From time to time getting such an error:

:exception=\>#\<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed between tokens

even if line in log file is correctly ended with \n

Any ideas ? May it be NFS problem ?

Thanks in advance !

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 16, 2018, 11:09am UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/2 "2018-01-16T11:09:15Z")

</div>

The error message indicates nul characters in the JSON data and that's not valid. You can use a tool like hexdump to inspect what the file really contains, including non-printable characters like nul.

---

<div class="post-metadata">

**Author:** ![Ged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ged/32/65761_2.png) [@Ged](https://discuss.elastic.co/u/Ged)\
**Post date:** [January 16, 2018, 6:49pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/3 "2018-01-16T18:49:05Z")

</div>

HI,

Do you think that issue is not related to:

[https://discuss.elastic.co/t/logstash-vs-nfs-null-characters/25918](https://discuss.elastic.co/t/logstash-vs-nfs-null-characters/25918)

Thank you !

---

<div class="post-metadata">

**Author:** ![Ged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ged/32/65761_2.png) [@Ged](https://discuss.elastic.co/u/Ged)\
**Post date:** [January 20, 2018, 7:07pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/4 "2018-01-20T19:07:40Z")

</div>

Looks like reading file via NFS could cause this problem:  
[https://serverfault.com/questions/707034/disable-file-cache-for-mount-point](https://serverfault.com/questions/707034/disable-file-cache-for-mount-point)

Is it known issue ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 21, 2018, 12:10pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/5 "2018-01-21T12:10:49Z")

</div>

This does appear to be related. Storing log files on NFS sounds like a bad idea in general.

---

<div class="post-metadata">

**Author:** ![Ged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ged/32/65761_2.png) [@Ged](https://discuss.elastic.co/u/Ged)\
**Post date:** [January 22, 2018, 8:52am UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/6 "2018-01-22T08:52:23Z")

</div>

Thanks Magnus,

Do you mean storing remotely is bad idea generally or only using NFS ?  
I think most companies using remote log's storing as they have many instances of applications.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2018, 10:19am UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/7 "2018-01-23T10:19:29Z")

</div>

> Do you mean storing remotely is bad idea generally or only using NFS ?

Logging is a fundamental part of any application and you always want logging to work. You also don't want the logging itself to break the application. Therefore I'd always want log files to reside in a local volume so that the risk of failure is minimized.

> I think most companies using remote log's storing as they have many instances of applications.

I agree that many companies run many instances of applications but I don't think that's a valid reason for storing the logs on NFS.

---

<div class="post-metadata">

**Author:** ![Ged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ged/32/65761_2.png) [@Ged](https://discuss.elastic.co/u/Ged)\
**Post date:** [January 23, 2018, 10:41am UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/8 "2018-01-23T10:41:41Z")

</div>

You're right, logs should be stored as files locally and sent to centralized system in other way.

Solution I plan to use is to send data (as JSON single lines) from log4j2 to logstash input TCP or UDP plugin with json codec applied on input.

So i think we can close this thread.  
Many thanks for your explanations !

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2018, 11:30am UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/9 "2018-01-23T11:30:37Z")

</div>

> Solution I plan to use is to send data (as JSON single lines) from log4j2 to logstash input TCP or UDP plugin with json codec applied on input.

Then you're still dependent on the network. Write logs to local files and ship those files independently of your application.

---

<div class="post-metadata">

**Author:** ![Ged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ged/32/65761_2.png) [@Ged](https://discuss.elastic.co/u/Ged)\
**Post date:** [January 24, 2018, 2:16pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/10 "2018-01-24T14:16:30Z")

</div>

Yes, i know. Would be perfect to store files locally and additionally to send them remotely to Logstash for statistics purposes. But it not depends on me unfortunately.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2018, 2:16pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292/11 "2018-02-21T14:16:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
