# Logstash throwing error .Unexpected character (‘\<’ (code 60))

**URL:** <https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005>\
**Category:** Logstash\
**Created:** [August 13, 2019, 10:26am UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005 "2019-08-13T10:26:06Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![prabhat\_ranjan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhat_ranjan/32/44446_2.png) [@prabhat\_ranjan](https://discuss.elastic.co/u/prabhat_ranjan)\
**Post date:** [August 13, 2019, 10:26am UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/1 "2019-08-13T10:26:07Z")

</div>

Hi team,  
I am using ELK on the cloud. Using filebeat to send data to logstash. Logstash sends to ES which pushes it in Kibana.It appears most of the data is properly parsed and sent to Kibana successfully. There are instance now where i find few data's are missing in Kibana. I am getting json parse error though JSON body looks good and validated.

I see logstash error as below:

Error parsing json {:source=\>"message", :raw=\>"{"timestamp":"2019-08-13T07:10:10.713Z", "type":"CacheRedis", "name":"insertWatsonRecord", "body":{"operation":"insertWatsonRecord", "table":"watsonIntentAndEntity", "query":{"intents":\<Java::JavaUtil::ArrayList:-302032833 [{"intent":"greet", "confidence":1}]\>, "entities":\<Java::JavaUtil::ArrayList:1 \>, "input":{"text":"hi"}, "output":{"nodes\_visited":\<Java::JavaUtil::ArrayList:1 \>, "warning":"No dialog node condition matched to true in the last dialog round - context.nodes\_visited is empty. Falling back to the root node in the next round.", "log\_messages":\<Java::JavaUtil::ArrayList:1910241773 [{"level":"warn", "msg":"No dialog node condition matched to true in the last dialog round - context.nodes\_visited is empty. Falling back to the root node in the next round."}]\>, "generic":\<Java::JavaUtil::ArrayList:1 \>, "text":\<Java::JavaUtil::ArrayList:1 \>}, "context":{"conversation\_id":"97dee768-da32-490a-aa79-0e103df828aa", "system":{"dialog\_request\_counter":1, "initialized":true, "dialog\_stack":\<Java::JavaUtil::ArrayList:22214298 [{"dialog\_node":"root"}]\>, "dialog\_turn\_counter":1}}}, "startTime":1565680210701}, "duration":12, "success":true, "itemType":"dependency"}", :exception=\>#

**\<LogStash::Json:🙂 ParserError: Unexpected character ('\<' (code 60)): expected** a valid value (number, String, array, object, 'true', 'false' or 'null')  
at [Source: (byte)"{"timestamp":"2019-08-13T07:10:10.713Z", "type":"CacheRedis", "name":"insertWatsonRecord", "body":{"operation":"insertWatsonRecord", "table":"watsonIntentAndEntity", "query":{"intents":\<Java::JavaUtil::ArrayList:-302032833 [{"intent":"greet", "confidence":1}]\>, "entities":\<Java::JavaUtil::ArrayList:1 \>, "input":{"text":"hi"}, "output":{"nodes\_visited":\<Java::JavaUtil::ArrayList:1 \>, "warning":"No dialog node condition matched to true in the last dialog round - context.nodes\_visited is empty."[truncated 686 bytes]; line: 1, column: 187]\>}

Filebeat log:-  
I verified the json body present in filebeat and it seems to be valid .  
{  
"timestamp":"2019-08-12T12:39:25.255Z",  
"type":"CacheRedis",  
"name":"insertWatsonRecord",  
"body":"{\n "operation": "insertWatsonRecord",\n "table": "watsonIntentAndEntity",\n "query": "{\"intents\":[{\"intent\":\"greet\",\"confidence\":1}],\"entities\":,\"input\":{\"text\":\"hi\"},\"output\":{\"generic\":,\"text\":,\"nodes\_visited\":,\"warning\":\"No dialog node condition matched to true in the last dialog round - context.nodes\_visited is empty. Falling back to the root node in the next round.\",\"log\_messages\":[{\"level\":\"warn\",\"msg\":\"No dialog node condition matched to true in the last dialog round - context.nodes\_visited is empty. Falling back to the root node in the next round.\"}]},\"context\":{\"timezone\":\"America/New\_York\",\"conversation\_id\":\"cd40ae3c-f12d-4c5a-aa71-0bfd2c05e984\",\"system\":{\"initialized\":true,\"dialog\_stack\":[{\"dialog\_node\":\"root\"}],\"dialog\_turn\_counter\":1,\"dialog\_request\_counter\":1}}}",\n "startTime": 1565613565251\n}",  
"duration":4,  
"success":true,  
"itemType":"dependency"  
}

Here is my logstash.yml

logstash.conf: |  
# Sample Logstash configuration for creating a simple  
# Beats -\> Logstash -\> Elasticsearch pipeline.

```
input {
  beats {
   client_inactivity_timeout => 3000
   port => 5044
   ssl => false
        }
  
      }
      
filter {

  **mutate {**

```

\*\* gsub =\> [\*\*  
\*\* "message", '=\>', ':',\*\*  
\*\* "message", ':,', ':,'\*\*  
\*\* ]\*\*  
\*\* }\*\*

```
json {
       source => "message"
  }
}

output {
  elasticsearch {
  hosts => [" *******"]
  index => "max-log-%{+YYYY.MM.dd}"
  user => " ****"
  password => "***"
  cacert => ["***"]
  ssl => true

} 

stdout {
  codec => json
  }

```

Here is filebeat:  
data:  
filebeat.yml: |-  
filebeat.autodiscover:  
providers:  
- type: kubernetes  
include\_pod\_uid: true  
in\_cluster: true  
hints.enabled: true  
include\_annotations: '\*'  
templates:  
- condition.regexp:  
kubernetes.container.name: '{{ condition.kubernetescontainername }}' config: - type: docker combine\_partial: true cri.parse\_flags: true cri.force: true containers: path: "{{ .Values.containers.logpath1 }}" ids: - "{data.kubernetes.container.name}"

```
                path: "{{ .Values.containers.logpath2 }}"
              
processors:
  - add_kubernetes_metadata:
      in_cluster: true
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 8

output.logstash:
  hosts: ["{{ $creds.servicename }}.{{ $creds.namespace }}.svc.cluster.local:{{ $creds.port }}"]

```

Is it something with mutate in logstash.yml ? Kindly correct me if anything is wrong here.I assume it is because of array patterns present in my json body. Here is pattern for which it is failing continuously.

{  
"timestamp":"2019-08-12T12:39:19.384Z",  
"name":"AnalyticsWorkflow",  
"body":"{\n "header": {\n "user": "98ab7383281d0f2892c297ef99fc5634",\n "fullname": "b738bb2e7d5b69c73f96c6b9c629b707",\n "fragmentID": 0,\n "channel": "directline",\n "tenant": "ibmqa",\n "startTime": "2019-08-12T12:39:17.142Z",\n "bot": {\n "Name": "IBMCloudQA"\n }\n },\n "workflow": {\n "step": [\n {\n "value": "tips",\n "message": "{\"conversationIntent\":\"greet\",\"subIntent\":\"greetTips\"}",\n "elapsedTime": 1,\n "time": "2019-08-12T12:39:17.143Z"\n }\n]\n },\n "nlu": {\n "userText": "hi",\n "intent": {\n "name": "greet",\n "confidence": 1\n },\n "entities": {}\n },\n "endEvent": "clearConversation"\n}",  
"itemType":"customEvent"  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2019, 12:52pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/2 "2019-08-13T12:52:38Z")

</div>

> [@prabhat\_ranjan](#):
>
> "query":{"intents":\<Java::JavaUtil::ArrayList:-302032833 [{"intent":"greet", "confidence":1}]\>

That's not valid JSON. The json filter is complaining when it reaches the \<

---

<div class="post-metadata">

**Author:** ![prabhat\_ranjan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhat_ranjan/32/44446_2.png) [@prabhat\_ranjan](https://discuss.elastic.co/u/prabhat_ranjan)\
**Post date:** [August 13, 2019, 12:59pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/3 "2019-08-13T12:59:33Z")

</div>

hi it is valid,just that while pasting it is not taking proper format

---

<div class="post-metadata">

**Author:** ![prabhat\_ranjan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhat_ranjan/32/44446_2.png) [@prabhat\_ranjan](https://discuss.elastic.co/u/prabhat_ranjan)\
**Post date:** [August 13, 2019, 1:02pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/4 "2019-08-13T13:02:25Z")

</div>

```
{
	"timestamp": "2019-08-12T12:39:19.384Z",
	"name": "AnalyticsWorkflow",
	"body": "{\n \"header\": {\n \"user\": \"98ab7383281d0f2892c297ef99fc5634\",\n \"fullname\": \"b738bb2e7d5b69c73f96c6b9c629b707\",\n \"fragmentID\": 0,\n \"channel\": \"directline\",\n \"tenant\": \"ibmqa\",\n \"startTime\": \"2019-08-12T12:39:17.142Z\",\n \"bot\": {\n \"Name\": \"IBMCloudQA\"\n }\n },\n \"workflow\": {\n \"step\": [\n {\n \"value\": \"tips\",\n \"message\": \"{\\\"conversationIntent\\\":\\\"greet\\\",\\\"subIntent\\\":\\\"greetTips\\\"}\",\n \"elapsedTime\": 1,\n \"time\": \"2019-08-12T12:39:17.143Z\"\n }\n]\n },\n \"nlu\": {\n \"userText\": \"hi\",\n \"intent\": {\n \"name\": \"greet\",\n \"confidence\": 1\n },\n \"entities\": {}\n },\n \"endEvent\": \"clearConversation\"\n}",
	"itemType": "customEvent"
}
```

---

<div class="post-metadata">

**Author:** ![prabhat\_ranjan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhat_ranjan/32/44446_2.png) [@prabhat\_ranjan](https://discuss.elastic.co/u/prabhat_ranjan)\
**Post date:** [August 13, 2019, 1:25pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/5 "2019-08-13T13:25:24Z")

</div>

My json object contains array and logstash not able to read /parse the same.I am not getting issue for normal bodies but there should be way where logstash can parse the json correctly if it contains multi array .

Any suggestion here would be appreciated.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2019, 1:43pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/6 "2019-08-13T13:43:12Z")

</div>

Again, this is not valid JSON. If you edit your post, select the JSON, and click on \</\> it may then display valid JSON. Or not.

---

<div class="post-metadata">

**Author:** ![prabhat\_ranjan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhat_ranjan/32/44446_2.png) [@prabhat\_ranjan](https://discuss.elastic.co/u/prabhat_ranjan)\
**Post date:** [August 14, 2019, 5:14am UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/7 "2019-08-14T05:14:57Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/7/a/7a9439a5372f329db7f0211befe2d7f3a2f217df.png)

i have verified on JSONlint. pls see it saying valid json. i will re-check but we are parsing the body and validating before sending to filebeat.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 14, 2019, 12:51pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/8 "2019-08-14T12:51:23Z")

</div>

OK, that is valid JSON and a json filter will parse it. What problem are you having?

---

<div class="post-metadata">

**Author:** ![prabhat\_ranjan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhat_ranjan/32/44446_2.png) [@prabhat\_ranjan](https://discuss.elastic.co/u/prabhat_ranjan)\
**Post date:** [August 14, 2019, 6:04pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/9 "2019-08-14T18:04:35Z")

</div>

Same error which i posted earlier....unexpected character . same as below:  
Error parsing json {:source=\>"message", :raw=\>"{"name":"Redis", "body":{"operation":"insertWatsonRecord", "table":"watsonIntentAndEntity", "query":{"intents":\<Java::JavaUtil::ArrayList:-302032833 [{"confidence":1, "intent":"greet"}]\>, "entities":\<Java::JavaUtil::ArrayList:1

Can you please help me with this settings.If anything is incorrect here. Am pushing data from kubernetes.pod.log to filebeat,then logstash and further.  
processors:  
- add\_kubernetes\_metadata:  
in\_cluster: true  
- decode\_json\_fields:  
fields: ["message"]  
process\_array: true  
max\_depth: 8

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 14, 2019, 6:50pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/10 "2019-08-14T18:50:08Z")

</div>

And once again - that is not valid JSON and you should not expect a json filter to parse it.

---

<div class="post-metadata">

**Author:** ![prabhat\_ranjan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhat_ranjan/32/44446_2.png) [@prabhat\_ranjan](https://discuss.elastic.co/u/prabhat_ranjan)\
**Post date:** [August 19, 2019, 7:55am UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/11 "2019-08-19T07:55:30Z")

</div>

hey i was away from work for some time. As i mentioned earlier,i tested the json body and it looks good to me.  
One question,is Kubernetes adds any metadata to file beat or log stash while proccessing the JSON body?  
Application is creating proper json (i verified them on multiple times). This JSON goes to POD consoles and gets dumped into log files generated by kubernetes container. These log files are input to filebeats and logstash.  
Do you think that kubernetes container adds any metadata to JSON body while building the log files?  
PS: if i run this setup on local(excludes Kubernetes) it work absolutely fine and i do not see any JSON parse errors.  
Any suggestions would be great here.

---

<div class="post-metadata">

**Author:** ![prabhat\_ranjan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhat_ranjan/32/44446_2.png) [@prabhat\_ranjan](https://discuss.elastic.co/u/prabhat_ranjan)\
**Post date:** [August 21, 2019, 12:14pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/12 "2019-08-21T12:14:58Z")

</div>

Any suggestion here would be great. On my local machine,we are able to parse the json successfully but on the kubernetes world ,log stash is not able to parse the same json.

Is it possible that kubernetes adds some metadata here? Or is there any known issue with the parsing on containers.

my filebeat settings:

filebeat.autodiscover:  
providers:  
- type: kubernetes  
include\_pod\_uid: true  
in\_cluster: true  
hints.enabled: true  
include\_annotations: '\*'  
templates:  
- condition.regexp:  
kubernetes.container.name: '{{ condition.kubernetescontainername }}' config: - type: docker combine\_partial: true cri.parse\_flags: true cri.force: true containers: path: "{{ .Values.containers.logpath1 }}" ids: - "{data.kubernetes.container.name}"

```
                path: "{{ .Values.containers.logpath2 }}"
              
processors:
  - add_kubernetes_metadata:
      in_cluster: true
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 8

```

Here is again my sample json body which is failing on cloud.

`{"timestamp":"2019-08-21T07:15:30.149Z","type":"CacheRedis","name":"insertWatsonRecord","body":"{\n \"operation\": \"insertWatsonRecord\",\n \"table\": \"watsonIntentAndEntity\",\n \"query\": \"{\\\"intents\\\":[{\\\"intent\\\":\\\"no\\\",\\\"confidence\\\":1}],\\\"entities\\\":[],\\\"input\\\":{\\\"text\\\":\\\"dislike\\\"},\\\"output\\\":{\\\"generic\\\":[],\\\"text\\\":[],\\\"nodes_visited\\\":[],\\\"warning\\\":\\\"No dialog node condition matched to true in the last dialog round - context.nodes_visited is empty. Falling back to the root node in the next round.\\\",\\\"log_messages\\\":[{\\\"level\\\":\\\"warn\\\",\\\"msg\\\":\\\"No dialog node condition matched to true in the last dialog round - context.nodes_visited is empty. Falling back to the root node in the next round.\\\"}]},\\\"context\\\":{\\\"timezone\\\":\\\"America/New_York\\\",\\\"conversation_id\\\":\\\"5151b74a-f54a-44ba-88ae-c24b0124ce30\\\",\\\"system\\\":{\\\"initialized\\\":true,\\\"dialog_stack\\\":[{\\\"dialog_node\\\":\\\"root\\\"}],\\\"dialog_turn_counter\\\":1,\\\"dialog_request_counter\\\":1}}}\",\n \"startTime\": 1566371716767\n}","duration":13382,"success":true,"itemType":"dependency"}`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 18, 2019, 12:15pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005/13 "2019-09-18T12:15:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.