# \[Elastic Cloud\] Shipped JSON logs sometimes not parsed

**URL:** <https://discuss.elastic.co/t/elastic-cloud-shipped-json-logs-sometimes-not-parsed/251382>\
**Category:** Elasticsearch\
**Created:** [October 8, 2020, 7:06am UTC](https://discuss.elastic.co/t/elastic-cloud-shipped-json-logs-sometimes-not-parsed/251382 "2020-10-08T07:06:51Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [October 8, 2020, 7:06am UTC](https://discuss.elastic.co/t/elastic-cloud-shipped-json-logs-sometimes-not-parsed/251382/1 "2020-10-08T07:06:51Z")

</div>

Recently, Elastic Cloud has added the ability to also ship logs to a remote monitoring cluster. Enabling that feature worked great, however, I'm seeing parsing issues in the `elasticsearch.server` dataset.

Two subsequent log documents sourced from the same node (let's call it instance-44) and the same JSON log file (`/app/logs/fb503f***_server.json`) are not parsed the same way. All fields are pretty much identical except for the `message` field. In the first document, it looks like this:

```auto
"@timestamp" : "2020-10-08T05:16:55.630Z",
"message" : "duration [999ms], collections [1]/[1.3s], total [999ms]/[2s], memory [9.1gb]->[9.2gb]/[29gb], all_pools {[young] [712mb]->[16mb]/[0b]}{[old] [6.5gb]->[8.8gb]/[29gb]}{[survivor] [1.8gb]->[472mb]/[0b]}",

```

While in the second document, like this:

```auto
"@timestamp" : "2020-10-08T05:16:55.639Z",
"message" : """{"type": "server", "timestamp": "2020-10-08T05:16:55,639Z", "level": "WARN", "component": "o.e.m.j.JvmGcMonitorService", "cluster.name": "fb503f***", "node.name": "instance-0000000044", "message": "[gc][3308] overhead, spent [999ms] collecting in the last [1.3s]", "cluster.uuid": "_TWTJE05***", "node.id": "fug7snda***" }""",

```

Instead the message field should be:

```auto
"message": "[gc][3308] overhead, spent [999ms] collecting in the last [1.3s]"

```

All the fields from that stringified JSON are actually in the end document, so it looks like the `message` field was indeed parsed, however, it was not replaced with the actual message `[gc][3308] overhead...`

When looking those two logs directly in the ES Cloud Console, they both look right, though:

 ![Capture d’écran 2020-10-08 à 08.52.44](https://us1.discourse-cdn.com/elastic/original/3X/7/8/78603e6f5b1ce4d9cdd172b3cd9d58cfa5ab9c99.png)

Also worth noting that the index called `elastic-cloud-logs-7-2020.10.08-000001` has no `index.final_pipeline` or `index.default_pipeline` set up that could potentially fail to do the parsing job, the issue must be happening on the Filebeat side that is installed within Elastic Cloud.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 5, 2020, 7:07am UTC](https://discuss.elastic.co/t/elastic-cloud-shipped-json-logs-sometimes-not-parsed/251382/2 "2020-11-05T07:07:11Z")

</div>

Closed on November 5, 2020, 7:07am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.