# Courier Fetch Errors

**URL:** <https://discuss.elastic.co/t/courier-fetch-errors/74321>\
**Category:** Kibana\
**Created:** [February 8, 2017, 6:00am UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321 "2017-02-08T06:00:30Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 6:00am UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/1 "2017-02-08T06:00:30Z")

</div>

I recently upgraded my 6 data nodes from 4 CPUs to 8 CPUs each.

Now I am seeing these errors in Kibana that I've never seen before:

```
Shard Failures
The following shard failures ocurred:

Index: logstash-2017.02.08 Shard: 0 Reason: {"type":"transport_serialization_exception","reason":"Failed to deserialize response of type [org.elasticsearch.search.fetch.FetchSearchResult]","caused_by":{"type":"illegal_state_exception","reason":"unexpected byte [0x69]"}}

```

Any suggestions?

---

<div class="post-metadata">

**Author:** ![ppisljar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppisljar/32/11588_2.png) [@ppisljar](https://discuss.elastic.co/u/ppisljar)\
**Post date:** [February 8, 2017, 9:34am UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/2 "2017-02-08T09:34:52Z")

</div>

seems like an elastic search error .... could you try asking in the elastic search discuss group ?

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 2:39pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/3 "2017-02-08T14:39:52Z")

</div>

I did. Many posts I've found, point to Kibana. I don't know which is at fault, nor do I entirely care at this point. It is just another issue, for another made up reason, that I continue to have problems.

All my API checks show the ES cluster healthy.

I've spent months running down issue after issue, upgrading, rebuilding, etc.

All I did shut down my hosts, change from 4 CPUs to 8 CPUs and now....I get this stupid a$$ error randomly.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 4:01pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/4 "2017-02-08T16:01:37Z")

</div>

I was able to snag this:

```
Discover: Unable to parse/serialize body

 Less Info
OK
Error: Unable to parse/serialize body
    at respond (http://KIBANA/bundles/kibana.bundle.js?v=14695:14:6482)
    at checkRespForFailure (http://KIBANA/bundles/kibana.bundle.js?v=14695:14:6156)
    at http://KIBANA/bundles/kibana.bundle.js?v=14695:1:24479
    at processQueue (http://KIBANA/bundles/commons.bundle.js?v=14695:38:23621)
    at http://KIBANA/bundles/commons.bundle.js?v=14695:38:23888
    at Scope.$eval (http://KIBANA/bundles/commons.bundle.js?v=14695:39:4619)
    at Scope.$digest (http://KIBANA/bundles/commons.bundle.js?v=14695:39:2359)
    at Scope.$apply (http://KIBANA/bundles/commons.bundle.js?v=14695:39:5037)
    at done (http://KIBANA/bundles/commons.bundle.js?v=14695:37:25027)
    at completeRequest (http://KIBANA/bundles/commons.bundle.js?v=14695:37:28702)
```

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 5:01pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/5 "2017-02-08T17:01:32Z")

</div>

It's almost like I need to reinstall.

If that's the case, that's fine, I just need to know.

---

<div class="post-metadata">

**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\
**Post date:** [February 8, 2017, 5:36pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/6 "2017-02-08T17:36:10Z")

</div>

Hi @kopacko,

sorry to hear that you're encountering so many problems. The two messages you posted both look like Elasticsearch is not responding correctly. You said your health checks indicated that the cluster is healthy. Would you mind posting the responses of calling `http[s]://${YOURCLUSTER}/_cat/nodes?v` and `http[s]://${YOURCLUSTER}/_cat/indices?v`?

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 6:12pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/7 "2017-02-08T18:12:50Z")

</div>

Sure, give me a moment.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 6:17pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/8 "2017-02-08T18:17:41Z")

</div>

Here you go:

```
MASTERNODE-01:~$ curl -XGET 'http://localhost:9200/_cat/nodes?v'
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
x.x.x.x 28 97 24 2.90 2.56 2.40 di - DATANODE-04
x.x.x.x 50 96 47 2.78 2.77 2.65 di - DATANODE-06
x.x.x.x 2 75 0 0.00 0.00 0.00 m - MASTERNODE-02
x.x.x.x 2 75 0 0.00 0.01 0.01 m * MASTERNODE-03
x.x.x.x 19 99 23 3.11 2.99 2.87 di - DATANODE-01
x.x.x.x 6 97 15 2.12 2.33 2.48 di - DATANODE-05
x.x.x.x 43 90 0 0.00 0.00 0.00 - - KIBANANODE-01
x.x.x.x 7 98 38 2.45 2.60 2.84 di - DATANODE-02
x.x.x.x 52 96 22 2.38 2.15 2.07 di - DATANODE-03
x.x.x.x 2 75 0 0.00 0.00 0.00 m - MASTERNODE-01

MASTERNODE-01:~$ curl -XGET 'http://localhost:9200/_cat/indices?v&s=index'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana FeTfCUGPQXalqH2_ukmEOA 1 0 152 3 393.8kb 393.8kb
green open logstash-2017.01.30 -aIieugMQ2e2C1w3W4R_fw 6 0 3716 0 4.1mb 4.1mb
green open logstash-2017.01.31 YaJbI3DbTHCyazLZ2mXsVw 6 0 2782506 0 2.1gb 2.1gb
green open logstash-2017.02.01 UXJXqMxlQeWPhjdlWImVhA 6 0 144779115 0 240.5gb 240.5gb
green open logstash-2017.02.02 MGbreY1wSHGW59nU2aHQqw 6 0 140247069 0 239.6gb 239.6gb
green open logstash-2017.02.03 MyfveMTcQXCOcevpYL-REw 6 0 158329442 0 293.3gb 293.3gb
green open logstash-2017.02.04 T0dv7nM8Ssecrr0W9RbKHw 6 0 112505237 0 202.7gb 202.7gb
green open logstash-2017.02.05 5f6mnrjnQmOqA2BmBrjraw 6 0 106129925 0 189.3gb 189.3gb
green open logstash-2017.02.06 uIuBbhYxSbO6OkTQfrBvQQ 6 0 160544435 0 291.5gb 291.5gb
green open logstash-2017.02.07 euU_74VhTHizqCV8qN94og 6 0 138924007 5 260.7gb 260.7gb
green open logstash-2017.02.08 o4g_UvovQ62c1oc5fTH1ew 6 0 96698802 1 187.6gb 187.6gb
```

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 6:18pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/9 "2017-02-08T18:18:12Z")

</div>

Here are a few others:

```
MASTERNODE-01:~$ curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "CLUSTER",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 10,
  "number_of_data_nodes" : 6,
  "active_primary_shards" : 61,
  "active_shards" : 61,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

MASTERNODE-01:~$ curl -XGET 'http://localhost:9200/_cat/nodes?v&s=name&h=master,name,version,node.role,disk.avail,heap.max,ram.max,ram.current,cpu,uptime,jdk'
master name version node.role disk.avail heap.max ram.max ram.current cpu uptime jdk
- DATANODE-01 5.2.0 di 1.5tb 29.9gb 39.2gb 38.4gb 20 12.4h 1.8.0_121
- DATANODE-02 5.2.0 di 1.5tb 29.9gb 39.2gb 38.1gb 20 12.4h 1.8.0_121
- DATANODE-03 5.2.0 di 1.5tb 29.9gb 39.2gb 37.9gb 11 12.4h 1.8.0_121
- DATANODE-04 5.2.0 di 1.5tb 29.9gb 39.2gb 38.1gb 25 12.4h 1.8.0_121
- DATANODE-05 5.2.0 di 1.5tb 29.9gb 39.2gb 37.9gb 22 12.4h 1.8.0_121
- DATANODE-06 5.2.0 di 1.5tb 29.9gb 39.2gb 38.5gb 12 12.4h 1.8.0_121
- KIBANANODE-01 5.2.0 - 38.6gb 11.9gb 15.6gb 14.1gb 0 12.4h 1.8.0_121
- MASTERNODE-01 5.2.0 m 39.3gb 15.9gb 23.5gb 17.6gb 0 12.4h 1.8.0_121
- MASTERNODE-02 5.2.0 m 39.3gb 15.9gb 23.5gb 17.6gb 0 12.4h 1.8.0_121
* MASTERNODE-03 5.2.0 m 39.3gb 15.9gb 23.5gb 17.6gb 0 12.4h 1.8.0_121

MASTERNODE-01:~$ curl -XGET 'http://localhost:9200/_cat/nodes?v&s=name&h=master,name,indexing.index_total,indexing.index_current,indexing.index_failed,indexing.delete_total'
master name indexing.index_total indexing.index_current indexing.index_failed indexing.delete_total
- DATANODE-01 12310817 0 0 0
- DATANODE-02 12138538 0 8 0
- DATANODE-03 12316266 1 0 0
- DATANODE-04 12318182 1 0 0
- DATANODE-05 12313275 0 0 0
- DATANODE-06 12134054 2 8 0
- KIBANANODE-01 0 0 0 0
- MASTERNODE-01 0 0 0 0
- MASTERNODE-02 0 0 0 0
* MASTERNODE-03 0 0 0 0

MASTERNODE-01:~$ curl -XGET 'http://localhost:9200/_cat/nodes?v&s=name&h=master,name,merges.total,merges.current,merges.total_size,merges.current_size'
master name merges.total merges.current merges.total_size merges.current_size
- DATANODE-01 3335 1 82.3gb 957.5mb
- DATANODE-02 3198 0 82.1gb 0b
- DATANODE-03 3040 0 82.8gb 0b
- DATANODE-04 3106 2 81.3gb 1gb
- DATANODE-05 3107 0 82.9gb 0b
- DATANODE-06 3220 1 81gb 968.5mb
- KIBANANODE-01 0 0 0b 0b
- MASTERNODE-01 0 0 0b 0b
- MASTERNODE-02 0 0 0b 0b
* MASTERNODE-03 0 0 0b 0b

MASTERNODE-01:~$ curl -XGET 'http://localhost:9200/_cat/nodes?v&s=name&h=master,name,search.fetch_total,search.fetch_current,search.query_total,search.query_current'
master name search.fetch_total search.fetch_current search.query_total search.query_current
- DATANODE-01 63 0 14149 0
- DATANODE-02 63 0 14149 0
- DATANODE-03 60 0 14149 0
- DATANODE-04 60 0 14149 0
- DATANODE-05 61 0 14118 0
- DATANODE-06 17962 0 32048 0
- KIBANANODE-01 0 0 0 0
- MASTERNODE-01 0 0 0 0
- MASTERNODE-02 0 0 0 0
* MASTERNODE-03 0 0 0 0

MASTERNODE-01:~$ curl -XGET 'http://localhost:9200/_cat/nodes?v&s=name&h=master,name,segments.count,segments.memory,suggest.total,suggest.current'
master name segments.count segments.memory suggest.total suggest.current
- DATANODE-01 318 413.3mb 0 0
- DATANODE-02 312 406.6mb 0 0
- DATANODE-03 318 413.8mb 0 0
- DATANODE-04 302 410.3mb 0 0
- DATANODE-05 296 406.4mb 0 0
- DATANODE-06 299 407.3mb 0 0
- KIBANANODE-01 0 0b 0 0
- MASTERNODE-01 0 0b 0 0
- MASTERNODE-02 0 0b 0 0
* MASTERNODE-03 0 0b 0 0
```

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 6:58pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/10 "2017-02-08T18:58:24Z")

</div>

Oh and it is Kibana version 5.2.0.

```
KIBANANODE-01:~$ sudo /usr/share/kibana/bin/kibana --version
5.2.0
```

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 7:25pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/11 "2017-02-08T19:25:29Z")

</div>

The plot thickens:

```
Shard Failures
The following shard failures ocurred:

Index: logstash-2017.02.08 Shard: 0 Reason: {"type":"node_not_connected_exception","reason":"[DATANODE-05][x.x.x.x:9300] Node not connected"}

```

Soon as I saw this, I check the APIs and the node question was green, all shards accounted for, etc.

**EDIT UPDATE: I can repeat this error anytime I search for something in Kibana that does not exist.**

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 7:38pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/12 "2017-02-08T19:38:36Z")

</div>

I was able to pull this from a Logstash shipper's logs:

```
[2017-02-08T13:31:08,363][WARN][logstash.outputs.elasticsearch] Failed action.

 {:status=>500, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.02.08", :_type=>"logs", :_routing=>nil},

 ####<SYSLOG MESSAGE REMOVED>####],

 :response=>{"index"=>{"_index"=>"logstash-2017.02.08",
   "_type"=>"logs",
   "_id"=>"AVofNUFqw60d2OLj9AA8",
   "status"=>500,
   "error"=>{"type"=>"illegal_state_exception",
   "reason"=>"Message not fully read (request) for requestId [8882329],
   action [indices:data/write/bulk[s]],
   available [14377];

 resetting"}}}}
```

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 8, 2017, 9:36pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/13 "2017-02-08T21:36:43Z")

</div>

@ppisljar

New topic created for Logstash related errors that I think are tied to the Kibana errors.

> [@Failed Action](https://discuss.elastic.co/t/failed-action/74433):
>
> Last night I upgraded my Elasticsearch data nodes from 4 CPUs to 8 CPUs and now I am getting these errors from my Logstash shippers. Please help!!!!! [2017-02-08T15:31:14,344][WARN][logstash.outputs.elasticsearch] Failed action. {:status=\>500, :action=\>["index", {:\_id=\>nil, :\_index=\>"logstash-2017.02.08", :\_type=\>"logs", :\_routing=\>nil}, ####\<LOG MESSAGE REMOVED\>####, :response=\>{"index"=\>{"\_index"=\>"logstash-2017.02.08", "\_type"=\>"logs", "\_id"=\>"AVofozXYJ8M9dMOnBjdZ", "status"=\>500, …

---

<div class="post-metadata">

**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\
**Post date:** [February 9, 2017, 9:53am UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/14 "2017-02-09T09:53:19Z")

</div>

All those errors indicate that something is wrong with the communication within the Elasticsearch cluster. The log output from the Elasticsearch nodes (e.g. DATANODE-05) does not show anything that could be correlated to this? Where are you hosting your cluster? If it is AWS, there are known issues caused by the networking setup of the AWS Ubuntu images as described in [http://logz.io/blog/elasticsearch-cluster-disconnects/](http://logz.io/blog/elasticsearch-cluster-disconnects/).

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 9, 2017, 2:35pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/15 "2017-02-09T14:35:33Z")

</div>

On site servers.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 9, 2017, 2:42pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/16 "2017-02-09T14:42:42Z")

</div>

Well, there is more to the story.

Last night, I completely rebuilt these 10 servers. I deleted their disks and started from scratch.

I soon as I started the Logstash service on one of my shipping servers, I was immediately hit with these errors. On the ES side, I am getting errors on all 6 data nodes who are receiving the logs.

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 9, 2017, 3:05pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/17 "2017-02-09T15:05:54Z")

</div>

UPDATE: I had turned of shipping to one of my data center clusters while I finalized my setup with the 6 datanode cluster. In that, the CPU upgrade is the apparent root cause of the issues now. What I did, this morning, was turned off shipping to the 6 datanode cluster and pointed it back at my older, 4 datanode cluster that I had not upgraded the CPUs from 4 to 8 on. So far, as it stands, I have not received a SINGLE error.

---

<div class="post-metadata">

**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\
**Post date:** [February 9, 2017, 3:21pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/18 "2017-02-09T15:21:29Z")

</div>

What are the errors you are seeing in the Elasticsearch output?

---

<div class="post-metadata">

**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\
**Post date:** [February 9, 2017, 3:37pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/19 "2017-02-09T15:37:26Z")

</div>

Here you go:

```
[2017-02-08T23:45:05,054][DEBUG][o.e.a.b.TransportShardBulkAction] [KNOX-LOGDB-01] [logstash-2017.02.08][5] failed to execute bulk item (index) index {

[logstash-2017.02.08][wineventlog][AVohMGiU11AUrPoYaON-],

<#### LOG MESSAGE REMOVED ####>

}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [process_id]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:438) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:564) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:384) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:361) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:275) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:533) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:510) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:196) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:201) ~[elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:348) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:155) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:134) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:120) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:73) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:76) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:49) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:914) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:884) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:327) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:262) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:864) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:861) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1652) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:873) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:92) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:279) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:258) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:250) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) [elasticsearch-5.2.0.jar:5.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.0.jar:5.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.lang.NumberFormatException: For input string: "0x0000000000000c08"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_121]
        at java.lang.Long.parseLong(Long.java:589) ~[?:1.8.0_121]
        at java.lang.Long.parseLong(Long.java:631) ~[?:1.8.0_121]
        ... 40 more
```

---

<div class="post-metadata">

**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\
**Post date:** [February 9, 2017, 3:44pm UTC](https://discuss.elastic.co/t/courier-fetch-errors/74321/20 "2017-02-09T15:44:23Z")

</div>

So what this exception indicates is that the field `process_id` is set to a `long` type in the mapping, but the document being indexed contains the string "0x0000000000000c08" in that field. So either the mapping seems to be incorrect or Logstash seems to ship documents with incorrect field types. I am not sure how that could result in the cluster returning such errors as shown before, but maybe its a result of getting swamped with such indexing errors. Is there any error regarding the cluster memberships in the logs?

[Next page](https://discuss.elastic.co/t/courier-fetch-errors/74321.md?page=2)