# Packetbeat - “ERR Failed to read integer reply: Expected digit”

**URL:** <https://discuss.elastic.co/t/packetbeat-err-failed-to-read-integer-reply-expected-digit/74352>\
**Category:** Beats\
**Tags:** packetbeat\
**Created:** [February 8, 2017, 9:52am UTC](https://discuss.elastic.co/t/packetbeat-err-failed-to-read-integer-reply-expected-digit/74352 "2017-02-08T09:52:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![moooofly](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@moooofly](https://discuss.elastic.co/u/moooofly)\
**Post date:** [February 8, 2017, 9:52am UTC](https://discuss.elastic.co/t/packetbeat-err-failed-to-read-integer-reply-expected-digit/74352/1 "2017-02-08T09:52:38Z")

</div>

when analyzing pcap file for **redis** protocol with `packetbeat`, I get outputs as follow:

```auto
➜ packetbeat git:(master) ✗ ./packetbeat -c ./packetbeat.yml -e -I redis_xg-bjdev-rediscluster-1_prot-7101_20161222110711_20161222110721.pcap -E packetbeat.protocols.redis.ports=7101 -t
...
2017/01/11 09:42:37.664290 protos.go:89: INFO registered protocol plugin: amqp
2017/01/11 09:42:37.664309 protos.go:89: INFO registered protocol plugin: http
2017/01/11 09:42:37.664315 protos.go:89: INFO registered protocol plugin: mysql
2017/01/11 09:42:37.664320 protos.go:89: INFO registered protocol plugin: redis
2017/01/11 09:42:37.665784 beat.go:207: INFO packetbeat start running.
2017/01/11 09:47:45.218365 redis_parse.go:306: ERR Failed to read integer reply: Expected digit
2017/01/11 09:42:38.430211 sniffer.go:384: INFO Input finish. Processed 40644 packets. Have a nice day!
2017/01/11 09:42:38.430657 util.go:48: INFO flows worker loop stopped
2017/01/11 09:42:38.430709 logp.go:245: INFO Total non-zero values: libbeat.publisher.published_events=8080 tcp.dropped_because_of_gaps=15 redis.unmatched_responses=15
2017/01/11 09:42:38.430722 logp.go:246: INFO Uptime: 909.957024ms
2017/01/11 09:42:38.430728 beat.go:211: INFO packetbeat stopped.
➜ packetbeat git:(master) ✗

```

The error message is " **_ERR Failed to read integer reply: Expected digit_**".

After analyzing with Wireshark in contrast, I find the root case behind this: **When REDIS response is big enough and network is not good enough, "_packet loss_" might happen** , which result in the ERROR above.

### Snapshot in my test:

 ![](https://raw.githubusercontent.com/moooofly/ImageCache/master/Pictures/HMGET%20%E7%9A%84%E5%BA%94%E7%AD%94%E6%95%B0%E6%8D%AE%E5%88%86%E5%8C%85%E5%9B%9E%E5%A4%8D%E9%81%87%E5%88%B0%E4%B8%A2%E5%8C%85%E9%97%AE%E9%A2%98.png)

According to the sequence of packets, I find：

- No.37642 - `HMGET` with key `hr-e0acd6e0-4c21-4917-a676-c4fd8094f2aa:user`

```auto
*10
$5
HMGET
$44
hr-e0acd6e0-4c21-4917-a676-c4fd8094f2aa:user
$5
18370
$5
52708
$1
0
$5
18370
$4
1117
$2
13
$3
153
$3
147

```

- No.37642 - _[TCP Previous segment not captured]_, this response is relative to `HMGET`'s last two fields.

```auto
:{"id":153,"email":"xiaojiao.xie@xxx","work_code":"E000027","mobile":186xxxx0925,"name":".........","walle_id":77287,"status":6,"pinyin_name":"xxj","sex":1,"security_level":60,"certificate_type":0,"certificate_number":"42068xxxxxxxxx3733","created_at":1431550029000,"updated_at":1449228237000,"nchr_id":"0001A910000000002EQP"}}
$519
{"userId":147,"userBuList":[3175],"tagsList":[],"userBuRoleDto":[{"id":81524,"bu_id":3175,"bu_name":"............BU","role_id":859,"role_name":".........","user_id":147,"user_name":"......"}],"user":{"id":147,"email":"xin.jin@xxxx","work_code":"E000029","mobile":186xxxx5626,"name":"......","walle_id":56063,"status":6,"pinyin_name":"jx","sex":1,"security_level":70,"certificate_type":0,"certificate_number":"420106xxxxxx510","created_at":1431550030000,"updated_at":1449228115000,"nchr_id":"0001A910000000002ERE"}}

```

- No.37648 - _[TCP Fast Retransmission]_, just retransmit the lost data segments.

```auto
*8
$532
{"userId":18370,"userBuList":[4594],"tagsList":[],"userBuRoleDto":[{"id":120993,"bu_id":4594,"bu_name":"..................","role_id":924,"role_name":"......","user_id":18370,"user_name":"......"}],"user":{"id":18370,"email":"hui.yaobj@xxx","work_code":"E019529","mobile":137xxxx8281,"name":"......","walle_id":23156752,"status":6,"pinyin_name":"yh","sex":1,"security_level":20,"certificate_type":0,"certificate_number":"13098xxxxxx033","created_at":1438657893000,"updated_at":1449228019000,"nchr_id":"0001A910000000013PM5"}}
...
$526
{"userId":153,"userBuList":[3174],"tagsList":[],"userBuRoleDto":[{"id":53306,"bu_id":3174,"bu_name":"............","role_id":922,"role_name":"......","user_id":153,"user_name":"........."}],"user"

```

In this case, redis\_parse.go module dose not work correctly, and packetbeat will stop running as soon as the ERROR happens.

so, I think it is a bug, or how can i fix it ? thanks in advance.

---

<div class="post-metadata">

**Author:** ![moooofly](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@moooofly](https://discuss.elastic.co/u/moooofly)\
**Post date:** [February 8, 2017, 9:59am UTC](https://discuss.elastic.co/t/packetbeat-err-failed-to-read-integer-reply-expected-digit/74352/2 "2017-02-08T09:59:35Z")

</div>

BTW, once the Redis response is split from another position, the Error message might be different I think.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [February 8, 2017, 12:33pm UTC](https://discuss.elastic.co/t/packetbeat-err-failed-to-read-integer-reply-expected-digit/74352/3 "2017-02-08T12:33:10Z")

</div>

the protocol analyzers in packetbeat must synchronize to the network stream. If they're not in sync, the parser might fail. In this case the internal state is dropped and a new parser instance is generated, in the hope of us being in sync with the next packet. As packet-loss might occur at any time, we have to drop the parser state in most cases and try to resync.

packetbeat stops running, because all packets have been send to the protcol analyzers. Not all events might be published yet. try `-waitstop 10s` to wait for 10 more seconds wether any events are stuck.

Without the original pcap and without being able to debug it myself I can not comment on any events you observed.

---

<div class="post-metadata">

**Author:** ![moooofly](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@moooofly](https://discuss.elastic.co/u/moooofly)\
**Post date:** [February 9, 2017, 6:51am UTC](https://discuss.elastic.co/t/packetbeat-err-failed-to-read-integer-reply-expected-digit/74352/4 "2017-02-09T06:51:44Z")

</div>

I did some work on my original pcap file, and split it into several parts.

- [37642-37651\_plrt\_s0\_pbErr.pcap](https://github.com/moooofly/pcaphub/blob/master/redis/37642-37651_plrt_s0_pbErr.pcap) is the one result in packetbeat ERROR as above.
- [18135-18144\_plrt\_s1.pcap](https://github.com/moooofly/pcaphub/blob/master/redis/18135-18144_plrt_s1.pcap) is almost same as last one, but `packetbeat` can process it correctly.
- [39212-39227\_ooo\_s0\_3req.pcap](https://github.com/moooofly/pcaphub/blob/master/redis/39212-39227_ooo_s0_3req.pcap) indicates that `packetbeat` can process out-of-order issue correctly.

so, my conclusion now is: when network is bad and redis response is big enough, something wrong might happen depending on splitting position from raw data.

> packetbeat stops running, because all packets have been send to the protcol analyzers.

It's right, I get it.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [February 9, 2017, 3:13pm UTC](https://discuss.elastic.co/t/packetbeat-err-failed-to-read-integer-reply-expected-digit/74352/5 "2017-02-09T15:13:57Z")

</div>

what means big enough? Currently packetbeat internally drops a stream if the active message exceeds 10MB I think.

---

<div class="post-metadata">

**Author:** ![moooofly](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@moooofly](https://discuss.elastic.co/u/moooofly)\
**Post date:** [February 10, 2017, 2:31am UTC](https://discuss.elastic.co/t/packetbeat-err-failed-to-read-integer-reply-expected-digit/74352/6 "2017-02-10T02:31:00Z")

</div>

"big enough" means that the raw data in redis response should be split into multiple segmented packets (i.e. [PSH, ACK] segment in [37642-37651\_plrt\_s0\_pbErr.pcap](https://github.com/moooofly/pcaphub/blob/master/redis/37642-37651_plrt_s0_pbErr.pcap)). Sorry about my description.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 10, 2017, 2:31am UTC](https://discuss.elastic.co/t/packetbeat-err-failed-to-read-integer-reply-expected-digit/74352/7 "2017-03-10T02:31:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.