# Understanding HA with Kafka

**URL:** <https://discuss.elastic.co/t/understanding-ha-with-kafka/116162>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 19, 2018, 5:10am UTC](https://discuss.elastic.co/t/understanding-ha-with-kafka/116162 "2018-01-19T05:10:35Z")\
**Posts on this page:** 4\
**Page:** 2

<div class="post-metadata">

**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\
**Post date:** [January 26, 2018, 5:15pm UTC](https://discuss.elastic.co/t/understanding-ha-with-kafka/116162/23 "2018-01-26T17:15:46Z")

</div>

I reported here: [https://jira.mesosphere.com/projects/DCOS\_OSS/issues/DCOS\_OSS-2112](https://jira.mesosphere.com/projects/DCOS_OSS/issues/DCOS_OSS-2112)

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [January 27, 2018, 2:17pm UTC](https://discuss.elastic.co/t/understanding-ha-with-kafka/116162/24 "2018-01-27T14:17:16Z")

</div>

Out of curiosity, does ignoring `stdout.logrotate.state` help with the lost documents?

Is this the 'state' file as used by [logrotate](https://linux.die.net/man/8/logrotate)? Depending on distribution you might also find this file at `/var/lib/logrotate.status` or `/var/lib/logrotate/status`.

Logrotates updates this file by first writing its state into a new temporary file and then using `mv` to replace the old state file with a new file. Filebeat detects this by inode change and would start reporting the status file from beginning. If the old status file has not been fully processed, contents is lost.

> When DC/OS log rotation happens DC/OS touches this file and makes 2 line entry in it.

Are these lines appended, or is the file truncated before?

---

<div class="post-metadata">

**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\
**Post date:** [January 27, 2018, 4:18pm UTC](https://discuss.elastic.co/t/understanding-ha-with-kafka/116162/25 "2018-01-27T16:18:33Z")

</div>

Hi @steffens, yes, as indicated using exclude\_files ["stdout.logrotate.state"] helps.

According to the DC/OS guys, the standard Linux logrotate is used. Each "containerized" application gets its own state file.

As far as I can tell, the file is truncated. It only ever contains 2 lines. Exactly this...

```
logrotate state -- version 2
"/var/lib/mesos/slave/slaves/be77bad4-de55-4c2a-b832-babfb5e43646-S1/frameworks/be77bad4-de55-4c2a-b832-babfb5e43646-0001/executors/auto-scale-prod-raange.8e08cb64-01fb-11e8-a0a1-bed7ac162486/runs/a33d447f-5e32-4b09-8e3a-7440a0d2ddd1/stdout" 2018-1-25-21:54:29
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 24, 2018, 4:18pm UTC](https://discuss.elastic.co/t/understanding-ha-with-kafka/116162/26 "2018-02-24T16:18:40Z")

</div>

Closed on February 24, 2018, 4:18pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/understanding-ha-with-kafka/116162.md?page=1)
