# The logs aren't read by Logstash deployed with Docker after rotating

**URL:** <https://discuss.elastic.co/t/the-logs-arent-read-by-logstash-deployed-with-docker-after-rotating/311685>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [August 9, 2022, 4:27am UTC](https://discuss.elastic.co/t/the-logs-arent-read-by-logstash-deployed-with-docker-after-rotating/311685 "2022-08-09T04:27:51Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kent010341](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@kent010341](https://discuss.elastic.co/u/kent010341)\
**Post date:** [August 9, 2022, 4:27am UTC](https://discuss.elastic.co/t/the-logs-arent-read-by-logstash-deployed-with-docker-after-rotating/311685/1 "2022-08-09T04:27:51Z")

</div>

I'm using log4j to write logs to files, and whenever the log rotates (compressed to a .gz file), the new log file is never read by Logstash (using version 7.14.4, on MacOS 12.5).

I think it's important to track the inode number and sincedb file, so here are my records.

The `ls -li` result of the log folder was:

```auto
7435564 -rw-r--r-- 1 kent staff 137279 8 9 12:00 karaf.log

```

and after the log rotation:

```auto
7436195 -rw-r--r-- 1 kent staff 39322 8 9 12:01 karaf_2022-08-09.7.log.gz
7436188 -rw-r--r-- 1 kent staff 339420 8 9 12:01 karaf.log

```

However, in the folder (mounts the log folder) of the Logstash Docker container,

before:

```auto
7430919 -rw-r--r-- 1 logstash logstash 512008 Aug 9 03:47 karaf.log

```

after:

```auto
7436195 -rw-r--r-- 1 logstash logstash 39322 Aug 9 04:01 karaf_2022-08-09.7.log.gz
7430919 -rw-r--r-- 1 logstash logstash 512008 Aug 9 03:47 karaf.log

```

before and after rotation the sincedb are both

```auto
7430919 0 123 512008 1660016822.58378 /usr/share/logstash/proj_log/karaf.log

```

I think the log isn't read is because the inode number doesn't change, so Logstash keeps waiting for something newer than the current byte offset (`1660016822.58378`).

Does anyone know how to solve this problem?

Here's my logstash.conf

```auto
input {
  file {
    path => ["/usr/share/logstash/proj_log/karaf*.log"]
    start_position => "beginning"
    codec => multiline {
      pattern => "^\D"
      what => "previous"
    }
  }
  file {
    path => ["/usr/share/logstash/proj_log/karaf*.log.gz"]
    start_position => "beginning"
    mode => "read"
    codec => multiline {
      pattern => "^\D"
      what => "previous"
    }
  }
}

filter {
  grok {
    match => {
      "message" => "(?<timestamp>^\S[^\|]*\S)\s*\|\s*(?<level>\S[^\|]*\S)\s*\|\s*(?<thread>\S[^\|]*\S)\s*\|\s*(?<logger>\S[^\|]*\S)\s*\|\s*(?<bundle>\S[^\|]*\S)\s*\|\s*(?<msg>\S*.*)"
    }
    remove_field => ["message"]
  }

  fingerprint {
    concatenate_sources => true
    source => ["timestamp", "msg"]
    method => "MD5"
  }

  date {
    match => ["timestamp", "ISO8601"]
    remove_field => ["timestamp"]
    timezone => "Asia/Taipei"
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "karaf-%{+YYYY.MM.dd}"
    document_id => "%{fingerprint}"
  }
} 

```

---

<div class="post-metadata">

**Author:** ![kent010341](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@kent010341](https://discuss.elastic.co/u/kent010341)\
**Post date:** [August 11, 2022, 8:22am UTC](https://discuss.elastic.co/t/the-logs-arent-read-by-logstash-deployed-with-docker-after-rotating/311685/2 "2022-08-11T08:22:03Z")

</div>

Does anyone know how to fix it?

---

<div class="post-metadata">

**Author:** ![kent010341](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@kent010341](https://discuss.elastic.co/u/kent010341)\
**Post date:** [August 16, 2022, 5:57am UTC](https://discuss.elastic.co/t/the-logs-arent-read-by-logstash-deployed-with-docker-after-rotating/311685/3 "2022-08-16T05:57:23Z")

</div>

I just found another clue:

After the log file rotates, the contents of the log file (in my case, it is karaf.log) in the Logstash Docker container aren't the same as the contents of the log file in the folder mounted by the Logstash Docker container.

In short, after the log rotates, the file in the mounted folder remains with the old contents.

Therefore, unless I restart the Logstash container, the "new" karaf.log won't be read.

I think it might be a Docker issue. Does anyone met this problem before and know how to solve it?

Thanks.

Here's my logstash service at docker-compose.yml

```auto
logstash:
  image: docker.elastic.co/logstash/logstash:7.17.4
  container_name: logstash
  links:
    - elasticsearch
  volumes:
    - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    - $PROJ_DATA/log:/usr/share/logstash/proj_log:ro
  ports:
    - "5044:5044"

```

---

<div class="post-metadata">

**Author:** ![kent010341](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@kent010341](https://discuss.elastic.co/u/kent010341)\
**Post date:** [September 6, 2022, 2:45am UTC](https://discuss.elastic.co/t/the-logs-arent-read-by-logstash-deployed-with-docker-after-rotating/311685/4 "2022-09-06T02:45:59Z")

</div>

I found that running on Ubuntu 20.04 doesn't have this problem. The new log file keeps being read. Maybe running on macOS (arm64) have some bugs?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 6, 2022, 2:54am UTC](https://discuss.elastic.co/t/the-logs-arent-read-by-logstash-deployed-with-docker-after-rotating/311685/5 "2022-09-06T02:54:00Z")

</div>

It looks like this is a Docker issue while running on Mac, not a Logstash one.

You can check this [issue](https://github.com/docker/for-mac/issues/6219) about it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 4, 2022, 2:54am UTC](https://discuss.elastic.co/t/the-logs-arent-read-by-logstash-deployed-with-docker-after-rotating/311685/6 "2022-10-04T02:54:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.