# Logstash still holding onto deleted logstash application logs

**URL:** <https://discuss.elastic.co/t/logstash-still-holding-onto-deleted-logstash-application-logs/325479>\
**Category:** Logstash\
**Created:** [February 14, 2023, 1:42pm UTC](https://discuss.elastic.co/t/logstash-still-holding-onto-deleted-logstash-application-logs/325479 "2023-02-14T13:42:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shreesh\_Narayanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreesh_narayanan/32/87312_2.png) [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\
**Post date:** [February 14, 2023, 1:42pm UTC](https://discuss.elastic.co/t/logstash-still-holding-onto-deleted-logstash-application-logs/325479/1 "2023-02-14T13:42:55Z")

</div>

Hello,

It seems logstash refuses to let go of deleted logs (logstash's own logs) and this takes up all the space on disks , until a service restart takes place . Is there a way , we could fix this ? Is something need to be done on log4j.properties file?

```auto

root@prod-nexus-app02:~# lsof | grep -i deleted
none 668 root txt REG 0,1 17032 33859 / (deleted)
java 173958 root 1w REG 253,0 14309790752 5242902 /app/logstash/logs/logstash-plain-2023-02-14-1.log (deleted)
java 173958 root 2w REG 253,0 14309790752 5242902 /app/logstash/logs/logstash-plain-2023-02-14-1.log (deleted)
java 173958 root 68w REG 253,0 0 5242885 /app/logstash/logs/logstash-json.log (deleted)
java 173958 root 69w REG 253,0 0 5242886 /app/logstash/logs/logstash-slowlog-plain.log (deleted)
java 173958 root 70w REG 253,0 464 5242887 /app/logstash/logs/logstash-deprecation.log (deleted)
java 173958 root 71w REG 253,0 0 5242888 /app/logstash/logs/logstash-slowlog-json.log (deleted)
java 173958 174001 LogStash: root 1w REG 253,0 14309790752 5242902 /app/logstash/logs/logstash-plain-2023-02-14-1.log (deleted)
java 173958 174001 LogStash: root 2w REG 253,0 14309790752 5242902 /app/logstash/logs/logstash-plain-2023-02-14-1.log (deleted)
java 173958 174001 LogStash: root 68w REG 253,0 0 5242885 /app/logstash/logs/logstash-json.log (deleted)
java 173958 174001 LogStash: root 69w REG 253,0 0 5242886 /app/logstash/logs/logstash-slowlog-plain.log (deleted)
java 173958 174001 LogStash: root 70w REG 253,0 464 5242887 /app/logstash/logs/logstash-deprecation.log (deleted)
java 173958 174001 LogStash: root 71w REG 253,0 0 5242888 /app/logstash/logs/logstash-slowlog-json.log (deleted)
java 173958 174002 GC\x20Thr root 1w REG 253,0 14309790752 5242902 /app/logstash/logs/logstash-plain-2023-02-14-1.log (deleted)
java 173958 174002 GC\x20Thr root 2w REG 253,0 14309790752 5242902 /app/logstash/logs/logstash-plain-2023-02-14-1.log (deleted)

```

---

<div class="post-metadata">

**Author:** ![Shreesh\_Narayanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreesh_narayanan/32/87312_2.png) [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\
**Post date:** [February 14, 2023, 1:49pm UTC](https://discuss.elastic.co/t/logstash-still-holding-onto-deleted-logstash-application-logs/325479/2 "2023-02-14T13:49:54Z")

</div>

Here's my log4j.properties contents ..I guess log4j takes care of logstash application log rotation , is there something missing here ?

```auto
root@prod-nexus-app02:~# cat /etc/logstash/log4j2.properties
status = error
name = LogstashPropertiesConfig

appender.console.type = Console
appender.console.name = plain_console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n

appender.json_console.type = Console
appender.json_console.name = json_console
appender.json_console.layout.type = JSONLayout
appender.json_console.layout.compact = true
appender.json_console.layout.eventEol = true

appender.rolling.type = RollingFile
appender.rolling.name = plain_rolling
appender.rolling.fileName = ${sys:ls.logs}/logstash-plain.log
appender.rolling.filePattern = ${sys:ls.logs}/logstash-plain-%d{yyyy-MM-dd}-%i.log.gz
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size = 100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 30
appender.rolling.avoid_pipelined_filter.type = PipelineRoutingFilter

appender.json_rolling.type = RollingFile
appender.json_rolling.name = json_rolling
appender.json_rolling.fileName = ${sys:ls.logs}/logstash-json.log
appender.json_rolling.filePattern = ${sys:ls.logs}/logstash-json-%d{yyyy-MM-dd}-%i.log.gz
appender.json_rolling.policies.type = Policies
appender.json_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.json_rolling.policies.time.interval = 1
appender.json_rolling.policies.time.modulate = true
appender.json_rolling.layout.type = JSONLayout
appender.json_rolling.layout.compact = true
appender.json_rolling.layout.eventEol = true
appender.json_rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.json_rolling.policies.size.size = 100MB
appender.json_rolling.strategy.type = DefaultRolloverStrategy
appender.json_rolling.strategy.max = 30
appender.json_rolling.avoid_pipelined_filter.type = PipelineRoutingFilter

appender.routing.type = PipelineRouting
appender.routing.name = pipeline_routing_appender
appender.routing.pipeline.type = RollingFile
appender.routing.pipeline.name = appender-${ctx:pipeline.id}
appender.routing.pipeline.fileName = ${sys:ls.logs}/pipeline_${ctx:pipeline.id}.log
appender.routing.pipeline.filePattern = ${sys:ls.logs}/pipeline_${ctx:pipeline.id}.%i.log.gz
appender.routing.pipeline.layout.type = PatternLayout
appender.routing.pipeline.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %m%n
appender.routing.pipeline.policy.type = SizeBasedTriggeringPolicy
appender.routing.pipeline.policy.size = 100MB
appender.routing.pipeline.strategy.type = DefaultRolloverStrategy
appender.routing.pipeline.strategy.max = 30

rootLogger.level = ${sys:ls.log.level}
rootLogger.appenderRef.console.ref = ${sys:ls.log.format}_console
rootLogger.appenderRef.rolling.ref = ${sys:ls.log.format}_rolling
rootLogger.appenderRef.routing.ref = pipeline_routing_appender

# Slowlog

appender.console_slowlog.type = Console
appender.console_slowlog.name = plain_console_slowlog
appender.console_slowlog.layout.type = PatternLayout
appender.console_slowlog.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %m%n

appender.json_console_slowlog.type = Console
appender.json_console_slowlog.name = json_console_slowlog
appender.json_console_slowlog.layout.type = JSONLayout
appender.json_console_slowlog.layout.compact = true
appender.json_console_slowlog.layout.eventEol = true

appender.rolling_slowlog.type = RollingFile
appender.rolling_slowlog.name = plain_rolling_slowlog
appender.rolling_slowlog.fileName = ${sys:ls.logs}/logstash-slowlog-plain.log
appender.rolling_slowlog.filePattern = ${sys:ls.logs}/logstash-slowlog-plain-%d{yyyy-MM-dd}-%i.log.gz
appender.rolling_slowlog.policies.type = Policies
appender.rolling_slowlog.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling_slowlog.policies.time.interval = 1
appender.rolling_slowlog.policies.time.modulate = true
appender.rolling_slowlog.layout.type = PatternLayout
appender.rolling_slowlog.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %m%n
appender.rolling_slowlog.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling_slowlog.policies.size.size = 100MB
appender.rolling_slowlog.strategy.type = DefaultRolloverStrategy
appender.rolling_slowlog.strategy.max = 30

appender.json_rolling_slowlog.type = RollingFile
appender.json_rolling_slowlog.name = json_rolling_slowlog
appender.json_rolling_slowlog.fileName = ${sys:ls.logs}/logstash-slowlog-json.log
appender.json_rolling_slowlog.filePattern = ${sys:ls.logs}/logstash-slowlog-json-%d{yyyy-MM-dd}-%i.log.gz
appender.json_rolling_slowlog.policies.type = Policies
appender.json_rolling_slowlog.policies.time.type = TimeBasedTriggeringPolicy
appender.json_rolling_slowlog.policies.time.interval = 1
appender.json_rolling_slowlog.policies.time.modulate = true
appender.json_rolling_slowlog.layout.type = JSONLayout
appender.json_rolling_slowlog.layout.compact = true
appender.json_rolling_slowlog.layout.eventEol = true
appender.json_rolling_slowlog.policies.size.type = SizeBasedTriggeringPolicy
appender.json_rolling_slowlog.policies.size.size = 100MB
appender.json_rolling_slowlog.strategy.type = DefaultRolloverStrategy
appender.json_rolling_slowlog.strategy.max = 30

logger.slowlog.name = slowlog
logger.slowlog.level = trace
logger.slowlog.appenderRef.console_slowlog.ref = ${sys:ls.log.format}_console_slowlog
logger.slowlog.appenderRef.rolling_slowlog.ref = ${sys:ls.log.format}_rolling_slowlog
logger.slowlog.additivity = false

logger.licensereader.name = logstash.licensechecker.licensereader
logger.licensereader.level = error

# Silence http-client by default
logger.apache_http_client.name = org.apache.http
logger.apache_http_client.level = fatal

# Deprecation log
appender.deprecation_rolling.type = RollingFile
appender.deprecation_rolling.name = deprecation_plain_rolling
appender.deprecation_rolling.fileName = ${sys:ls.logs}/logstash-deprecation.log
appender.deprecation_rolling.filePattern = ${sys:ls.logs}/logstash-deprecation-%d{yyyy-MM-dd}-%i.log.gz
appender.deprecation_rolling.policies.type = Policies
appender.deprecation_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.deprecation_rolling.policies.time.interval = 1
appender.deprecation_rolling.policies.time.modulate = true
appender.deprecation_rolling.layout.type = PatternLayout
appender.deprecation_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n
appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.deprecation_rolling.policies.size.size = 100MB
appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy
appender.deprecation_rolling.strategy.max = 30

logger.deprecation.name = org.logstash.deprecation, deprecation
logger.deprecation.level = WARN
logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_plain_rolling
logger.deprecation.additivity = false

logger.deprecation_root.name = deprecation
logger.deprecation_root.level = WARN
logger.deprecation_root.appenderRef.deprecation_rolling.ref = deprecation_plain_rolling
logger.deprecation_root.additivity = false

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 14, 2023, 6:06pm UTC](https://discuss.elastic.co/t/logstash-still-holding-onto-deleted-logstash-application-logs/325479/3 "2023-02-14T18:06:03Z")

</div>

> [@Shreesh\_Narayanan](#):
>
> It seems logstash refuses to let go of deleted logs (logstash's own logs) and this takes up all the space on disks , until a service restart takes place

That is the way UNIX filesystems work. A program can open a temporary file, delete it, then continue to use it for temporary storage until it exits, knowing that the filesystem will release the space when it exits, so it doesn't have to free it up itself (which might be impossible if it crashes). If you only have a couple of megabytes of memory (like the PDP-11 back in the 1970s) then the ability to extend storage to disk is sometimes useful.

Once this contract between the filesystem and the programmer was set (back in the 70's), it became impossible to change. The space used by a file is only freed when the last reference to it is deleted.

---

<div class="post-metadata">

**Author:** ![Shreesh\_Narayanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreesh_narayanan/32/87312_2.png) [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\
**Post date:** [March 6, 2023, 3:09pm UTC](https://discuss.elastic.co/t/logstash-still-holding-onto-deleted-logstash-application-logs/325479/4 "2023-03-06T15:09:12Z")

</div>

I did find that systemd unit file (customized) had a hold on the logging files, once that was rectified . The behavior has gone away ... . Many thanks @Badger🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 3, 2023, 3:09pm UTC](https://discuss.elastic.co/t/logstash-still-holding-onto-deleted-logstash-application-logs/325479/5 "2023-04-03T15:09:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.