How to fix "rm: cannot remove '/usr/share/filebeat/bin/data/registry/filebeat': Directory not empty"?

[kensaku-okada]

I made a docker which 1) let a shellscript repetitively run Filebeat to read a file and transmit its data to the local Logstash and 2) run a logstash to transfer the received data to a web server.

Since I wanted to do a performance test of the web server, I ran the plural docker containers in parallel in the computer. Then, I got the following error message (with "docker logs ""container id" command) and some of the containers terminated by themselves.

----- error message start
...
[INFO ] 2019-07-30 03:42:01.970 [[main]>worker1] http - Retrying http request, will sleep for 49 seconds
2019-07-30T03:42:02.255Z INFO beater/filebeat.go:432 Stopping filebeat
2019-07-30T03:42:02.255Z INFO crawler/crawler.go:139 Stopping Crawler
2019-07-30T03:42:02.256Z INFO crawler/crawler.go:149 Stopping 1 inputs
2019-07-30T03:42:02.256Z INFO input/input.go:149 input ticker stopped
2019-07-30T03:42:02.256Z INFO input/input.go:167 Stopping Input: 10158334123345793842
2019-07-30T03:42:02.256Z INFO log/harvester.go:275 Reader was closed: /home/download/logstash-tutorial.log. Closing.
2019-07-30T03:42:02.256Z INFO crawler/crawler.go:165 Crawler stopped
2019-07-30T03:42:02.256Z INFO registrar/registrar.go:367 Stopping Registrar
2019-07-30T03:42:02.256Z INFO registrar/registrar.go:293 Ending Registrar
rm: cannot remove '/usr/share/filebeat/bin/data/registry/filebeat': Directory not empty

----- error message end

Clould you tell me why it happened and how to fix this error to keep running the dockers ???

Here is the shellscript I made.
----- shellscript start
#!/bin/sh -eu
while true
do
# delete the cache file of filebeat if exists
if test -e /usr/share/filebeat/bin/data ; then
rm -r /usr/share/filebeat/bin/data
fi
# change directory to filebeat.yml
cd /usr/share/filebeat/bin
# start filebeat in backend (= &)
./filebeat -e -c filebeat.yml -d "publish" &
# get the process id of the last command
PID=$!
# wait for 4 seconds
sleep 8s
# stop filebeat
kill $PID
done

----- shellscript end

Here is the filebeat.yml
----- filebeat start
filebeat.inputs:

• type: log
  paths:
  • /home/download/logstash-tutorial.log
    output.logstash:
    hosts: ["localhost:5044"]
    ----- filebeat end

Here is the logstash.conf.
----- error message start
input {
beats {
port => 5044
}
}
filter {
ruby {
# How can i get the hostname of linux server using logstash
init => "require 'socket'"
code => "event.set('serverHostname', Socket.gethostname)"
}
}
output {
# stdout { codec => rubydebug }
# file { path => "/srv/logstash/filebeatLog-%{[host][name]}-%{[serverHostname]}-%{+YYYYMMddHHmm}.json" }
http {
http_method => "post"
# url or dns name to the destination
url => "your target url"
cacert => "your path to crt"
}
}
----- error message end

The log file was downloaded from: https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz

Thank you.

[rugenl (Len Rugen)]

This should fix it: rm -rf /usr/share/filebeat/bin/data

Remember "rm -rf" is dangerous...

[kensaku-okada]

Thank you for your comment. I modified the shell script as you suggested and ran the containers. Unfortunately, it did not solve the problem.

Here is the error log when the container terminated.
///////////////////////////////////////////////////////////////////////////////////
2019-08-01T08:56:56.324Z INFO instance/beat.go:280 Setup Beat: filebeat; Version: 7.0.1
2019-08-01T08:56:56.324Z INFO [publisher] pipeline/module.go:97 Beat name: LogstashSenderHttps
2019-08-01T08:56:56.324Z ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2019-08-01T08:56:56.324Z INFO instance/beat.go:391 filebeat start running.
2019-08-01T08:56:56.324Z INFO registrar/migrate.go:104 No registry home found. Create: /usr/share/filebeat/bin/data/registry/filebeat
2019-08-01T08:56:56.324Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2019-08-01T08:56:56.324Z INFO registrar/migrate.go:112 Initialize registry meta file
2019-08-01T08:56:56.331Z INFO registrar/registrar.go:108 No registry file found under: /usr/share/filebeat/bin/data/registry/filebeat/data.json. Creating a new registry file.
2019-08-01T08:56:56.340Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/bin/data/registry/filebeat/data.json
2019-08-01T08:56:56.340Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2019-08-01T08:56:56.340Z WARN beater/filebeat.go:357 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-08-01T08:56:56.340Z INFO crawler/crawler.go:72 Loading Inputs: 1
2019-08-01T08:56:56.340Z INFO log/input.go:138 Configured paths: [/home/download/logstash-tutorial.log]
2019-08-01T08:56:56.340Z INFO input/input.go:114 Starting input of type: log; ID: 10158334123345793842
2019-08-01T08:56:56.340Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-08-01T08:56:56.340Z INFO log/harvester.go:254 Harvester started for file: /home/download/logstash-tutorial.log
2019-08-01T08:56:57.341Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://localhost:5044))
2019-08-01T08:56:57.341Z INFO pipeline/output.go:105 Connection to backoff(async(tcp://localhost:5044)) established
2019-08-01T08:57:00.296Z INFO beater/filebeat.go:432 Stopping filebeat
2019-08-01T08:57:00.296Z INFO crawler/crawler.go:139 Stopping Crawler
2019-08-01T08:57:00.296Z INFO crawler/crawler.go:149 Stopping 1 inputs
2019-08-01T08:57:00.296Z INFO input/input.go:149 input ticker stopped
2019-08-01T08:57:00.296Z INFO input/input.go:167 Stopping Input: 10158334123345793842
2019-08-01T08:57:00.296Z INFO log/harvester.go:275 Reader was closed: /home/download/logstash-tutorial.log. Closing.
2019-08-01T08:57:00.296Z INFO crawler/crawler.go:165 Crawler stopped
2019-08-01T08:57:00.296Z INFO registrar/registrar.go:367 Stopping Registrar
2019-08-01T08:57:00.296Z INFO registrar/registrar.go:293 Ending Registrar
rm: cannot remove '/usr/share/filebeat/bin/data/registry/filebeat': Directory not empty
///////////////////////////////////////////////////////////////////////////////////

Here is the shell script I executed (I just changed "rm -r /usr/share/filebeat/bin/data" into "rm -rf /usr/share/filebeat/bin/data" and "sleep 4s" into "sleep 8s" ).
//////////////////////////////////////////////////////////////////////////////////
#!/bin/sh -eu
while true
do
# delete the cache file of filebeat if exists
if test -e /usr/share/filebeat/bin/data ; then
rm -rf /usr/share/filebeat/bin/data
fi
# change directory to filebeat.yml
cd /usr/share/filebeat/bin
# start filebeat in backend (= &)
./filebeat -e -c filebeat.yml -d "publish" &
# get the process id of the last command
PID=$!
# wait for 4 seconds
sleep 4s
# stop filebeat
kill $PID
done
//////////////////////////////////////////////////////////////////////////////////

I post the status of containers I ran for reference.