Hi @pk.241011
How are you starting logstash. Perhaps you have a permission issues with the data dir. It should get initialized at startup. BTW that is a pretty big queue you have set up 300gb... perhaps just try the defaults first?
I changed 1 line
# Internal queuing model, "memory" for legacy in-memory based queuing and
# "persisted" for disk-based acked queueing. Defaults is memory
#
queue.type: persisted
I removed the logstash data directory to make sure I started clean
I started a local Elasticsearch and ran this conf which reads the license file.
##################################
# Read License file
##################################
input {
file {
path => "/Users/sbrown/workspace/elastic-install/7.11.2/logstash-7.11.2/LICENSE.txt"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output {
# pump to stdout for debug
stdout {codec => rubydebug}
elasticsearch {
hosts => ["localhost:9200"]
index => "test-persistent-queue"
}
}
I used this command the -r means I can reload the conf and it will re-read the data / License File
sudo ./bin/logstash -r -f ./simple-file.conf
Logstash ran as expected and loaded 223 documents / lines
The data/queue directory is initialized... if you are not seeing that that is a problem. Are there any errors in your logstash startup logs? How are you starting elasticsearch?
ceres:logstash-7.11.2 sbrown$ ls -lR data/queue/
total 0
drwxr-xr-x 6 root staff 192 Mar 29 19:35 main/
data/queue//main:
total 128
-rw-r--r-- 1 root staff 34 Mar 29 19:35 checkpoint.head
-rw-r--r-- 1 root staff 67108864 Mar 29 19:35 page.0
ceres:logstash-7.11.2 sbrown$
BTW then I stopped Elasticsearch,
then re-saved the conf file which cause the data to be re-read and logstash could not send....
[2021-03-29T19:40:17,785][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Then the I re-started Elasticsearch did not touch logstash and once logstash reconnected the data was sent I had 446 documents
[2021-03-29T19:41:47,145][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}