Output to file getting deleted

dirtyballer3 · September 12, 2018, 5:45pm

Hi there,

Very new to this setup as im using Docker to implement Logstash and i cannot tell if i have something wrong with my Logstash setup or if its a Docker issue so wanted to check here. I have a simple input/output setup and i am just trying to test my setup. When viewing the logs it appears the input and output is working however the file that i'm writing to on my output seems to get deleted when the Docker container exits.

example of logstash.conf:

input {
generator {
lines => [
"line 1",
"line 2",
"line 3"
]
count => 3
}
}

output {
file {
path => "/usr/share/logstash/logstash_output.txt"
codec => line { format => "%{message}"}
}
}

Debug output:

[2018-09-12T17:08:24,614][DEBUG][logstash.outputs.file ] File, writing event to file. {:filename=>"/usr/share/logstash/logstash_output.txt"}

[2018-09-12T17:08:24,631][DEBUG][logstash.outputs.file ] File, writing event to file. {:filename=>"/usr/share/logstash/logstash_output.txt"}

[2018-09-12T17:08:24,632][DEBUG][logstash.outputs.file ] File, writing event to file. {:filename=>"/usr/share/logstash/logstash_output.txt"}

[2018-09-12T17:08:24,633][DEBUG][logstash.outputs.file ] File, writing event to file. {:filename=>"/usr/share/logstash/logstash_output.txt"}

[2018-09-12T17:08:24,681][DEBUG][logstash.outputs.file ] Required path was deleted, creating the file again {:path=>"/usr/share/logstash/logstash_output.txt"}

[2018-09-12T17:08:24,687][INFO ][logstash.outputs.file ] Opening file {:path=>"/usr/share/logstash/logstash_output.txt"}

[2018-09-12T17:08:24,774][DEBUG][logstash.api.service ] [api-service] start

[2018-09-12T17:08:24,805][DEBUG][logstash.outputs.file ] Closing {:plugin=>"LogStash::Outputs::File"}

[2018-09-12T17:08:25,174][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[2018-09-12T17:08:25,676][DEBUG][logstash.outputs.file ] Close: closing files

[2018-09-12T17:08:25,692][DEBUG][logstash.outputs.file ] Closed file /usr/share/logstash/logstash_output.txt {:fd=>#<IOWriter:0x76073a71 @active=true, @io=#<File:/usr/share/logstash/logstash_output.txt (closed)>>}

[2018-09-12T17:08:25,698][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x3d2f6b88 run>"}

[2018-09-12T17:08:25,996][DEBUG][logstash.instrument.periodicpoller.os] Stopping

[2018-09-12T17:08:26,027][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping

[2018-09-12T17:08:26,033][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping

[2018-09-12T17:08:26,035][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping

[2018-09-12T17:08:26,168][DEBUG][logstash.agent ] Shutting down all pipelines {:pipelines_count=>1}

[2018-09-12T17:08:26,188][DEBUG][logstash.agent ] Converging pipelines state {:actions_count=>1}

[2018-09-12T17:08:26,201][DEBUG][logstash.agent ] Executing action {:action=>LogStash::PipelineAction::Stop/pipeline_id:main}

[2018-09-12T17:08:26,251][DEBUG][logstash.pipeline ] Stopping inputs {:pipeline_id=>"main", :thread=>"#<Thread:0x3d2f6b88 dead>"}

[2018-09-12T17:08:26,267][DEBUG][logstash.inputs.generator] Stopping {:plugin=>"LogStash::Inputs::Generator"}

[2018-09-12T17:08:26,277][DEBUG][logstash.pipeline ] Stopped inputs {:pipeline_id=>"main", :thread=>"#<Thread:0x3d2f6b88 dead>"}

[2018-09-12T17:08:26,288][DEBUG][logstash.pipeline ] Worker terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x20227c50@[main]>worker0 dead>"}

dirtyballer3 · September 12, 2018, 7:07pm

So i did a test to help narrow down to if its Logstash or Docker related and looks to be Docker. I spun up my container and manually started logstash from inside the container, i saw the file getting written to, checked and the file was created with data. Then i exited my container and when i went back into that same container the file was gone, so i must be misunderstanding something related to Docker.

Please disregard this topic as it appears to be not related to Logstash.

system · October 10, 2018, 7:07pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.