Writing to sincedb_ptah using Logstash and docker doesn't work

Hi everyone,

Background Information:

I have a dockerfile that creates an image for me to run my Logstash. Using this method will not write a .sincedb_* file which is my problem. In my past experiences with Logstash, I used ./bin/logstash -e which generated a .sincedb_* in .../data/plugins/inputs/file directory automatically once Logstash program terminated, but using docker it doesn't work.

Questions:

  1. How can I specify in my Dockerfile the directory that Logstash is authorized to write to?
    I found this code online. (you can see it in my Dockerfile down below)

    RUN echo 'chown -R logstash /usr/share/logstash/data/plugins/inputs/file' >> docker-entrypoint.sh
    RUN echo 'exec "$@"' >> docker-entrypoint.sh
    
  2. Is my Logstash configuration correct for sincedb_path and sincedb_write_interval?

Dockerfile:

FROM docker.elastic.co/logstash/logstash:7.0.1

RUN rm -f /usr/share/logstash/pipeline/logstash.conf
RUN rm -f /usr/share/logstash/config/pipeline.yml
RUN rm -r /usr/share/logstash/data
RUN /usr/share/logstash/bin/logstash-plugin install logstash-filter-json && \
/usr/share/logstash/bin/logstash-plugin install logstash-input-cloudwatch_logs
ADD pipeline/ /usr/share/logstash/pipeline
ADD config/ /usr/share/logstash/config
ADD data/ /usr/share/logstash/data

RUN echo 'chown -R logstash /usr/share/logstash/data/plugins/inputs/file' >> docker-entrypoint.sh
RUN echo 'exec "$@"' >> docker-entrypoint.sh

Logstash configuration:

input{
  cloudwatch_logs {
    log_group_prefix => true
    log_group => ["/aws-glue/crawlers"]
    region => "${AWS_REGION}"
    start_position => "beginning"
    sincedb_path => "data/plugins/inputs/file/.sincedb_123"
    sincedb_write_interval => "3"
  }
} 

filter {...}

output {...}

Error:

kourosh@itecoation-Lenovo-ideapad-330S-15IKB:~/Desktop/lambda_logs_to_es$ docker run cloudwatch
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-06-20T06:45:30,060][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2019-06-20T06:45:30,076][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2019-06-20T06:45:30,469][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.0.1"}
[2019-06-20T06:45:30,496][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"7e9c15e8-2331-40f4-9f06-0b488d673fe5", :path=>"/usr/share/logstash/data/uuid"}
[2019-06-20T06:45:46,110][ERROR][logstash.inputs.cloudwatch_logs] Unknown setting 'sincedb_write_interval' for cloudwatch_logs
[2019-06-20T06:45:46,128][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:pipeline1, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:86:in `config_init'", "/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:60:in `initialize'", "org/logstash/plugins/PluginFactoryExt.java:255:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:117:in `buildInput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
[2019-06-20T06:45:46,582][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-06-20T06:45:51,485][INFO ][logstash.runner          ] Logstash shut down.

I appreciate any suggestions/feedback, thank you!

I'm still looking for feedback on this one.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.