Error while configuring multiple JDBC inputs with unique sql_last_value in docker-compose

Elastic Stack Logstash
1

I am trying to set up multiple JDBC pipelines to be able to use multiple SQL-statements. In these pipelines I want to use unique sql_last_value's. I am using docker-compose and have volumed the files correctly (I think). I am using a modified version of: https://github.com/deviantony/docker-elk .

I am getting the following error when trying to run the ELK-stack:

logstash_1       | [2019-10-25T09:38:59,145][ERROR][logstash.javapipeline    ] Pipeline aborted due to error {:pipeline_id=>"philipsmonitor", :exception=>#<Errno::EACCES: Permission denied - /usr/share/logstash/config/.logstash_jdbc_last_run1>, :backtrace=>["org/jruby/RubyFile.java:1295:in `delete'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:100:in `clean'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/value_tracking.rb:26:in `build_last_value_tracker'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:216:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:192:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:292:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:248:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:105:in `block in start'"], :thread=>"#<Thread:0x5ca8898f@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:102 run>"}
logstash_1       | [2019-10-25T09:38:59,148][ERROR][logstash.agent           ] Failed to execute action {:id=>:philipsmonitor, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<philipsmonitor>, action_result: false", :backtrace=>nil}

I have checked the permissions within the docker container and it is as follows:

bash-4.2$ ls -ll .logstash_jdbc_last_run1 .logstash_jdbc_last_run
-rw-r--r-- 1 logstash root 6 Oct 25 09:32 .logstash_jdbc_last_run
-rw-r--r-- 1 logstash root 6 Oct 25 08:35 .logstash_jdbc_last_run1

Where the first one is the default one and the second one I have created on my own.

My docker-compose.yml:

version: '2'

services:

  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
      - ./elasticsearch/data:/usr/share/elasticsearch/data/
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk

  logstash:
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./logstash/config/ignite-core-2.7.5.jar:/opt/logstash/logstash-core/lib/jars/ignite-core-2.7.5.jar:ro
      - ./logstash/config/templates/data_template.json:/usr/share/logstash/config/data_template.json:ro
      - ./logstash/config/templates/philipsmonitor_template.json:/usr/share/logstash/config/philipsmonitor_template.json:ro
      - ./logstash/config/templates/beans_template.json:/usr/share/logstash/config/beans_template.json:ro
      - ./logstash/config/jmxconf/jmxconf.json:/usr/share/logstash/config/jmxconf/jmxconf.json:ro
      - ./logstash/config/.logstash_jdbc_last_run1:/usr/share/logstash/config/.logstash_jdbc_last_run1:rw
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
      - ./logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
    ports:
      - "5000:5000"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"

    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: kibana/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch

networks:

  elk:
    driver: bridge

I honestly can't tell what is wrong and I have tried looking through the forum and haven't found a solution that works yet. The strange thing is that the default one works without throwing an error, but the second one (with the same permissions) dies. Can someone please help in this matter?

2

Alright, so now I feel kind of dumb. Apparently when you define last_run_metadata_path it automatically creates that file if it does not already exist. So all I had to do was remove the line
./logstash/config/.logstash_jdbc_last_run1:/usr/share/logstash/config/.logstash_jdbc_last_run1:rw from my docker-compose.yml, since that file is already being created. I imagine the error was that the user logstash didn't have permission to remove my copied file, so it couldn't create the new one.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.