Build logstash image with self-defined config failed

[NashMiao]

Here is my Dockerfile:

FROM docker.elastic.co/logstash/logstash:7.2.0
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
ADD ./pipeline/ /usr/share/logstash/pipeline/
ADD ./config/ /usr/share/logstash/config/

In fact, it came from Logstash's document in here.

Here is my logstash.conf:

input {
	file {
		path => "/usr/share/logstash/pipeline/movies.csv"
        start_position => "beginning"
        sincedb_path => "/dev/null"
	}
}

filter {
    csv {
        separator => ","
        columns => ["id","content","genre"]
    }
    
    mutate {
        split => { "genre" => "|" }
        remove_field => ["path", "host","@timestamp","message"]
    }
    
    mutate {
        split => ["content", "("]
        add_field => { "title" => "%{[content][0]}"}
        add_field => { "year" => "%{[content][1]}"}
    }

    mutate {
        convert => {
            "year" => "integer"
        }
        strip => ["title"]
        remove_field => ["path", "host","@timestamp","message","content"]
    }
}

output {
	elasticsearch {
		hosts => "http://es72-01:9200"
        index => "movies"
        document_id => "%{id}"
	}
    stdout {}
}

Here is my docker-compose:

version: "3.7"
services:
  kibana:
    image: docker.elastic.co/kibana/kibana:7.2.0
    container_name: kibana72
    depends_on:
      - elastic_1
    environment:
      - XPACK_GRAPH_ENABLED=true
      - TIMELION_ENABLED=true
      - XPACK_MONITORING_COLLECTION_ENABLED="true"
      - ELASTICSEARCH_HOSTS="http://es72-01:9200"
    ports:
      - "5601:5601"
    networks:
      - es72net

  logstash:
    build: ./logstash
    container_name: logstash72
    depends_on:
      - elastic_1
    ports:
      - 9600:9600
    networks:
      - es72net
  
  elastic_1:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    container_name: es72-01
    environment:
      - cluster.name=swarm
      - node.name=es72-01
      - discovery.seed_hosts=es72-01
      - cluster.initial_master_nodes=es72-01
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es72data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - es72net

volumes:
  es72data1:
    driver: local

networks:
  es72net:
    driver: bridge

However, my Logstash failed:

$ docker attach logstash72
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-07-26T02:44:38,097][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2019-07-26T02:44:38,122][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2019-07-26T02:44:38,483][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-07-26T02:44:38,508][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"d08466ed-d60b-4146-8400-243b78936f9f", :path=>"/usr/share/logstash/data/uuid"}
[2019-07-26T02:44:40,166][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 43, column 1 (byte 852) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
[2019-07-26T02:44:40,478][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-07-26T02:44:45,373][INFO ][logstash.runner          ] Logstash shut down.

In fact, I had nothing in line 43:

By the way, my movies.csv looks like this:

I'm confused about this issue for few days...

[guyboertje]

The line 43 mentioned in the syntax error means that the parser got to the end of the file and did not find the character sequence it was expecting.

We will need to see your full config. Don't use screenshots if possible. try to paste text inside triple backticks.

[NashMiao]

Thanks, I provided all the files I written in this post.

All the files encode in UTF-8.

I don't think this issue comes from config file, because it's depends on where the .csv file be placed or when the .csv file appears.

Because of it's all built on Docker, I think you can reproduce the issue by the files I provide.

You know what? I move .csv file form pipeline to logstash folder, everything runs smoothly.

Running failed:

$ tree /F
C:.
│  docker-compose.yml
│
└─logstash
    │  Dockerfile
    │
    ├─config
    │      logstash.yml
    │
    └─pipeline
        │     logstash.conf
        └─  movies.csv

Running successful:

$ tree /F
C:.
│  docker-compose.yml
│
└─logstash
    │  Dockerfile
    │  movies.csv
    │
    ├─config
    │      logstash.yml
    │
    └─pipeline
            logstash.conf

I think it's strange, so I also open a issue in GitHub.

Besides, If I add .csv file after logstash runs successfully, it also can handle data in .csv file successfully.

[Badger]

That is expected if path.config points to a directory. logstash will concatenate all the files in the directory to form the configuration. In this case that means it concatenates logstash.conf and movies.csv, and fails to parse the first line of movies.csv.

[NashMiao]

Thanks, as document says:

Every file in the host directory ~/pipeline/ will then be parsed by Logstash as pipeline configuration.

It's my fault.