# Build logstash image with self-defined config failed **URL:** \ **Category:** Logstash\ **Created:** [July 26, 2019, 2:55am UTC](https://discuss.elastic.co/t/build-logstash-image-with-self-defined-config-failed/192344 "2019-07-26T02:55:09Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![NashMiao](https://avatars.discourse-cdn.com/v4/letter/n/7ab992/32.png) [@NashMiao](https://discuss.elastic.co/u/NashMiao)\ **Post date:** [July 26, 2019, 2:55am UTC](https://discuss.elastic.co/t/build-logstash-image-with-self-defined-config-failed/192344/1 "2019-07-26T02:55:09Z")
Here is my Dockerfile: ``` FROM docker.elastic.co/logstash/logstash:7.2.0 RUN rm -f /usr/share/logstash/pipeline/logstash.conf ADD ./pipeline/ /usr/share/logstash/pipeline/ ADD ./config/ /usr/share/logstash/config/ ``` In fact, it came from Logstash's document in [here](https://www.elastic.co/guide/en/logstash/current/docker-config.html#_custom_images). Here is my logstash.conf: ``` input { file { path => "/usr/share/logstash/pipeline/movies.csv" start_position => "beginning" sincedb_path => "/dev/null" } } filter { csv { separator => "," columns => ["id","content","genre"] } mutate { split => { "genre" => "|" } remove_field => ["path", "host","@timestamp","message"] } mutate { split => ["content", "("] add_field => { "title" => "%{[content][0]}"} add_field => { "year" => "%{[content][1]}"} } mutate { convert => { "year" => "integer" } strip => ["title"] remove_field => ["path", "host","@timestamp","message","content"] } } output { elasticsearch { hosts => "http://es72-01:9200" index => "movies" document_id => "%{id}" } stdout {} } ``` Here is my docker-compose: ``` version: "3.7" services: kibana: image: docker.elastic.co/kibana/kibana:7.2.0 container_name: kibana72 depends_on: - elastic_1 environment: - XPACK_GRAPH_ENABLED=true - TIMELION_ENABLED=true - XPACK_MONITORING_COLLECTION_ENABLED="true" - ELASTICSEARCH_HOSTS="http://es72-01:9200" ports: - "5601:5601" networks: - es72net logstash: build: ./logstash container_name: logstash72 depends_on: - elastic_1 ports: - 9600:9600 networks: - es72net elastic_1: image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0 container_name: es72-01 environment: - cluster.name=swarm - node.name=es72-01 - discovery.seed_hosts=es72-01 - cluster.initial_master_nodes=es72-01 ulimits: memlock: soft: -1 hard: -1 volumes: - es72data1:/usr/share/elasticsearch/data ports: - 9200:9200 networks: - es72net volumes: es72data1: driver: local networks: es72net: driver: bridge ``` However, my Logstash failed: ``` $ docker attach logstash72 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Thread.exclusive is deprecated, use Thread::Mutex Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties [2019-07-26T02:44:38,097][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} [2019-07-26T02:44:38,122][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} [2019-07-26T02:44:38,483][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.2.0"} [2019-07-26T02:44:38,508][INFO][logstash.agent] No persistent UUID file found. Generating new UUID {:uuid=>"d08466ed-d60b-4146-8400-243b78936f9f", :path=>"/usr/share/logstash/data/uuid"} [2019-07-26T02:44:40,166][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 43, column 1 (byte 852) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]} [2019-07-26T02:44:40,478][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2019-07-26T02:44:45,373][INFO][logstash.runner] Logstash shut down. ``` In fact, I had nothing in line 43: ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/2/d208824ddd3c6d0a58030d11644e5bfa3bdeff81.png) By the way, my movies.csv looks like this: ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/d/2df4835b2d3c65e3a08715c2db53afd5527ea4bf.png) I'm confused about this issue for few days... ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [July 30, 2019, 4:37pm UTC](https://discuss.elastic.co/t/build-logstash-image-with-self-defined-config-failed/192344/3 "2019-07-30T16:37:37Z")
The line 43 mentioned in the syntax error means that the parser got to the end of the file and did not find the character sequence it was expecting. We will need to see your full config. Don't use screenshots if possible. try to paste text inside triple backticks. ---
**Author:** ![NashMiao](https://avatars.discourse-cdn.com/v4/letter/n/7ab992/32.png) [@NashMiao](https://discuss.elastic.co/u/NashMiao)\ **Post date:** [July 31, 2019, 2:29am UTC](https://discuss.elastic.co/t/build-logstash-image-with-self-defined-config-failed/192344/4 "2019-07-31T02:29:10Z")
Thanks, I provided all the files I written in this post. All the files encode in UTF-8. I don't think this issue comes from config file, because it's depends on where the `.csv` file be placed or when the `.csv` file appears. Because of it's all built on Docker, I think you can reproduce the issue by the files I provide. You know what? I move `.csv` file form `pipeline` to `logstash` folder, everything runs smoothly. Running failed: ```auto $ tree /F C:. │ docker-compose.yml │ └─logstash │ Dockerfile │ ├─config │ logstash.yml │ └─pipeline │ logstash.conf └─ movies.csv ``` Running successful: ```auto $ tree /F C:. │ docker-compose.yml │ └─logstash │ Dockerfile │ movies.csv │ ├─config │ logstash.yml │ └─pipeline logstash.conf ``` I think it's strange, so I also open a issue in [GitHub](https://github.com/elastic/logstash/issues/11003). Besides, If I add `.csv` file after logstash runs successfully, it also can handle data in `.csv` file successfully. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 31, 2019, 1:21pm UTC](https://discuss.elastic.co/t/build-logstash-image-with-self-defined-config-failed/192344/5 "2019-07-31T13:21:20Z")
That is expected if path.config points to a directory. logstash will concatenate all the files in the directory to form the configuration. In this case that means it concatenates logstash.conf and movies.csv, and fails to parse the first line of movies.csv. ---
**Author:** ![NashMiao](https://avatars.discourse-cdn.com/v4/letter/n/7ab992/32.png) [@NashMiao](https://discuss.elastic.co/u/NashMiao)\ **Post date:** [August 1, 2019, 2:46am UTC](https://discuss.elastic.co/t/build-logstash-image-with-self-defined-config-failed/192344/6 "2019-08-01T02:46:37Z")
Thanks, as document says: > Every file in the host directory `~/pipeline/` will then be parsed by Logstash as pipeline configuration. It's my fault. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 29, 2019, 2:49am UTC](https://discuss.elastic.co/t/build-logstash-image-with-self-defined-config-failed/192344/7 "2019-08-29T02:49:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.