# Logstash pipeline problem **URL:** \ **Category:** Logstash\ **Created:** [October 23, 2019, 3:21pm UTC](https://discuss.elastic.co/t/logstash-pipeline-problem/204902 "2019-10-23T15:21:44Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![basavyr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/basavyr/32/55663_2.png) [@basavyr](https://discuss.elastic.co/u/basavyr)\ **Post date:** [October 23, 2019, 3:21pm UTC](https://discuss.elastic.co/t/logstash-pipeline-problem/204902/1 "2019-10-23T15:21:44Z")
So I have to achieve a Logstash setup where I have two (or maybe even more) log files, each with its own input and filters (output will always be sent to Elastic and then visualized in Kibana). I saw this is very doable with the `pipelines.yml` configuration. I've done this for two logs (two conf files in the default config folders) and the pipeline setup looks like this: ```auto - pipeline.id: filter1 path.config: "/etc/logstash/conf.d/log_conf.conf" pipeline.workers: 3 - pipeline.id: filter2 path.config: "/etc/logstash/conf.d/log_conf2.conf" queue.type: persisted ``` I'm running my development project in a centOS 7 machine at work. Now, I tested this with pipeline by going to `/usr/share/logstash/` and running `./bin/logstash` to see if it works. The `pipeline.yml` file is in the correct folder and also the config files as well. Everything works OK, the data is sent from Logstash to Elastic and then I can see it in Kibana. But here comes the problem: I want to have my ELK setup running all the time, so I want to do `systemctl start logstash.service` and then let the logstash pipeline work in the background, without having to start it from the binaries. However, when the process starts, no more data is going to Kibana, basically it's just like the logs are not even sent to Elastic/Kibana. I tried to see the logs for starting the logstash service with `systemctl status logstash.service` and here is what I got: ```auto Oct 23 18:14:52 elk.nipne.ro logstash[31677]: Pipeline_id:filter2 Oct 23 18:14:52 elk.nipne.ro logstash[31677]: Plugin: "beginning", path=>["/home/robert.poenaru/elk/arc_slurm_jobs.txt"], id=>"0c16b9ff1fe1aaca45b6072f213460ef2e62993c22d521cccd05ebe4e4d66e1b", sincedb_path=>"NULL", enable_metric=>true, codec=>"plain_7cca3f5f-6719-4396-b596-bf12ac253b57", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc"> Oct 23 18:14:52 elk.nipne.ro logstash[31677]: Error: Permission denied - NULL Oct 23 18:14:52 elk.nipne.ro logstash[31677]: Exception: Errno::EACCES Oct 23 18:14:52 elk.nipne.ro logstash[31677]: Stack: org/jruby/RubyIO.java:1237:in `sysopen' Oct 23 18:14:52 elk.nipne.ro logstash[31677]: org/jruby/RubyFile.java:367:in `initialize' Oct 23 18:14:52 elk.nipne.ro logstash[31677]: org/jruby/RubyIO.java:1156:in `open' Oct 23 18:14:52 elk.nipne.ro logstash[31677]: uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1136:in `block in touch' Oct 23 18:14:52 elk.nipne.ro logstash[31677]: org/jruby/RubyArray.java:1800:in `each' Oct 23 18:14:52 elk.nipne.ro logstash[31677]: uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1130:in `touch' ``` Why is there a permission problem? I run the ELK stack with sudo rights. Also, the structure of logstash in the etc/ folder is: ```auto [root@elk logstash]# pwd/etc/logstash [root@elk logstash]# tree -h. ├── [49] conf.d (two simple configs for reading logs from two files) │ ├── [503] log_conf2.conf │ └── [502] log_conf.conf ├── [2.0K] jvm.options ├── [4.9K] log4j2.properties ├── [342] logstash-sample.conf ├── [8.1K] logstash.yml ├── [578] pipelines.yml (content of the file the one given above) └── [1.7K] startup.options ``` Any ideas what's the issue? How can I make the pipeline work by using the logstash.service process instead of running it from the `bin/` directory? Thank you in advance 🙂 ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 23, 2019, 4:30pm UTC](https://discuss.elastic.co/t/logstash-pipeline-problem/204902/2 "2019-10-23T16:30:09Z")
> [@basavyr](#): > > sincedb\_path=\>"NULL" On UNIX that will create a file called NULL in whichever directory logstash is running in, and when running as a service that may well not be writeable. If you do not want to persist the sincedb across restarts then use NUL on Windows and /dev/null on UNIX. If that's not the problem please supply more of the stack trace, at least to the point that we can see where in the filter it is getting an error. ---
**Author:** ![basavyr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/basavyr/32/55663_2.png) [@basavyr](https://discuss.elastic.co/u/basavyr)\ **Post date:** [October 24, 2019, 5:11am UTC](https://discuss.elastic.co/t/logstash-pipeline-problem/204902/3 "2019-10-24T05:11:35Z")
Ok, As a first step, since I'm on UNIX, I've changed the path for both config files in Logstash, and now the service starts with NO errors: ```auto [2019-10-24T07:58:24,683][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.4.1"} [2019-10-24T07:58:28,869][INFO][org.reflections.Reflections] Reflections took 68 ms to scan 1 urls, producing 20 keys and 40 values [2019-10-24T07:58:33,896][INFO][logstash.outputs.elasticsearch][filter2] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} [2019-10-24T07:58:33,894][INFO][logstash.outputs.elasticsearch][filter1] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} [2019-10-24T07:58:34,283][WARN][logstash.outputs.elasticsearch][filter2] Restored connection to ES instance {:url=>"http://localhost:9200/"} [2019-10-24T07:58:34,287][WARN][logstash.outputs.elasticsearch][filter1] Restored connection to ES instance {:url=>"http://localhost:9200/"} [2019-10-24T07:58:34,375][INFO][logstash.outputs.elasticsearch][filter1] ES Output version determined {:es_version=>7} [2019-10-24T07:58:34,385][INFO][logstash.outputs.elasticsearch][filter2] ES Output version determined {:es_version=>7} [2019-10-24T07:58:34,386][WARN][logstash.outputs.elasticsearch][filter1] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [2019-10-24T07:58:34,386][WARN][logstash.outputs.elasticsearch][filter2] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [2019-10-24T07:58:34,450][INFO][logstash.outputs.elasticsearch][filter1] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]} [2019-10-24T07:58:34,454][INFO][logstash.outputs.elasticsearch][filter2] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]} [2019-10-24T07:58:34,855][INFO][logstash.outputs.elasticsearch][filter2] Creating rollover alias [2019-10-24T07:58:34,861][INFO][logstash.outputs.elasticsearch][filter1] Creating rollover alias [2019-10-24T07:58:35,089][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][filter1] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [2019-10-24T07:58:35,091][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][filter2] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [2019-10-24T07:58:35,094][INFO][logstash.javapipeline][filter1] Starting pipeline {:pipeline_id=>"filter1", "pipeline.workers"=>3, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>375, :thread=>"#"} [2019-10-24T07:58:35,097][INFO][logstash.javapipeline][filter2] Starting pipeline {:pipeline_id=>"filter2", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#"} [2019-10-24T07:58:35,179][INFO][logstash.outputs.elasticsearch][filter1] Rollover Alias already exists. Skipping [2019-10-24T07:58:36,193][INFO][logstash.javapipeline][filter1] Pipeline started {"pipeline.id"=>"filter1"} [2019-10-24T07:58:36,193][INFO][logstash.javapipeline][filter2] Pipeline started {"pipeline.id"=>"filter2"} [2019-10-24T07:58:36,361][INFO][filewatch.observingtail][filter2] START, creating Discoverer, Watch with file and sincedb collections [2019-10-24T07:58:36,365][INFO][filewatch.observingtail][filter1] START, creating Discoverer, Watch with file and sincedb collections [2019-10-24T07:58:36,482][INFO][logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:filter2, :filter1], :non_running_pipelines=>[]} [2019-10-24T07:58:37,059][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} ``` However, the data is still not reaching Kibana. There is an index pattern if I go to Elasticsearch management, but as you can see, document count is always zero, no matter how much I modify content from those two log files. ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af740660ef34dfdf40f50a1f20ed79ac7c85b1b3.png) Remember that log files are properly seen if I open logstash from the binaries folder. So what could be the issue now? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 24, 2019, 1:10pm UTC](https://discuss.elastic.co/t/logstash-pipeline-problem/204902/4 "2019-10-24T13:10:24Z")
> [@basavyr](#): > > So what could be the issue now? Impossible to say without seeing the pipeline configurations. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 21, 2019, 1:10pm UTC](https://discuss.elastic.co/t/logstash-pipeline-problem/204902/5 "2019-11-21T13:10:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.