# Logstash cant start the pipeline in the conf.d file **URL:** \ **Category:** Logstash\ **Created:** [May 7, 2022, 9:29am UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172 "2022-05-07T09:29:35Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![ahmed\_barki](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahmed_barki/32/105376_2.png) [@ahmed\_barki](https://discuss.elastic.co/u/ahmed_barki)\ **Post date:** [May 7, 2022, 9:29am UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/1 "2022-05-07T09:29:35Z")
Hello everyone, when i use the command /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/simple.conf here is what i get ```auto '[INFO] 2022-05-07 09:58:02.271 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]} [INFO] 2022-05-07 09:58:02.665 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@localhost:9200/]}} [WARN] 2022-05-07 09:58:02.946 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://elastic:xxxxxx@localhost:9200/"} [INFO] 2022-05-07 09:58:02.962 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.17.3) {:es_version=>7} [WARN] 2022-05-07 09:58:02.965 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [INFO] 2022-05-07 09:58:03.024 [Ruby-0-Thread-10: :1] elasticsearch - Config is not compliant with data streams. `data_stream => auto` resolved to `false` [INFO] 2022-05-07 09:58:03.025 [[main]-pipeline-manager] elasticsearch - Config is not compliant with data streams. `data_stream => auto` resolved to `false` [WARN] 2022-05-07 09:58:03.038 [[main]-pipeline-manager] grok - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [INFO] 2022-05-07 09:58:03.083 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled} [INFO] 2022-05-07 09:58:03.293 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/simple.conf"], :thread=>"#"} [INFO] 2022-05-07 09:58:04.665 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.37} [INFO] 2022-05-07 09:58:04.684 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"} [INFO] 2022-05-07 09:58:04.730 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"} [INFO] 2022-05-07 09:58:04.831 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [INFO] 2022-05-07 09:58:04.877 [[main] 5044 } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] } } output { elasticsearch { hosts =>["localhost:9200"] user => "elastic" password => "fPAbmCodQi6q390fLLU3" } stdout {codec => rubydebug } } ``` logstash.yml ```auto xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.username: logstash_system xpack.monitoring.elasticsearch.password: Dv5cvj9uTdxMSD1jOgiv ' ``` before enabling xpack all was good. ```auto ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 7, 2022, 2:32pm UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/2 "2022-05-07T14:32:02Z")
Hi @ahmed_barki Welcome to the community. 1st what version of the stack are you using? Filebeat, Logstash Elasticsearch 2nd The pipeline is started and waiting on input from the beats... until it receives some are you sending and data through beats? 3rd what do you mean it works before you turn on monitoring 4th Are you trying to use data streams or normal indices? 5th This has bad syntax ```auto xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.username: logstash_system xpack.monitoring.elasticsearch.password: Dv5cvj9uTdxMSD1jOgiv ' **Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 7, 2022, 2:47pm UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/3 "2022-05-07T14:47:07Z") (EDITED) waiting on answers above ---
**Author:** ![ahmed\_barki](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahmed_barki/32/105376_2.png) [@ahmed\_barki](https://discuss.elastic.co/u/ahmed_barki)\ **Post date:** [May 8, 2022, 9:15am UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/4 "2022-05-08T09:15:33Z")
Hello @stephenb 1- I'm using logstash 7.17.3, Elasticsearch 7.17.3, and Filebeat 7.17.3 2- filebeat should send data to logstash(5044) .i commented out the Elasticsearch output in the filebeat.yml 3- Before enabling x pack on Elasticsearch and setuping passwords to Beats, logstash, Elasticsearch, and kibana the same config file was working with no problem ```auto input { beats { port => 5044 } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] } } output { elasticsearch { hosts =>["localhost:9200"] } stdout {codec => rubydebug } } ``` i just added Elasticsearch passwords 4- normal indices 5- i just added this extra mark by a mistake when i was typing in the discuss thanks for your answer. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 8, 2022, 2:43pm UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/5 "2022-05-08T14:43:13Z")
Thanks for the details.. > [@ahmed\_barki](#): > > 3- Before enabling x pack on Elasticsearch and setuping passwords to Beats, logstash, Elasticsearch, and kibana the same config file was working with no problem Ahh... What exactly did you setup up / enable with xpack in elasticsearch? Authentication, HTTPS etc? Did Kibana work after that as well? I am confused when you mentioned monitoring (perhaps we can leave that for part II ) After you set up xpack (security) that could you connect with `curl`... Can you share your elasticsearch.yml What did the output section of the logstash.conf look like after you added xpack? ---
**Author:** ![ahmed\_barki](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahmed_barki/32/105376_2.png) [@ahmed\_barki](https://discuss.elastic.co/u/ahmed_barki)\ **Post date:** [May 8, 2022, 3:49pm UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/6 "2022-05-08T15:49:56Z")
After adding xpack i used this command bin/Elasticsearch-setup-passwords auto. I can login to kibana using elastic user.(kibana is working i can visualize logs via metricbeat ) I added this to my conf file output ```auto user => "elastic" password => "fPAbmCodQi6q390fLLU3" ``` Thank you for your help. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 8, 2022, 3:53pm UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/7 "2022-05-08T15:53:30Z")
Apologies @ahmed_barki Is it working now? ---
**Author:** ![ahmed\_barki](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahmed_barki/32/105376_2.png) [@ahmed\_barki](https://discuss.elastic.co/u/ahmed_barki)\ **Post date:** [May 8, 2022, 5:19pm UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/8 "2022-05-08T17:19:00Z")
No, its not. The same problem When i use usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/simple.conf it stack on ```auto [INFO] 2022-05-07 09:58:04.877 [[main] **Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [May 8, 2022, 7:38pm UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/9 "2022-05-08T19:38:06Z") LS has been started and waiting for a stream. Check is your firewall active on LS. Also have a look on FB side, enable debug. Maybe a file is already parsed. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 8, 2022, 7:49pm UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/10 "2022-05-08T19:49:00Z")
Yes, what @Rios said is very important if you want to send the same file again, you have to remove the data directory in filebeat because keeps track of what you've already loaded So it won't send the same file (s) again To me. It looks like logstash is up and running and waiting for more log lines from filebeat. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 5, 2022, 7:49pm UTC](https://discuss.elastic.co/t/logstash-cant-start-the-pipeline-in-the-conf-d-file/304172/11 "2022-06-05T19:49:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.