# Problem ingest data filebeat with multi-pipeline **URL:** \ **Category:** Logstash\ **Created:** [December 28, 2018, 10:50am UTC](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337 "2018-12-28T10:50:15Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![hermann102](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hermann102/32/41615_2.png) [@hermann102](https://discuss.elastic.co/u/hermann102)\ **Post date:** [December 28, 2018, 10:50am UTC](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337/1 "2018-12-28T10:50:15Z")
Hello I have successfully configured a multi-pipeline. Each pipeline normally loads the data. My multi-pipeline consists of three jdbc pipeline and one filebeat entry. This morning I was facing a problem loading data via filebeat to logstash. Filebeat copied the files to his directory and executed the transfer to logstash. But I do not know why, logstash did not ingest this data. I'm have this questions 1. Is it possible for one pipeline to impact data load another during its execution? 2. How can I be reassured that logstash will process all connector inputs? Thank you for your suggestions ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 29, 2018, 9:04am UTC](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337/2 "2018-12-29T09:04:01Z")
What does your configuration look like? ---
**Author:** ![hermann102](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hermann102/32/41615_2.png) [@hermann102](https://discuss.elastic.co/u/hermann102)\ **Post date:** [January 3, 2019, 8:07am UTC](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337/3 "2019-01-03T08:07:20Z")
Hello Christian Sorry for the late response, I was missing these days. My pipeline.yml file is as follows: ``` pipeline.id: pipeline_1 path.config: "/etc/logstash/conf.d/pipeline_1.conf" pipeline.id: pipeline_2 path.config: "/etc/logstash/conf.d/pipeline_2.conf" pipeline.id: pipeline_3 path.config: "/etc/logstash/conf.d/pipeline_3.conf" ``` I share with you, **pipeline\_1** which have problem. ``` input { beats { port => 5044 type => "logs" } jdbc { jdbc_driver_library => "/data/driverSQL/sqljdbc42.jar" jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc_connection_string => "jdbc:sqlserver://16.10.25.36:236;instanceName=jhhdhjjdj;databasename=database" jdbc_user => "User_here" jdbc_password => "Password_here" #every first sunday of the month, at 22:00 schedule => "0 18 * * sun#1" statement => "SELECT * FROM [database].[dbo].[table] where date > :sql_last_value" use_column_value => true tracking_column => "date" tracking_column_type => "timestamp" clean_run => false last_run_metadata_path => "/etc/logstash/.logstash_jdbc_last_run" type => "database" } } filter { } output { if [type] == "logs" { elasticsearch { hosts => ["localhost:9200"] user => "user_login" password => "password" index => "indexfile" document_type => "My_document_type" document_id => "%{[fields_id]}" } }else { elasticsearch { hosts => ["localhost:9200"] user => "user_login" password => "password" index => "indexdatabase" document_id => "%{[fields_id]}" } } } ``` This pipeline includes the input beat and one jdbc input. I have the impression that this is my problem. I plan to separate these two entries into different pipelines, because when I comment the jdbc entry, the beat entry works fine. I have a question about the **pipeline.yml** file Is there a problem using only the main pipeline to start all the connectors and beat input? ``` pipeline.id: main path.config: ''/etc/logstash/conf.d/*.conf'' ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 3, 2019, 8:11am UTC](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337/4 "2019-01-03T08:11:07Z")
If you are only going to use a main pipeline, there is no need to use pipelines.yml as this is the default behaviour. I would however recommend splitting it up into 4 pipelines as they seem to have different combinations of inputs and outputs. ---
**Author:** ![hermann102](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hermann102/32/41615_2.png) [@hermann102](https://discuss.elastic.co/u/hermann102)\ **Post date:** [January 3, 2019, 8:14am UTC](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337/5 "2019-01-03T08:14:25Z")
Ok, thank you. I come back to you, to confirm is ok. ---
**Author:** ![hermann102](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hermann102/32/41615_2.png) [@hermann102](https://discuss.elastic.co/u/hermann102)\ **Post date:** [January 7, 2019, 5:32am UTC](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337/6 "2019-01-07T05:32:14Z")
Hello Christian All work properly with the 4 separate inputs. Thank you ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 4, 2019, 5:32am UTC](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337/7 "2019-02-04T05:32:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.