Runninning two configs in parallel without conflict with schedule

[geothomas]

I am trying to create elasticsearch index from oracle table. I have two configs, one delta.conf

*input {*

* jdbc*
*{*

*    jdbc_driver_library => "<path>/ojdbc10.jar"*

*    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"*

*    jdbc_connection_string => "connection string"*

*    jdbc_user => <username>*

*    jdbc_password => <pwd>*

*    statement => "SELECT * FROM my table name WHERE last_modified_date >= :sql_last_value"*

*    schedule => "*/15 * * * *"*
*"*

*}*

*filter {*

*  if !("$FULL_IMPORT_RUNNING" in [tags])*
*    {*

*   *
*    }*
*    else {*

*    drop {}*

*    }*

*  }*


*}*
*output *
*{*

*     if !("$FULL_IMPORT_RUNNING" in [tags]) *
*     {*

*        elasticsearch *
*        {*
*            hosts => ["http://localhost:9200"]*
*                index => "index_with_id"*
*                document_id => "%{col1}-%{col2}-%{col3}"*
*                action => "update"*
*            doc_as_upsert => true*
*            document_id => "%{col1}-%{col2}-%{col3}"*
*        }*
*     }*

*}*

and full.conf

*input {*

* jdbc*
*{*

*    jdbc_driver_library => "<path>/ojdbc10.jar"*

*    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"*

*    jdbc_connection_string => "connection string"*

*    jdbc_user => <username>*

*    jdbc_password => <pwd>*

*    statement => "SELECT * FROM my table name"*

*    schedule => "0 0 * * *"*
*"*

*}*

*filter {*

*  mutate {*

*    add_tag => ["$FULL_IMPORT_RUNNING"]*

*  }*

*}*



*}*
*output *
*{*


*elasticsearch *
*{*
*    hosts => ["http://localhost:9200"]*
*        index => "index_with_id"*
*        document_id => "%{col1}-%{col2}-%{col3}"*
*        action => "update"*
*    doc_as_upsert => true*
*    document_id => "%{col1}-%{col2}-%{col3}"*
*}*
*    *

*}*

If I start logstash with either one of this configs, everything works fine. But when I try to start logstash like : ./logstash -f ../config/delta.conf -f ../config/full.conf I seet thats its started, but nothing happens after tht, no error as well. I am trying to run these flows with a schedule, one for fulltruth and one for delta and trying to avoid execution at the same time

[Priscilla_Parodi]

Hello @geothomas,

Welcome to the community!

If you need to run more than one pipeline in the same process, Logstash provides a way to do this through a configuration file called pipelines.yml.

Hope it helps!

[geothomas]

Thanks @Priscilla_Parodi , that triggered the execution in parallel. But what I need is to make sure my delta pipeline to stop execution when my full pipeline is runnining. So delta should run only if full is not executing

[Badger]

logstash does not provide a way to do that. The pipelines run independently.