# Increasing load on indices without any change in the database records

**URL:** <https://discuss.elastic.co/t/increasing-load-on-indices-without-any-change-in-the-database-records/356960>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [April 8, 2024, 10:59am UTC](https://discuss.elastic.co/t/increasing-load-on-indices-without-any-change-in-the-database-records/356960 "2024-04-08T10:59:52Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhinavtyagi](https://avatars.discourse-cdn.com/v4/letter/a/e79b87/32.png) [@abhinavtyagi](https://discuss.elastic.co/u/abhinavtyagi)\
**Post date:** [April 8, 2024, 10:59am UTC](https://discuss.elastic.co/t/increasing-load-on-indices-without-any-change-in-the-database-records/356960/1 "2024-04-08T10:59:53Z")

</div>

Hello, I am a newbie and I tried syncing Elasticsearch with MySQL database (two tables from my database). Even they are synced I am facing some issues as described below:

Code:

base.conf:

```auto
input {
  jdbc {
    jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-8.0.22.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://mysql_8:3306/pcs_accounts_db"
    jdbc_user => "pcs_db_user"
    jdbc_password => "laravel_db"
    sql_log_level => "debug"  
    clean_run => true 
    record_last_run => false
    type => "txn"
    statement => "SELECT * FROM ac_transaction_dump"
  }

  jdbc {
    jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-8.0.22.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://mysql_8:3306/pcs_accounts_db"
    jdbc_user => "pcs_db_user"
    jdbc_password => "laravel_db"
    sql_log_level => "debug"  
    clean_run => true 
    record_last_run => false
    type => "trial"
    statement => "SELECT * FROM ac_daily_trial_balance"
  }
}

filter {  
  mutate {
    remove_field => ["@version", "@timestamp"]
  }
}

output {

  stdout { codec => rubydebug { metadata => true } }

  if [type] == "txn" {
    elasticsearch {
      hosts => ["http://elasticsearch:9200"]
      data_stream => "false"
      index => "ac_transaction_dump"
      document_id => "%{transaction_dump_id}"
    }
  }

  if [type] == "trial" {
    elasticsearch {
      hosts => ["http://elasticsearch:9200"]
      data_stream => "false"
      index => "ac_daily_trial_balance"
      document_id => "%{daily_trial_balance_id}"
    }
  }
}

```

change.conf:

```auto
input {
  jdbc {
    jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-8.0.22.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://mysql_8:3306/pcs_accounts_db"
    jdbc_user => "pcs_db_user"
    jdbc_password => "laravel_db"
    type => "txn"
    use_column_value => true
    tracking_column => 'transaction_dump_id'
    last_run_metadata_path => "/usr/share/logstash/.logstash_jdbc_last_run_a'"
    sql_log_level => "debug"  
    schedule => "*/5 * * * * *"  
    statement => "
                  SELECT * 
                  FROM ac_transaction_dump 
                  WHERE (created_at > :sql_last_value)
                  OR (updated_at > :sql_last_value);
                "
  }

  jdbc {
    jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-8.0.22.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://mysql_8:3306/pcs_accounts_db"
    jdbc_user => "pcs_db_user"
    jdbc_password => "laravel_db"
    type => "trial"
    use_column_value => true
    tracking_column => 'daily_trial_balance_id'
    last_run_metadata_path => "/usr/share/logstash/.logstash_jdbc_last_run_b"
    sql_log_level => "debug"  
    schedule => "*/5 * * * * *"  
    statement => "
                  SELECT * 
                  FROM ac_daily_trial_balance 
                  WHERE (created_at > :sql_last_value)
                  OR (updated_at > :sql_last_value);
                "
  }
}

filter {
    if [deleted_at] {
    mutate { 
        add_field => { "[@metadata][action]" => "delete" }
        }
    }
    mutate {
    remove_field => ["@version", "@timestamp"]
    }
}

# stdout { codec => rubydebug { metadata => true } }
output {
    if [type] == "txn" {
        if [@metadata][action] == "delete" {
            elasticsearch {
            hosts => ["http://elasticsearch:9200"]
            index => "ac_transaction_dump"
            action => "delete"
            document_id => "%{transaction_dump_id}"
            }
        }
        else {
            elasticsearch {
            hosts => ["http://elasticsearch:9200"]
            index => "ac_transaction_dump"
            document_id => "%{transaction_dump_id}"
            }
        }
    }
    
    if [type] == "trial" {
        if [@metadata][action] == "delete" {
            elasticsearch {
            hosts => ["http://elasticsearch:9200"]
            index => "ac_daily_trial_balance"
            action => "delete"
            document_id => "%{daily_trial_balance_id}"
            }
        }
        else {
            elasticsearch {
            hosts => ["http://elasticsearch:9200"]
            index => "ac_daily_trial_balance"
            document_id => "%{daily_trial_balance_id}"
            }
        }
    }
}

```

pipelines.yml:

```auto
- pipeline.id: base-pipeline
  path.config: "/usr/share/logstash/pipeline/base.conf"

- pipeline.id: incremental-pipeline
  path.config: "/usr/share/logstash/pipeline/change.conf"

```

Issues:

After starting elasticsearch and kibana, under index management tab in kibana:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/2/92def00668a12b427f97b8ca231eae8285766a88.png)

After starting the logstash with "docker-compose up -d logstash" :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/5/b588bd10eaf999bdeab3c1e30d241ca8bd58531a.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/9/69788fd50517867d88780afbea224527d0a9cccc.png)

After executing the above queries, indices data changes again. Note that I neither added any records in the db nor updated or deleted anything.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/5/25921a1b7a02141509df9e3a3ab2a4aae711c9b6.png)

Again I executed the queries for both indices, and the storage and docs\_count increases again without any change in the database.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/2/3202ab346b65813568ba6f4151681cde36cc6074.png)

Also note that even there is no change in the database, executing queries each time is taking so much time:

![image](https://us1.discourse-cdn.com/elastic/original/3X/c/d/cd662830849410e780bd668edded117cf8b0ae05.png)

Please look at these issues and guide me where am I going wrong or if any change needed in the code.

Note: tracking\_column which is the primary key for each table is of type UUID.

Please reply if anymore info needed.