# Jdbc input plugin and rebuilding elasticsearch index?

**URL:** <https://discuss.elastic.co/t/jdbc-input-plugin-and-rebuilding-elasticsearch-index/38674>\
**Category:** Logstash\
**Created:** [January 7, 2016, 9:44pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-and-rebuilding-elasticsearch-index/38674 "2016-01-07T21:44:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joshua\_P](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@Joshua\_P](https://discuss.elastic.co/u/Joshua_P)\
**Post date:** [January 7, 2016, 9:44pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-and-rebuilding-elasticsearch-index/38674/1 "2016-01-07T21:44:52Z")

</div>

Is there a way to use the clean\_run parameter in the jdbc input plugin to rebuild an index once a day while still doing periodic updates throughout the day to the same index? Using clean\_run doesn't seem to work the way I expected it to in that it seems to reset the sql\_last\_start variable for the entire file, instead of per jdbc input. This is my logstash.conf file:

The idea was to have the rebuild run during the 6-7 hour and updates going on any other hours. But logstash is pulling the entire data set instead of a diff based on modified date. I'm using :sql\_last\_run in my query.

```
input {

  # Rebuild Property Index Data
  jdbc {
    jdbc_driver_library => "sqljdbc42.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://127.0.0.1;databaseName=db"
    jdbc_user => "user"
		jdbc_password => "pass"
    statement_filepath => "propertyQuery.sql"
    type => "rcaproperty"
    clean_run => true
    # run rebuild at 6:00am every day.
    schedule => "0 09 * * *"
  }
	# Update Property Index Data
	jdbc {
		jdbc_driver_library => "/sqljdbc42.jar"
		jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
		jdbc_connection_string => "jdbc:sqlserver://127.0.0.1;databaseName=db"
    jdbc_user => "user"
		jdbc_password => "pass"
		statement_filepath => "propertyQuery.sql"
		type => "rcaproperty"
		# update every ten minutes.
		schedule => "*/10 0-8,10-23 * * *"
	}
}

filter {
	# Property Index handling
	if [type] == "rcaproperty" {
		mutate {
			split => { "altaddresses" => "|"}
		}

		mutate {
			strip => ["altaddresses"]
		}

		if [lat_dbl] {
			if [lon_dbl] {
				mutate {
					add_field => { "geocode" => "%{lat_dbl},%{lon_dbl}"}
				}
			}
		}
	}
}

output {
	# Property Index Handling
	if [type] == "rcaproperty" {
		elasticsearch {
			hosts => ["localhost:9200"]
			document_id => "%{property_id}"
			index => "rcasearchindex_dev_%{+YYYY.MM.dd}"
			template_name => "rcasearchindex_template"
		}
	}
}
```

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [January 7, 2016, 10:13pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-and-rebuilding-elasticsearch-index/38674/2 "2016-01-07T22:13:53Z")

</div>

If you have multiple jdbc input blocks, you should have a separate metadata file for each:

```auto
input {
  jdbc {
    # ...my config options...
    # Path to file with last run time
    last_run_metadata_path => "/path/to/metadata/file1"
    # ...more config options...
  }
  jdbc {
    # ...my other config options...
    # Path to file with last run time
    last_run_metadata_path => "/path/to/metadata/file2"
    # ...more config options...
  }
}

```

The default storage place is: `#{ENV['HOME']}/.logstash_jdbc_last_run`, so they're colliding.

---

<div class="post-metadata">

**Author:** ![Joshua\_P](https://avatars.discourse-cdn.com/v4/letter/j/c89c15/32.png) [@Joshua\_P](https://discuss.elastic.co/u/Joshua_P)\
**Post date:** [January 7, 2016, 10:52pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-and-rebuilding-elasticsearch-index/38674/3 "2016-01-07T22:52:57Z")

</div>

Thank you! That's exactly what I needed.

I missed the last\_run\_meta\_data option. Solved my problem.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:16am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-and-rebuilding-elasticsearch-index/38674/4 "2017-07-06T05:16:25Z")

</div>