# Need help to evaluate Logstash for syncing Oracle data into Elasticsearch (combining multiple 25+ table records into single nested document) **URL:** \ **Category:** Logstash\ **Created:** [December 20, 2021, 6:46am UTC](https://discuss.elastic.co/t/need-help-to-evaluate-logstash-for-syncing-oracle-data-into-elasticsearch-combining-multiple-25-table-records-into-single-nested-document/292430 "2021-12-20T06:46:13Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Tejas\_Damle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tejas_damle/32/80037_2.png) [@Tejas\_Damle](https://discuss.elastic.co/u/Tejas_Damle)\ **Post date:** [December 20, 2021, 6:46am UTC](https://discuss.elastic.co/t/need-help-to-evaluate-logstash-for-syncing-oracle-data-into-elasticsearch-combining-multiple-25-table-records-into-single-nested-document/292430/1 "2021-12-20T06:46:13Z")
**Requirement:** We are planning to use Logstash to sync data from Oracle to Elasticsearch. There are around 25+ tables in Oracle. Plan is to to have single index in Elasticsearch with nested/flattened documents which will contain data from all associated tables. We are evaluating if Logstash can be used - to get records from Oracle (by joining all tables), transform into nested structure and then load into Elasticsearch. Oracle tables can have one to many relationship, so corresponding data needs to be structured as list of objects in JSON document. And this structure can become very complex. **Questions:** 1. We would like to know if Logstash will be suitable for our requirement and will be performant enough to load large volume of data. 2. How can we scale Logstash to improve transformation and indexing performance? 3. If tracking column is not present in associated child tables, then how will it sync data? Will it be required to write trigger to update tracking column (may be updated date) in parent table (so that Logstash will start sync process)? ---
**Author:** ![Alex\_Marquardt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_marquardt/32/42925_2.png) [@Alex\_Marquardt](https://discuss.elastic.co/u/Alex_Marquardt)\ **Post date:** [December 20, 2021, 9:38am UTC](https://discuss.elastic.co/t/need-help-to-evaluate-logstash-for-syncing-oracle-data-into-elasticsearch-combining-multiple-25-table-records-into-single-nested-document/292430/2 "2021-12-20T09:38:25Z")
You could make use of Elastic's [update](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html), [upsert](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#upserts), or [scripted upsert](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#scripted_upsert) functionality to combine data from different sources into a single destination. Examples of scripted upserts executed by Logstash can be found in the following blogs: - [Using Logstash and Elasticsearch to calculate transaction duration in a microservices architecture](https://alexmarquardt.com/2020/09/16/using-logstash-and-elasticsearch-scripted-upserts-to-calculate-transaction-duration-from-out-of-order-events/) - [Using Logstash and Elasticsearch scripted upserts to transform eCommerce purchasing data](https://alexmarquardt.com/2019/12/17/logstash-and-elasticsearch-painless-scripted-upserts-transform-data/) In conjunction with scripted upserts it may be necessary to iterate over all fields in the parameters that are passed in. You may find the follow article useful for guidance on how to do this in painless: - [Using Elasticsearch Painless scripting to recursively iterate through JSON fields](https://alexmarquardt.com/2020/11/06/using-elasticsearch-painless-scripting-to-iterate-through-fields/). You may also find the following article to be relevant: - [How to keep Elasticsearch synced with a RDBMS using Logstash and JDBC | Elastic Blog](https://www.elastic.co/blog/how-to-keep-elasticsearch-synchronized-with-a-relational-database-using-logstash) You may also wish to review the following comment about search-time operations: - [Tune for search speed | Elasticsearch Guide [master] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-search-speed.html#_document_modeling) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 17, 2022, 9:39am UTC](https://discuss.elastic.co/t/need-help-to-evaluate-logstash-for-syncing-oracle-data-into-elasticsearch-combining-multiple-25-table-records-into-single-nested-document/292430/3 "2022-01-17T09:39:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.