# How can I input data from java to ES in real time? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 7, 2015, 1:46pm UTC](https://discuss.elastic.co/t/how-can-i-input-data-from-java-to-es-in-real-time/21523 "2015-01-07T13:46:18Z")\ **Posts on this page:** 3\ **Page:** 2
**Author:** ![Ed\_Kim](https://avatars.discourse-cdn.com/v4/letter/e/53a042/32.png) [@Ed\_Kim](https://discuss.elastic.co/u/Ed_Kim)\ **Post date:** [January 13, 2015, 9:59pm UTC](https://discuss.elastic.co/t/how-can-i-input-data-from-java-to-es-in-real-time/21523/21 "2015-01-13T21:59:33Z")
Yea, since you mentioned real time, I thought you were concerned about data coming in 'today.' For older data, the only aspect of real time that applies is how soon you can search against ES once loaded. ES supports near real time so this isn't really an issue, and considering you have 10 year old data, I had assumed it's ok if the older data doesn't have to be readily available until the entire data set is loaded, especially since the intent for these type of things is to get an overall view of all the data points. Anyways, one thing I did in the past was to partition the historical data (in oracle) and then index the data into ES in time intervals. For example, store 1 year's worth of data in a single index. If doing it this way, you will have 10 indexes, and for query purposes, you can set aliases on the indexes and control your query using aliases. For instance, if most of your queries for analysis are for 'the last year', you can query against the index that contains data for 2014 and 2015. This way you don't query a giant index that can go back 10 years. Also, by breaking down into chunks, any indexing failures would not cause me to re-index the entire thing, which could take a very long time. Best wishes on your project! On Tuesday, January 13, 2015 at 1:07:31 PM UTC-8, Marian Valero wrote: > Thanks Ed. > > Yes, but I have data from 10 years ago and I have to input this data too > for analyze that logs with the data that is insert every day. There are > millions of logs. > > 2015-01-13 16:22 GMT-04:30 Ed Kim \<[edk...@gmail.com](mailto:edk...@gmail.com) \\>: > > > Not sure if this is an option for you, but if an application is feeding > > that log data into oracle, you could consider having that application also > > index into Elasticsearch. > > > > On Wednesday, January 7, 2015 at 5:46:18 AM UTC-8, Marian Valero wrote: > > > > > I'm reading data from my oracle database to java with JDBC but I want to > > > know how can I input the data that I'm getting in Elasticsearch in real > > > time. > > > > > > Thanks. > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/cnkq84aom1M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cnkq84aom1M/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1fd5e8fb-117e-419b-b5b7-cda96a55abec%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1fd5e8fb-117e-419b-b5b7-cda96a55abec%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marian\_Valero](https://avatars.discourse-cdn.com/v4/letter/m/b9e5f3/32.png) [@Marian\_Valero](https://discuss.elastic.co/u/Marian_Valero)\ **Post date:** [January 13, 2015, 10:08pm UTC](https://discuss.elastic.co/t/how-can-i-input-data-from-java-to-es-in-real-time/21523/22 "2015-01-13T22:08:03Z")
Thank you Ed, I'll remember that, about store index that way. 2015-01-13 17:29 GMT-04:30 Ed Kim [edkim81@gmail.com](mailto:edkim81@gmail.com): > Yea, since you mentioned real time, I thought you were concerned about > data coming in 'today.' > > For older data, the only aspect of real time that applies is how soon you > can search against ES once loaded. ES supports near real time so this isn't > really an issue, and considering you have 10 year old data, I had assumed > it's ok if the older data doesn't have to be readily available until the > entire data set is loaded, especially since the intent for these type of > things is to get an overall view of all the data points. > > Anyways, one thing I did in the past was to partition the historical data > (in oracle) and then index the data into ES in time intervals. For example, > store 1 year's worth of data in a single index. If doing it this way, you > will have 10 indexes, and for query purposes, you can set aliases on the > indexes and control your query using aliases. For instance, if most of your > queries for analysis are for 'the last year', you can query against the > index that contains data for 2014 and 2015. This way you don't query a > giant index that can go back 10 years. Also, by breaking down into chunks, > any indexing failures would not cause me to re-index the entire thing, > which could take a very long time. > > Best wishes on your project! > > On Tuesday, January 13, 2015 at 1:07:31 PM UTC-8, Marian Valero wrote: > > > Thanks Ed. > > > > Yes, but I have data from 10 years ago and I have to input this data too > > for analyze that logs with the data that is insert every day. There are > > millions of logs. > > > > 2015-01-13 16:22 GMT-04:30 Ed Kim [edk...@gmail.com](mailto:edk...@gmail.com): > > > > > Not sure if this is an option for you, but if an application is feeding > > > that log data into oracle, you could consider having that application also > > > index into Elasticsearch. > > > > > > On Wednesday, January 7, 2015 at 5:46:18 AM UTC-8, Marian Valero wrote: > > > > > > > I'm reading data from my oracle database to java with JDBC but I want > > > > to know how can I input the data that I'm getting in Elasticsearch in real > > > > time. > > > > > > > > Thanks. > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > topic/elasticsearch/cnkq84aom1M/unsubscribe. > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/cnkq84aom1M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cnkq84aom1M/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/1fd5e8fb-117e-419b-b5b7-cda96a55abec%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1fd5e8fb-117e-419b-b5b7-cda96a55abec%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/1fd5e8fb-117e-419b-b5b7-cda96a55abec%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1fd5e8fb-117e-419b-b5b7-cda96a55abec%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADT%2BR%3DZ2s4o%3De\_Px8PzPEjgcftM3%2BxtKLSJ6EQPrtfRS\_BiMaA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADT%2BR%3DZ2s4o%3De_Px8PzPEjgcftM3%2BxtKLSJ6EQPrtfRS_BiMaA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:39am UTC](https://discuss.elastic.co/t/how-can-i-input-data-from-java-to-es-in-real-time/21523/23 "2017-07-06T00:39:15Z")
Closed on July 6, 2017, 12:39am UTC [Previous page](https://discuss.elastic.co/t/how-can-i-input-data-from-java-to-es-in-real-time/21523.md?page=1)