Yea, since you mentioned real time, I thought you were concerned about data
coming in 'today.'
For older data, the only aspect of real time that applies is how soon you
can search against ES once loaded. ES supports near real time so this isn't
really an issue, and considering you have 10 year old data, I had assumed
it's ok if the older data doesn't have to be readily available until the
entire data set is loaded, especially since the intent for these type of
things is to get an overall view of all the data points.
Anyways, one thing I did in the past was to partition the historical data
(in oracle) and then index the data into ES in time intervals. For example,
store 1 year's worth of data in a single index. If doing it this way, you
will have 10 indexes, and for query purposes, you can set aliases on the
indexes and control your query using aliases. For instance, if most of your
queries for analysis are for 'the last year', you can query against the
index that contains data for 2014 and 2015. This way you don't query a
giant index that can go back 10 years. Also, by breaking down into chunks,
any indexing failures would not cause me to re-index the entire thing,
which could take a very long time.
Best wishes on your project!
On Tuesday, January 13, 2015 at 1:07:31 PM UTC-8, Marian Valero wrote:
Thanks Ed.
Yes, but I have data from 10 years ago and I have to input this data too
for analyze that logs with the data that is insert every day. There are
millions of logs.2015-01-13 16:22 GMT-04:30 Ed Kim <edk...@gmail.com <javascript:>>:
Not sure if this is an option for you, but if an application is feeding
that log data into oracle, you could consider having that application also
index into Elasticsearch.On Wednesday, January 7, 2015 at 5:46:18 AM UTC-8, Marian Valero wrote:
I'm reading data from my oracle database to java with JDBC but I want to
know how can I input the data that I'm getting in Elasticsearch in real
time.Thanks.
--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/cnkq84aom1M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com?utm_medium=email&utm_source=footer
.For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1fd5e8fb-117e-419b-b5b7-cda96a55abec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.