I have hadoop plugin with hdfs gateway but what I am seeing is that indexes
are still being written locally. Can you please help me understand why it's
being written locally?
ls -ltr data/elasticsearch/nodes/0/indices/twitter/0/index/
total 12
-rw-r--r-- 1 root root 0 May 14 15:33 write.lock
-rw-r--r-- 1 root root 20 May 14 15:33 segments.gen
-rw-r--r-- 1 root root 58 May 14 15:33 segments_1
-rw-r--r-- 1 root root 8 May 14 15:33 _checksums-1337034789114
I have hadoop plugin with hdfs gateway but what I am seeing is that
indexes are still being written locally. Can you please help me understand
why it's being written locally?
ls -ltr data/elasticsearch/nodes/0/indices/twitter/0/index/
total 12
-rw-r--r-- 1 root root 0 May 14 15:33 write.lock
-rw-r--r-- 1 root root 20 May 14 15:33 segments.gen
-rw-r--r-- 1 root root 58 May 14 15:33 segments_1
-rw-r--r-- 1 root root 8 May 14 15:33 _checksums-1337034789114
There are two ways to configure ES for persistence (for the data to survive
full cluster restart)
Local gateway, where the data persists on the servers
Shared or central gateway (S3, Hadoop, or shared file system) where data
is stored elsewhere.
In either case, data is still stored locally. With the shared gateway, data
is restored from that data store when a node restarts. For more
information, highly recommend reading the docs thoroughly.
Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype
I have hadoop plugin with hdfs gateway but what I am seeing is that
indexes are still being written locally. Can you please help me understand
why it's being written locally?
ls -ltr data/elasticsearch/nodes/0/indices/twitter/0/index/
total 12
-rw-r--r-- 1 root root 0 May 14 15:33 write.lock
-rw-r--r-- 1 root root 20 May 14 15:33 segments.gen
-rw-r--r-- 1 root root 58 May 14 15:33 segments_1
-rw-r--r-- 1 root root 8 May 14 15:33 _checksums-1337034789114
Yes I read that and also have done recovery testing too, which seems to
recover everything. My question was when does elasticsearch writes/commits
data to Hadoop? Is it synchronously or async? Should I expect to lose any
data that might be in elasticsearch memory? Just trying to understand the
basics.
On Mon, May 14, 2012 at 4:23 PM, Berkay Mollamustafaoglu mberkay@gmail.comwrote:
There are two ways to configure ES for persistence (for the data to
survive full cluster restart)
Local gateway, where the data persists on the servers
Shared or central gateway (S3, Hadoop, or shared file system) where
data is stored elsewhere.
In either case, data is still stored locally. With the shared gateway,
data is restored from that data store when a node restarts. For more
information, highly recommend reading the docs thoroughly. Elasticsearch Platform — Find real-time answers at scale | Elastic
Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype
I have hadoop plugin with hdfs gateway but what I am seeing is that
indexes are still being written locally. Can you please help me understand
why it's being written locally?
ls -ltr data/elasticsearch/nodes/0/indices/twitter/0/index/
total 12
-rw-r--r-- 1 root root 0 May 14 15:33 write.lock
-rw-r--r-- 1 root root 20 May 14 15:33 segments.gen
-rw-r--r-- 1 root root 58 May 14 15:33 segments_1
-rw-r--r-- 1 root root 8 May 14 15:33 _checksums-1337034789114
Yes I read that and also have done recovery testing too, which seems to
recover everything. My question was when does elasticsearch writes/commits
data to Hadoop? Is it synchronously or async? Should I expect to lose any
data that might be in elasticsearch memory? Just trying to understand the
basics.
On Mon, May 14, 2012 at 4:23 PM, Berkay Mollamustafaoglu < mberkay@gmail.com> wrote:
There are two ways to configure ES for persistence (for the data to
survive full cluster restart)
Local gateway, where the data persists on the servers
Shared or central gateway (S3, Hadoop, or shared file system) where
data is stored elsewhere.
In either case, data is still stored locally. With the shared gateway,
data is restored from that data store when a node restarts. For more
information, highly recommend reading the docs thoroughly. Elasticsearch Platform — Find real-time answers at scale | Elastic
Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype
I have hadoop plugin with hdfs gateway but what I am seeing is that
indexes are still being written locally. Can you please help me understand
why it's being written locally?
ls -ltr data/elasticsearch/nodes/0/indices/twitter/0/index/
total 12
-rw-r--r-- 1 root root 0 May 14 15:33 write.lock
-rw-r--r-- 1 root root 20 May 14 15:33 segments.gen
-rw-r--r-- 1 root root 58 May 14 15:33 segments_1
-rw-r--r-- 1 root root 8 May 14 15:33 _checksums-1337034789114
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.