Request for elastic search configuration

srikanth_ramineni · December 19, 2013, 12:53am

Hi Team,

We have below server configurations. We are trying to load more than 500
million data into elastic. When data reaches 40 millions, shards are going
down and data is deleting . Can you please help us with best configuration
for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

warkolm · December 19, 2013, 1:02am

It will help if you post (via a gist/pastebin/etc) what you have now.

Also what sort of data is it, how are you loading it, what versions of
things are you running on (OS, java, ES, etc etc)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 11:53, srikanth ramineni ri.srikanth@gmail.com wrote:

Hi Team,

We have below server configurations. We are trying to load more than 500
million data into elastic. When data reaches 40 millions, shards are going
down and data is deleting . Can you please help us with best configuration
for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

srikanth_ramineni · December 19, 2013, 3:18am

Hi Mark Walkom,

Thanks for your quick response...

Sorry , I am not sure how to post from gist/pastebin/etc. Below are your
requested details

Operating system : Linux
ES version : Elasticsearch-0.90.7
JAVA Version :JDK 1.7
Data : we didn't have any parent child relations. Just we are loading with
auto generated primary key.(29,666,482 documents= 21.9 GB).

Data Loading : we are loading data into elastic by writing 20 threads.

Client creation :

Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name",
"UCS_DEV_ES_CLUSTER").build();

elasticClient = new TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress("server1", 9300))
.addTransportAddress(new InetSocketTransportAddress("server2", 9300));
Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Please find enclosed elasticsearch.yml file which we are using.

Thanks,
Srikanth.

On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom markw@campaignmonitor.comwrote:

It will help if you post (via a gist/pastebin/etc) what you have now.

Also what sort of data is it, how are you loading it, what versions of
things are you running on (OS, java, ES, etc etc)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 11:53, srikanth ramineni ri.srikanth@gmail.comwrote:

Hi Team,

We have below server configurations. We are trying to load more than 500
million data into elastic. When data reaches 40 millions, shards are going
down and data is deleting . Can you please help us with best configuration
for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

warkolm · December 19, 2013, 3:28am

http://pastebin.com/ - Please don't send attachments to the list, it goes
to hundreds/thousands of people.

What JDK is it, oracle or open?
What sort of data is it - logs? Web pages? Text?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 14:18, srikanth ramineni ri.srikanth@gmail.com wrote:

Hi Mark Walkom,

Thanks for your quick response...

Sorry , I am not sure how to post from gist/pastebin/etc. Below are your
requested details

Operating system : Linux
ES version : Elasticsearch-0.90.7
JAVA Version :JDK 1.7
Data : we didn't have any parent child relations. Just we are loading with
auto generated primary key.(29,666,482 documents= 21.9 GB).

Data Loading : we are loading data into elastic by writing 20 threads.

Client creation :

Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name",
"UCS_DEV_ES_CLUSTER").build();

elasticClient = new TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress("server1", 9300))
.addTransportAddress(new InetSocketTransportAddress("server2", 9300));
Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Please find enclosed elasticsearch.yml file which we are using.

Thanks,
Srikanth.

On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom markw@campaignmonitor.comwrote:

It will help if you post (via a gist/pastebin/etc) what you have now.

Also what sort of data is it, how are you loading it, what versions of
things are you running on (OS, java, ES, etc etc)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 11:53, srikanth ramineni ri.srikanth@gmail.comwrote:

Hi Team,

We have below server configurations. We are trying to load more than
500 million data into elastic. When data reaches 40 millions, shards are
going down and data is deleting . Can you please help us with best
configuration for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b2JyfYmC5_mZkz%2BXQe_4hzrr79ALx3Hj0ww6%3DTeqHOHQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

srikanth_ramineni · December 19, 2013, 3:49am

Hi Mark Walkom,

I have updated the detail in pastebin . Pleae find below url for details.

http://pastebin.com/62aPBkPc

Please get back to me for further quries.

thanks,
Srikanth.

On Wed, Dec 18, 2013 at 7:28 PM, Mark Walkom markw@campaignmonitor.comwrote:

http://pastebin.com/ - Please don't send attachments to the list, it goes
to hundreds/thousands of people.

What JDK is it, oracle or open?
What sort of data is it - logs? Web pages? Text?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 14:18, srikanth ramineni ri.srikanth@gmail.comwrote:

Hi Mark Walkom,

Thanks for your quick response...

Sorry , I am not sure how to post from gist/pastebin/etc. Below are
your requested details

Operating system : Linux
ES version : Elasticsearch-0.90.7
JAVA Version :JDK 1.7
Data : we didn't have any parent child relations. Just we are loading
with auto generated primary key.(29,666,482 documents= 21.9 GB).

Data Loading : we are loading data into elastic by writing 20 threads.

Client creation :

Settings settings = ImmutableSettings.settingsBuilder().put("
cluster.name", "UCS_DEV_ES_CLUSTER").build();

elasticClient = new TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress("server1", 9300))
.addTransportAddress(new InetSocketTransportAddress("server2", 9300));
Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Please find enclosed elasticsearch.yml file which we are using.

Thanks,
Srikanth.

On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom markw@campaignmonitor.comwrote:

It will help if you post (via a gist/pastebin/etc) what you have now.

Also what sort of data is it, how are you loading it, what versions of
things are you running on (OS, java, ES, etc etc)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 11:53, srikanth ramineni ri.srikanth@gmail.comwrote:

Hi Team,

We have below server configurations. We are trying to load more than
500 million data into elastic. When data reaches 40 millions, shards are
going down and data is deleting . Can you please help us with best
configuration for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b2JyfYmC5_mZkz%2BXQe_4hzrr79ALx3Hj0ww6%3DTeqHOHQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGuNUAUNS3gwX-jNjz4_PL7TyY3qG0d0zCb9Dm5WHsRpCRQQ3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

warkolm · December 19, 2013, 4:01am

You will want to use Oracle java rather than OpenJDK, you will get a lot
better performance.

Now next question, what are you using for monitoring your cluster? Are you
using plugins like ElasticHQ, kopf, bighead, head, paramedic etc etc. If
not install one of those and it will give you insight into what is
happening with your cluster and what possible causes for the problems might
be.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 14:49, srikanth ramineni ri.srikanth@gmail.com wrote:

Hi Mark Walkom,

I have updated the detail in pastebin . Pleae find below url for details.

http://pastebin.com/62aPBkPc

Please get back to me for further quries.

thanks,
Srikanth.

On Wed, Dec 18, 2013 at 7:28 PM, Mark Walkom markw@campaignmonitor.comwrote:

http://pastebin.com/ - Please don't send attachments to the list, it
goes to hundreds/thousands of people.

What JDK is it, oracle or open?
What sort of data is it - logs? Web pages? Text?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 14:18, srikanth ramineni ri.srikanth@gmail.comwrote:

Hi Mark Walkom,

Thanks for your quick response...

Sorry , I am not sure how to post from gist/pastebin/etc. Below are
your requested details

Operating system : Linux
ES version : Elasticsearch-0.90.7
JAVA Version :JDK 1.7
Data : we didn't have any parent child relations. Just we are loading
with auto generated primary key.(29,666,482 documents= 21.9 GB).

Data Loading : we are loading data into elastic by writing 20 threads.

Client creation :

Settings settings = ImmutableSettings.settingsBuilder().put("
cluster.name", "UCS_DEV_ES_CLUSTER").build();

elasticClient = new TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress("server1", 9300))
.addTransportAddress(new InetSocketTransportAddress("server2", 9300));
Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Please find enclosed elasticsearch.yml file which we are using.

Thanks,
Srikanth.

On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom markw@campaignmonitor.comwrote:

It will help if you post (via a gist/pastebin/etc) what you have now.

Also what sort of data is it, how are you loading it, what versions of
things are you running on (OS, java, ES, etc etc)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 11:53, srikanth ramineni ri.srikanth@gmail.comwrote:

Hi Team,

We have below server configurations. We are trying to load more than
500 million data into elastic. When data reaches 40 millions, shards are
going down and data is deleting . Can you please help us with best
configuration for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b2JyfYmC5_mZkz%2BXQe_4hzrr79ALx3Hj0ww6%3DTeqHOHQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGuNUAUNS3gwX-jNjz4_PL7TyY3qG0d0zCb9Dm5WHsRpCRQQ3Q%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZLXXZMzchGSewoDhK9J3kK0WZjqxw8Pth9g2E-%3Du29MQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

srikanth_ramineni · December 19, 2013, 4:30am

Hi,

We installed head plugin for monitoring the cluster. once it loaded 27GB,
cluster is going to unavailable. We are not able to understand exact
problem..

Can you please suggest elasticsearch.yml file configurations for 4 nodes
for single cluster. we are loading data from oracle to elastic by using
java threads..

Thanks,
Srikanth.

On Wed, Dec 18, 2013 at 8:01 PM, Mark Walkom markw@campaignmonitor.comwrote:

You will want to use Oracle java rather than OpenJDK, you will get a lot
better performance.

Now next question, what are you using for monitoring your cluster? Are you
using plugins like ElasticHQ, kopf, bighead, head, paramedic etc etc. If
not install one of those and it will give you insight into what is
happening with your cluster and what possible causes for the problems might
be.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 14:49, srikanth ramineni ri.srikanth@gmail.comwrote:

Hi Mark Walkom,

I have updated the detail in pastebin . Pleae find below url for details.

http://pastebin.com/62aPBkPc

Please get back to me for further quries.

thanks,
Srikanth.

On Wed, Dec 18, 2013 at 7:28 PM, Mark Walkom markw@campaignmonitor.comwrote:

http://pastebin.com/ - Please don't send attachments to the list, it
goes to hundreds/thousands of people.

What JDK is it, oracle or open?
What sort of data is it - logs? Web pages? Text?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 14:18, srikanth ramineni ri.srikanth@gmail.comwrote:

Hi Mark Walkom,

Thanks for your quick response...

Sorry , I am not sure how to post from gist/pastebin/etc. Below are
your requested details

Operating system : Linux
ES version : Elasticsearch-0.90.7
JAVA Version :JDK 1.7
Data : we didn't have any parent child relations. Just we are loading
with auto generated primary key.(29,666,482 documents= 21.9 GB).

Data Loading : we are loading data into elastic by writing 20 threads.

Client creation :

Settings settings = ImmutableSettings.settingsBuilder().put("
cluster.name", "UCS_DEV_ES_CLUSTER").build();

elasticClient = new TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress("server1", 9300))
.addTransportAddress(new InetSocketTransportAddress("server2", 9300));
Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Please find enclosed elasticsearch.yml file which we are using.

Thanks,
Srikanth.

On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom <markw@campaignmonitor.com

wrote:

It will help if you post (via a gist/pastebin/etc) what you have now.

Also what sort of data is it, how are you loading it, what versions of
things are you running on (OS, java, ES, etc etc)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 11:53, srikanth ramineni ri.srikanth@gmail.comwrote:

Hi Team,

We have below server configurations. We are trying to load more than
500 million data into elastic. When data reaches 40 millions, shards are
going down and data is deleting . Can you please help us with best
configuration for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b2JyfYmC5_mZkz%2BXQe_4hzrr79ALx3Hj0ww6%3DTeqHOHQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGuNUAUNS3gwX-jNjz4_PL7TyY3qG0d0zCb9Dm5WHsRpCRQQ3Q%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZLXXZMzchGSewoDhK9J3kK0WZjqxw8Pth9g2E-%3Du29MQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGuNUAWyNOXr90qf%2BdUu9NqG1J%3DJwiDm-2yXJ5FcuPNy28oznA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · December 19, 2013, 6:12am

Anything in logs?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 19 déc. 2013 à 05:30, srikanth ramineni ri.srikanth@gmail.com a écrit :

Hi,

We installed head plugin for monitoring the cluster. once it loaded 27GB, cluster is going to unavailable. We are not able to understand exact problem..

Can you please suggest elasticsearch.yml file configurations for 4 nodes for single cluster. we are loading data from oracle to elastic by using java threads..

Thanks,
Srikanth.

On Wed, Dec 18, 2013 at 8:01 PM, Mark Walkom markw@campaignmonitor.com wrote:
You will want to use Oracle java rather than OpenJDK, you will get a lot better performance.

Now next question, what are you using for monitoring your cluster? Are you using plugins like ElasticHQ, kopf, bighead, head, paramedic etc etc. If not install one of those and it will give you insight into what is happening with your cluster and what possible causes for the problems might be.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com
On 19 December 2013 14:49, srikanth ramineni ri.srikanth@gmail.com wrote:
Hi Mark Walkom,

I have updated the detail in pastebin . Pleae find below url for details.

http://pastebin.com/62aPBkPc

Please get back to me for further quries.

thanks,
Srikanth.
On Wed, Dec 18, 2013 at 7:28 PM, Mark Walkom markw@campaignmonitor.com wrote:
http://pastebin.com/ - Please don't send attachments to the list, it goes to hundreds/thousands of people.

What JDK is it, oracle or open?
What sort of data is it - logs? Web pages? Text?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com
On 19 December 2013 14:18, srikanth ramineni ri.srikanth@gmail.com wrote:
Hi Mark Walkom,

Thanks for your quick response...

Sorry , I am not sure how to post from gist/pastebin/etc. Below are your requested details

Operating system : Linux
ES version : Elasticsearch-0.90.7
JAVA Version :JDK 1.7
Data : we didn't have any parent child relations. Just we are loading with auto generated primary key.(29,666,482 documents= 21.9 GB).

Data Loading : we are loading data into elastic by writing 20 threads.

Client creation :

Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "UCS_DEV_ES_CLUSTER").build();
   elasticClient = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress("server1", 9300))
  												 .addTransportAddress(new InetSocketTransportAddress("server2", 9300));
Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Please find enclosed elasticsearch.yml file which we are using.

Thanks,
Srikanth.

On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom markw@campaignmonitor.com wrote:
It will help if you post (via a gist/pastebin/etc) what you have now.

Also what sort of data is it, how are you loading it, what versions of things are you running on (OS, java, ES, etc etc)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 19 December 2013 11:53, srikanth ramineni ri.srikanth@gmail.com wrote:
Hi Team,

We have below server configurations. We are trying to load more than 500 million data into elastic. When data reaches 40 millions, shards are going down and data is deleting . Can you please help us with best configuration for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b2JyfYmC5_mZkz%2BXQe_4hzrr79ALx3Hj0ww6%3DTeqHOHQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGuNUAUNS3gwX-jNjz4_PL7TyY3qG0d0zCb9Dm5WHsRpCRQQ3Q%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZLXXZMzchGSewoDhK9J3kK0WZjqxw8Pth9g2E-%3Du29MQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGuNUAWyNOXr90qf%2BdUu9NqG1J%3DJwiDm-2yXJ5FcuPNy28oznA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/29E8DADA-93DD-4A4A-BA22-AA3DEB6652A4%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

shailendra_pathak9 · December 19, 2013, 10:44am

Hi Srikanth,and all elasticsearch developers.

i need your help regarding deploying ES server using chef solo..i have
made one ES instance ..i have 7 servers .can you please provide full
procedures for the same..

Thanks

On Thursday, December 19, 2013 6:23:45 AM UTC+5:30, srikanth ramineni wrote:

Hi Team,

We have below server configurations. We are trying to load more than 500
million data into elastic. When data reaches 40 millions, shards are going
down and data is deleting . Can you please help us with best configuration
for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e65841e-3edc-4a1a-b16a-dbf2c265a27c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · December 19, 2013, 6:47pm

Are you looking for this? Elasticsearch Platform — Find real-time answers at scale | Elastic

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 19 déc. 2013 à 11:44, shailendra.pathak9@gmail.com a écrit :

Hi Srikanth,and all elasticsearch developers.

i need your help regarding deploying ES server using chef solo..i have made one ES instance ..i have 7 servers .can you please provide full procedures for the same..

Thanks

On Thursday, December 19, 2013 6:23:45 AM UTC+5:30, srikanth ramineni wrote:
Hi Team,

We have below server configurations. We are trying to load more than 500 million data into elastic. When data reaches 40 millions, shards are going down and data is deleting . Can you please help us with best configuration for above requirement.

Below are the server details

we have 4 nodes.

Each node contains
CPU: 4 , RAM: 16G , Storage: 122G
ES_HEAP_SIZE 8 G

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e65841e-3edc-4a1a-b16a-dbf2c265a27c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/BA543D32-8882-4978-86AA-0FBC16EF77C1%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.