The indexing speed become more and more slowly

Elastic Stack Elasticsearch
1

Hi all,

  My elasticsearch indexing become more and more slowly by the size of

index increase.
e.x.
The 1 GB file indexing need 10 min at the beginning, and it become
more and more slowly.
like here:
1time 1GB file ------------- 10min indexing
2time 1GB file ------------- 12min indexing
3time 1GB file ------------- 14min indexing

  I found the indexing rate became 1 MB/s near the end.

  Is it normal?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

it's not :slight_smile: can we get more info about your cluster / setup. I am wondering
about things like:

  • indexing rate
  • are you doing bulk
  • memory settings
  • hardware configuration
  • any special settings like merge throtteling or custom merge policies
  • are you searching on the same cluster at the same time?

simon

On Monday, March 4, 2013 9:48:05 AM UTC+1, Hoony wrote:

Hi all,

  My elasticsearch indexing become more and more slowly by the size of

index increase.
e.x.
The 1 GB file indexing need 10 min at the beginning, and it become
more and more slowly.
like here:
1time 1GB file ------------- 10min indexing
2time 1GB file ------------- 12min indexing
3time 1GB file ------------- 14min indexing

  I found the indexing rate became 1 MB/s near the end.

  Is it normal?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

My cluster environment here:

  • indexing rate - I am not sure what you need, because the rate is not
    established and the speed becomes more and more slow , as I said before.
  • are you doing bulk - I used bulk and the bulk size is 2000.
  • memory settings - ram is 16GB and configured 8GB for ES
  • hardware configuration - CPU 4core 3.3 ghz
  • any special settings like merge throtteling or custom merge policies
  • I used the default merge setting.
  • are you searching on the same cluster at the same time? - I just
    indexing without searching on my cluster at the same time.

在 2013年3月4日星期一UTC+9下午6时11分33秒，simonw写道：

it's not :slight_smile: can we get more info about your cluster / setup. I am
wondering about things like:

  • indexing rate
  • are you doing bulk
  • memory settings
  • hardware configuration
  • any special settings like merge throtteling or custom merge policies
  • are you searching on the same cluster at the same time?

simon

On Monday, March 4, 2013 9:48:05 AM UTC+1, Hoony wrote:

Hi all,

  My elasticsearch indexing become more and more slowly by the size

of index increase.
e.x.
The 1 GB file indexing need 10 min at the beginning, and it become
more and more slowly.
like here:
1time 1GB file ------------- 10min indexing
2time 1GB file ------------- 12min indexing
3time 1GB file ------------- 14min indexing

  I found the indexing rate became 1 MB/s near the end.

  Is it normal?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi,

I think Hoony may be seeing the effect of Lucene segment merging, no?

We just did some ES indexing benchmarking and observed the same thing.

Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Monday, March 4, 2013 4:11:33 AM UTC-5, simonw wrote:

it's not :slight_smile: can we get more info about your cluster / setup. I am
wondering about things like:

  • indexing rate
  • are you doing bulk
  • memory settings
  • hardware configuration
  • any special settings like merge throtteling or custom merge policies
  • are you searching on the same cluster at the same time?

simon

On Monday, March 4, 2013 9:48:05 AM UTC+1, Hoony wrote:

Hi all,

  My elasticsearch indexing become more and more slowly by the size

of index increase.
e.x.
The 1 GB file indexing need 10 min at the beginning, and it become
more and more slowly.
like here:
1time 1GB file ------------- 10min indexing
2time 1GB file ------------- 12min indexing
3time 1GB file ------------- 14min indexing

  I found the indexing rate became 1 MB/s near the end.

  Is it normal?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.