.
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/447603d4-7a2c-40ae-b560-2f4b9cac67f6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/447603d4-7a2c-40ae-b560-2f4b9cac67f6%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/447603d4-7a2c-40ae-b560-2f4b9cac67f6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/447603d4-7a2c-40ae-b560-2f4b9cac67f6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8d73f1ac-77f0-4335-9438-61ecf137592c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8d73f1ac-77f0-4335-9438-61ecf137592c%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Jaguar](https://discuss.elastic.co/u/Jaguar)
#### Post date: [May 28, 2014, 5:07am UTC](https://discuss.elastic.co/t/looking-for-heavy-write-optimization/17768/7 "2014-05-28T05:07:10Z")
You could test with a single index and get an idea of write performance on
single one. One nice thing about ES is it scales almost linear, more
concurrent indices setup for writing, better write performance.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAP0hgQ09cN%2BEhZEnGBvrr7EUqkOnvLWLw-1f7JeMMjyKqx3KpQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAP0hgQ09cN%2BEhZEnGBvrr7EUqkOnvLWLw-1f7JeMMjyKqx3KpQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [May 28, 2014, 6:49am UTC](https://discuss.elastic.co/t/looking-for-heavy-write-optimization/17768/8 "2014-05-28T06:49:04Z")
For maximum write performance, you should
- use fastest disk subsystem (SSD)
- use RAID 0 with expensive controller to max out IO bandwidth
- do not run more than one ES instance per server
- do not use virtual servers, use physical servers
- for ES data folder, disable acess time flag (noatime), check filesystem
settings for more tweaking
- use mlockall and mmapfs (niofs is not faster than mmapfs)
- us bulk indexing (note that UDP is not reliable network protocol)
- reserve enough off-heap memory for mmapfs (you do not need large heaps
for fast bulk indexing)
- monitor network bandwidth for enough capacity
- for high sustainable bulk write ES configuration, see
[Elasticsearch configuration for high sustainable bulk feed · GitHub](https://gist.github.com/jprante/10666960)
- on SSD, disable Linux CFQ scheduler for noop scheduler, see Drew Raines'
presentation [https://speakerdeck.com/drewr/life-after-ec2](https://speakerdeck.com/drewr/life-after-ec2)
Jörg
On Wed, May 28, 2014 at 7:07 AM, 熊贻青 [xiong.jaguar@gmail.com](mailto:xiong.jaguar@gmail.com) wrote:
> You could test with a single index and get an idea of write performance on
> single one. One nice thing about ES is it scales almost linear, more
> concurrent indices setup for writing, better write performance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAP0hgQ09cN%2BEhZEnGBvrr7EUqkOnvLWLw-1f7JeMMjyKqx3KpQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAP0hgQ09cN%2BEhZEnGBvrr7EUqkOnvLWLw-1f7JeMMjyKqx3KpQ%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAP0hgQ09cN%2BEhZEnGBvrr7EUqkOnvLWLw-1f7JeMMjyKqx3KpQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAP0hgQ09cN%2BEhZEnGBvrr7EUqkOnvLWLw-1f7JeMMjyKqx3KpQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFOLO1x3zkQB%2BqRFQD9Z%2BuLhXM%3DenHCjN-wj\_yLwtUXKw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFOLO1x3zkQB%2BqRFQD9Z%2BuLhXM%3DenHCjN-wj_yLwtUXKw%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@otisg](https://discuss.elastic.co/u/otisg)
#### Post date: [May 30, 2014, 5:10am UTC](https://discuss.elastic.co/t/looking-for-heavy-write-optimization/17768/9 "2014-05-30T05:10:13Z")
Hi,
I see Jorg already provided a nice list of suggestions. But check that FS
type - ext2! That's ancient! Try ext3, ext4, or xfs. If you turn off
journaling things will be faster. You are using UDP you said, so you must
be OK with some data loss.
Btw. SPM [http://sematext.com/spm/](http://sematext.com/spm/) can monitor all 3 things you mentioned
- Kafka - Storm - ES. May be helpful to have all these metrics in a single
UI when optimizing things.
## Otis
Performance Monitoring \* Log Analytics \* Search Analytics
Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/)
On Tuesday, May 27, 2014 11:55:49 PM UTC-4, Zealot Yin wrote:
> CPU E5-2620 12 cores
> MEM 64G
> df -lhT
> Filesystem Type Size Used Avail Use% Mounted on
> /dev/sda3 ext2 1.4T 379G 962G 29% /home
>
> uname -a
> 2.6.32\_1-9-0-0 10 17:22:16 CST 2013 x86\_64 x86\_64 x86\_64 GNU/Linux
>
> elasticsearch started with _bin/elasticsearch -Xms10g -Xmx10g
> -Des.index.store.type=niofs -d_
>
> On Wednesday, May 28, 2014 11:49:17 AM UTC+8, Mark Walkom wrote:
>
> > What sort of hardware are you on?
> >
> > Regards,
> > Mark Walkom
> >
> > Infrastructure Engineer
> > Campaign Monitor
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> >
> > On 28 May 2014 13:44, Zealot Yin [0xma...@gmail.com](mailto:0xma...@gmail.com) wrote:
> >
> > > hi
> > >
> > > I am writing a real time analytics tool using kafka,storm and
> > > elasticsearch and want a elasticsearch that is write optimized for about
> > > 80K/sec inserts(7 machine cluster).
> > >
> > > For the purpose of high performance i using bulk udp to batch insert my
> > > doc(each doc is about 300B, only 4 field are indexed),and i'm using index.store.type=niofs
> > > for index store(mmap cause io-util 100%),but it's still seem not good
> > > enough for my scene, _all thing i need is writing performance, anybody
> > > has a idea for this problem?_
> > >
> > > here is my conf:
> > >
> > > bootstrap.mlockall: true
> > >
> > > threadpool.bulk.type: fixed
> > >
> > > threadpool.bulk.size: 30
> > >
> > > threadpool.bulk.queue\_size: 500
> > >
> > > index.refresh\_interval: 50s
> > >
> > > indices.store.throttle.type: merge
> > >
> > > indices.store.throttle.max\_bytes\_per\_sec: 80mb
> > >
> > > indices.memory.index\_buffer\_size: 30%
> > >
> > > indices.ttl.bulk\_size: 100000
> > >
> > > indices.memory.min\_shard\_index\_buffer\_size: 200mb
> > >
> > > bulk.udp.enabled : true
> > >
> > > bulk.udp.bulk\_actions : 10000
> > >
> > > bulk.udp.bulk\_size : 20mb
> > >
> > > bulk.udp.flush\_interval : 10s
> > >
> > > bulk.udp.concurrent\_requests : 4000
> > >
> > > bulk.udp.receive\_buffer\_size : 10mb
> > >
> > > index.cache.field.expire: 10m
> > >
> > > index.cache.field.max\_size: 500000
> > >
> > > index.cache.field.type: soft
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/c444a899-dcb3-442a-a33e-2b987fcad2e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c444a899-dcb3-442a-a33e-2b987fcad2e6%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/c444a899-dcb3-442a-a33e-2b987fcad2e6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c444a899-dcb3-442a-a33e-2b987fcad2e6%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/57fa553f-a376-4b01-83ef-486dd62d8f7a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/57fa553f-a376-4b01-83ef-486dd62d8f7a%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:25am UTC](https://discuss.elastic.co/t/looking-for-heavy-write-optimization/17768/10 "2017-07-06T01:25:58Z")