# ES write performance
**URL:** https://discuss.elastic.co/t/es-write-performance/14152
**Category:** Elasticsearch
**Created:** [October 29, 2013, 11:27am UTC](https://discuss.elastic.co/t/es-write-performance/14152 "2013-10-29T11:27:18Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 11:27am UTC](https://discuss.elastic.co/t/es-write-performance/14152/1 "2013-10-29T11:27:18Z")
Hi,
I wrote a python script using elasticsearch-py and another using pyes,
configured my bulk-size to be 5000 records (tested with more without
improvement), one node only, no refresh interval, no replicas, thrift
protocol and the node runs on top of SSD. The maximum insert performance
that I could achieve was 10.400 records per second. If I split my input
source by four, the write performance degrades to 6500/sec.
Interesting... if I put all the documents on a text file (332.400 total
documents ~100MB) and run the bulk insert from the curl command, I ended up
with total time of ~8 seconds which means ~41.000/sec.
Non-default ES config changes:
indices.memory.index\_buffer\_size: 50%
indices.memory.min\_index\_buffer\_size: 300mb
index.translog.flush\_threshold: 30000
index.store.type: mmapfs
index.merge.policy.use\_compound\_file: false
Is this write performance the best I could get with python? How can I
improve it?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [October 29, 2013, 12:21pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/2 "2013-10-29T12:21:13Z")
Some info is unknown about your setup: What about refresh interval during
bulk? What about concurrency in the bulk requests? What mapping do you use?
Do you measure the time after refresh, or optimize? Do you index over
network interface, from remote or local host? Do you use compression? How
much heap and RAM is available? How many CPU cores are running, at which
speed? How fast are sustainable writes by the SSD, is it using 3Gbit/s or
6Gbit/s interface (SATA) or even PCIe?
It is not easy to compare thrift and HTTP, since the protocols are very
different.
In Python I once was aware of a slow JSON standard codec but this can be
replaced.
Also I recommend describing indexing performance by MB/sec, not doc/sec,
since doc sizes may vary. Note that 100MB per 8 sec means ~12 MB/sec which
could be a limit in your single node hardware.
Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [October 29, 2013, 12:23pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/3 "2013-10-29T12:23:32Z")
Also to mention, the number of shards and replica, which affect a lot the
indexing performance.
Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)
#### Post date: [October 29, 2013, 12:55pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/4 "2013-10-29T12:55:55Z")
Hi,
and what was the bottle neck? Has the pyhton process maxed out the CPU or
was it waiting for network? You can try serializing the documents yourself
and passing json strings to the client's bulk() method to make sure that's
not the bottle neck (you can pass in list of strings or just one big string
and we will just pass it along).
The python client does more than curl - it serializes data and parses
output, that's at least 2 cpu intensive operations that need to happen. One
of them you can eliminate.
On Tue, Oct 29, 2013 at 1:23 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \<
[joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote:
> Also to mention, the number of shards and replica, which affect a lot the
> indexing performance.
>
> Jörg
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 12:59pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/5 "2013-10-29T12:59:21Z")
Hi Jorg,
First of all, thanks for your attention.
Not all info is unknown 🙂
- Refresh interval was set to -1
- Replica set to 0
- Shards: default (5)
- Concurrency? Was single process, multiple processes were introduced on a
second test just to see if there was an improvement, but actually was worse
- Mapping: {'worker':{'index': 'not\_analyzed','type': 'string',
'include\_in\_all' : 'true'}, 'filename':{'index': 'no','type': 'string',
'include\_in\_all' : 'false'}, 'extractionts':{'index': 'no','type': 'date',
'include\_in\_all' : 'false', 'format':'yyyy-MM-dd HH:mm'}, 'line': {'index':
'no','type': 'integer', 'include\_in\_all' : 'false'}, 'timest':{'index':
'not\_analyzed','type': 'date', 'include\_in\_all' : 'false',
'format':'dd/MMM/yyyy:HH:mm:ss'}, 'server':{'index': 'not\_analyzed','type':
'ip', 'include\_in\_all' : 'false'}, 'http\_method':{'index':
'not\_analyzed','type': 'string', 'include\_in\_all' : 'true'},'url':{'index':
'not\_analyzed','type': 'string', 'include\_in\_all' :
'true'},'http\_status':{'index': 'not\_analyzed','type': 'integer',
'include\_in\_all' : 'true'}}
- I measure the time after flushing at the end of sending all documents,
and only then I re-configure the refresh\_interval back to 1s and replica
back to 1
- I don't run optimize, I'm just focused on inserting data as fast as I can
get
- Single local node
- No compression
- Memory: 8Gb ram, 3gb set Xms=Xmx
- quad-core: 8 CPU 2.2Ghz
- SATA 3Gbit/s, performance tool gave me on average 217MB/s
- Average performance write ratio based on MB/s is 2.66MB/s!! too far from
the curl performance which was 12MB/sec
In the meantime, I'll look how can I change python default json encoder.
But this "enconding" happens on the python-es driver, right?!
On Tuesday, 29 October 2013 12:21:13 UTC, Jörg Prante wrote:
> Some info is unknown about your setup: What about refresh interval during
> bulk? What about concurrency in the bulk requests? What mapping do you use?
> Do you measure the time after refresh, or optimize? Do you index over
> network interface, from remote or local host? Do you use compression? How
> much heap and RAM is available? How many CPU cores are running, at which
> speed? How fast are sustainable writes by the SSD, is it using 3Gbit/s or
> 6Gbit/s interface (SATA) or even PCIe?
>
> It is not easy to compare thrift and HTTP, since the protocols are very
> different.
>
> In Python I once was aware of a slow JSON standard codec but this can be
> replaced.
>
> Also I recommend describing indexing performance by MB/sec, not doc/sec,
> since doc sizes may vary. Note that 100MB per 8 sec means ~12 MB/sec which
> could be a limit in your single node hardware.
>
> Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 1:07pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/6 "2013-10-29T13:07:58Z")
Hi Honza,
Ok, that could be a problem. I'm passing a python dictionary to pyes
driver. If I send a "string" json format I could pass the serialization?
Are you familiar with pyes driver?
I saw this method signature, but don't know what's the "header", and the
document can it be one full string with several documents?
index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
Function helper for fast inserting
Parameters:
- _header_ – a string with the bulk header must be ended with a newline
- _header_ – a json document string must be ended with a newline
On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
> Hi,
>
> and what was the bottle neck? Has the pyhton process maxed out the CPU or
> was it waiting for network? You can try serializing the documents yourself
> and passing json strings to the client's bulk() method to make sure that's
> not the bottle neck (you can pass in list of strings or just one big string
> and we will just pass it along).
>
> The python client does more than curl - it serializes data and parses
> output, that's at least 2 cpu intensive operations that need to happen. One
> of them you can eliminate.
>
> On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \ \<
> [joerg...@gmail.com](mailto:joerg...@gmail.com) \\> wrote:
>
> > Also to mention, the number of shards and replica, which affect a lot the
> > indexing performance.
> >
> > Jörg
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)
#### Post date: [October 29, 2013, 1:16pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/7 "2013-10-29T13:16:47Z")
I am not familiar with pyes, I did however write elasticsearch-py and made
sure you can bypass the serialization by doing it yourself. If needed you
can even supply your own serializer - just create an instance that has
.dumps() and loads() methods and behaves the same as
elasticsearch.serializer.JSONSerializer. you can then pass it to the
Elasticsearch class as an argument (serializer=my\_faster\_serializer)
On Tue, Oct 29, 2013 at 2:07 PM, Mauro Farracha [farracha@gmail.com](mailto:farracha@gmail.com) wrote:
> Hi Honza,
>
> Ok, that could be a problem. I'm passing a python dictionary to pyes
> driver. If I send a "string" json format I could pass the serialization?
> Are you familiar with pyes driver?
>
> I saw this method signature, but don't know what's the "header", and the
> document can it be one full string with several documents?
>
> index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
>
> Function helper for fast inserting
> Parameters:
>
> - _header_ – a string with the bulk header must be ended with a newline
> - _header_ – a json document string must be ended with a newline
>
> On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
>
> > Hi,
> >
> > and what was the bottle neck? Has the pyhton process maxed out the CPU or
> > was it waiting for network? You can try serializing the documents yourself
> > and passing json strings to the client's bulk() method to make sure that's
> > not the bottle neck (you can pass in list of strings or just one big string
> > and we will just pass it along).
> >
> > The python client does more than curl - it serializes data and parses
> > output, that's at least 2 cpu intensive operations that need to happen. One
> > of them you can eliminate.
> >
> > On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) [joerg...@gmail.com](mailto:joerg...@gmail.com)wrote:
> >
> > > Also to mention, the number of shards and replica, which affect a lot
> > > the indexing performance.
> > >
> > > Jörg
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > >
> > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > .
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [October 29, 2013, 1:51pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/8 "2013-10-29T13:51:12Z")
I suggest you should use the official python client
[https://github.com/elasticsearch/elasticsearch-py](https://github.com/elasticsearch/elasticsearch-py) instead of pyes, because
it has a much cleaner bulk API.
If you are sure your CPU is burnt with JSON serialization, maybe it is
worth to experiment with a faster JSON codec like ujson
[https://pypi.python.org/pypi/ujson/](https://pypi.python.org/pypi/ujson/) and replace standard json module in
[https://github.com/elasticsearch/elasticsearch-py/blob/master/elasticsearch/serializer.py](https://github.com/elasticsearch/elasticsearch-py/blob/master/elasticsearch/serializer.py)
Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 2:02pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/9 "2013-10-29T14:02:04Z")
Ahhhh you are the source! 🙂
As I mentioned on the first post, I wrote a python script using
elasticsearch-py also and the performance was equals to pyes, but I
couldn't get it working with Thrift. The documentation available for me was
not detailed enough so I could understand how to fully use all the features
and was a little bit confusing the Connection/Transport classes.
Maybe you could help me out... the error was:
self.client =
Elasticsearch(hosts=self.elasticsearch\_conn,connection\_class=ThriftConnection)
NameError: global name 'ThriftConnection' is not defined
I have ES thrift plugin installed (works on pyes), I have the thrift python
module installed and I import the class. Don't know what I'm missing.
On Tuesday, 29 October 2013 13:16:47 UTC, Honza Král wrote:
> I am not familiar with pyes, I did however write elasticsearch-py and made
> sure you can bypass the serialization by doing it yourself. If needed you
> can even supply your own serializer - just create an instance that has
> .dumps() and loads() methods and behaves the same as
> elasticsearch.serializer.JSONSerializer. you can then pass it to the
> Elasticsearch class as an argument (serializer=my\_faster\_serializer)
>
> On Tue, Oct 29, 2013 at 2:07 PM, Mauro Farracha \<[farr...@gmail.com](mailto:farr...@gmail.com)\
>
> > wrote:
>
> > Hi Honza,
> >
> > Ok, that could be a problem. I'm passing a python dictionary to pyes
> > driver. If I send a "string" json format I could pass the serialization?
> > Are you familiar with pyes driver?
> >
> > I saw this method signature, but don't know what's the "header", and the
> > document can it be one full string with several documents?
> >
> > index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
> >
> > Function helper for fast inserting
> > Parameters:
> >
> > - _header_ – a string with the bulk header must be ended with a
> > newline
> > - _header_ – a json document string must be ended with a newline
> >
> > On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
> >
> > > Hi,
> > >
> > > and what was the bottle neck? Has the pyhton process maxed out the CPU
> > > or was it waiting for network? You can try serializing the documents
> > > yourself and passing json strings to the client's bulk() method to make
> > > sure that's not the bottle neck (you can pass in list of strings or just
> > > one big string and we will just pass it along).
> > >
> > > The python client does more than curl - it serializes data and parses
> > > output, that's at least 2 cpu intensive operations that need to happen. One
> > > of them you can eliminate.
> > >
> > > On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) [joerg...@gmail.com](mailto:joerg...@gmail.com)wrote:
> > >
> > > > Also to mention, the number of shards and replica, which affect a lot
> > > > the indexing performance.
> > > >
> > > > Jörg
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > >
> > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > .
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)
#### Post date: [October 29, 2013, 2:32pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/10 "2013-10-29T14:32:12Z")
you need to import it before you intend to use it:
from elasticsearch.connection import ThriftConnection
On Tue, Oct 29, 2013 at 3:02 PM, Mauro Farracha [farracha@gmail.com](mailto:farracha@gmail.com) wrote:
> Ahhhh you are the source! 🙂
>
> As I mentioned on the first post, I wrote a python script using
> elasticsearch-py also and the performance was equals to pyes, but I
> couldn't get it working with Thrift. The documentation available for me was
> not detailed enough so I could understand how to fully use all the features
> and was a little bit confusing the Connection/Transport classes.
>
> Maybe you could help me out... the error was:
> self.client =
> Elasticsearch(hosts=self.elasticsearch\_conn,connection\_class=ThriftConnection)
> NameError: global name 'ThriftConnection' is not defined
>
> I have ES thrift plugin installed (works on pyes), I have the thrift
> python module installed and I import the class. Don't know what I'm missing.
>
> On Tuesday, 29 October 2013 13:16:47 UTC, Honza Král wrote:
>
> > I am not familiar with pyes, I did however write elasticsearch-py and
> > made sure you can bypass the serialization by doing it yourself. If needed
> > you can even supply your own serializer - just create an instance that has
> > .dumps() and loads() methods and behaves the same as
> > elasticsearch.serializer.\*\*JSONSerializer. you can then pass it to the
> > Elasticsearch class as an argument (serializer=my\_faster\_\*\*serializer)
> >
> > On Tue, Oct 29, 2013 at 2:07 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> >
> > > Hi Honza,
> > >
> > > Ok, that could be a problem. I'm passing a python dictionary to pyes
> > > driver. If I send a "string" json format I could pass the serialization?
> > > Are you familiar with pyes driver?
> > >
> > > I saw this method signature, but don't know what's the "header", and the
> > > document can it be one full string with several documents?
> > >
> > > index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
> > >
> > > Function helper for fast inserting
> > > Parameters:
> > >
> > > - _header_ – a string with the bulk header must be ended with a
> > > newline
> > > - _header_ – a json document string must be ended with a newline
> > >
> > > On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
> > >
> > > > Hi,
> > > >
> > > > and what was the bottle neck? Has the pyhton process maxed out the CPU
> > > > or was it waiting for network? You can try serializing the documents
> > > > yourself and passing json strings to the client's bulk() method to make
> > > > sure that's not the bottle neck (you can pass in list of strings or just
> > > > one big string and we will just pass it along).
> > > >
> > > > The python client does more than curl - it serializes data and parses
> > > > output, that's at least 2 cpu intensive operations that need to happen. One
> > > > of them you can eliminate.
> > > >
> > > > On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \<[joerg...@gmail.com](mailto:joerg...@gmail.com)
> > > >
> > > > > wrote:
> > > >
> > > > > Also to mention, the number of shards and replica, which affect a lot
> > > > > the indexing performance.
> > > > >
> > > > > Jörg
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to elasticsearc...@\*\*googlegroups.\*\*com.
> > > > >
> > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > .
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > .
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 3:02pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/11 "2013-10-29T15:02:37Z")
Thanks Honza.
I was importing the class. But not the right way 🙂 missing the connection
part in from.
I changed the serializer (used ujson as Jorg mentioned) and I got an
improvement from 2.66MB/s to 4.7MB/s.
Then I configured ThriftConnection and the write performance increased to
6.2MB/s.
Not bad, but still far off from the 12MB/s from curl.
Have two questions:
- using elasticsearch-py the index mapping is not the same on the server
side as when using pyes. Am I missing something? With pyes all the
properties were there, but using elasticsearch-py, only type appears on the
server side and are not the ones I specified. On the server log, It shows
"update\_mapping [accesslogs] (dynamic)" which doesn't happen with pyes. I'm
sure I'm missing some property/config.
- Also, can you guys share what's your performance on a single local node?
As I mention on my first post, these are my non-default configurations,
maybe there is still room for improvement? Not to mention of course, that
these same settings were responsible for the 12MB/s on curl.
indices.memory.index\_buffer\_size: 50%
indices.memory.min\_index\_buffer\_size: 300mb
index.translog.flush\_threshold: 30000
index.store.type: mmapfs
index.merge.policy.use\_compound\_file: false
On Tuesday, 29 October 2013 14:32:12 UTC, Honza Král wrote:
> you need to import it before you intend to use it:
>
> from elasticsearch.connection import ThriftConnection
>
> On Tue, Oct 29, 2013 at 3:02 PM, Mauro Farracha \<[farr...@gmail.com](mailto:farr...@gmail.com)\
>
> > wrote:
>
> > Ahhhh you are the source! 🙂
> >
> > As I mentioned on the first post, I wrote a python script using
> > elasticsearch-py also and the performance was equals to pyes, but I
> > couldn't get it working with Thrift. The documentation available for me was
> > not detailed enough so I could understand how to fully use all the features
> > and was a little bit confusing the Connection/Transport classes.
> >
> > Maybe you could help me out... the error was:
> > self.client =
> > Elasticsearch(hosts=self.elasticsearch\_conn,connection\_class=ThriftConnection)
> > NameError: global name 'ThriftConnection' is not defined
> >
> > I have ES thrift plugin installed (works on pyes), I have the thrift
> > python module installed and I import the class. Don't know what I'm missing.
> >
> > On Tuesday, 29 October 2013 13:16:47 UTC, Honza Král wrote:
> >
> > > I am not familiar with pyes, I did however write elasticsearch-py and
> > > made sure you can bypass the serialization by doing it yourself. If needed
> > > you can even supply your own serializer - just create an instance that has
> > > .dumps() and loads() methods and behaves the same as
> > > elasticsearch.serializer.\*\*JSONSerializer. you can then pass it to the
> > > Elasticsearch class as an argument (serializer=my\_faster\_\*\*serializer)
> > >
> > > On Tue, Oct 29, 2013 at 2:07 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > >
> > > > Hi Honza,
> > > >
> > > > Ok, that could be a problem. I'm passing a python dictionary to pyes
> > > > driver. If I send a "string" json format I could pass the serialization?
> > > > Are you familiar with pyes driver?
> > > >
> > > > I saw this method signature, but don't know what's the "header", and
> > > > the document can it be one full string with several documents?
> > > >
> > > > index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
> > > >
> > > > Function helper for fast inserting
> > > > Parameters:
> > > >
> > > > - _header_ – a string with the bulk header must be ended with a
> > > > newline
> > > > - _header_ – a json document string must be ended with a newline
> > > >
> > > > On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > and what was the bottle neck? Has the pyhton process maxed out the CPU
> > > > > or was it waiting for network? You can try serializing the documents
> > > > > yourself and passing json strings to the client's bulk() method to make
> > > > > sure that's not the bottle neck (you can pass in list of strings or just
> > > > > one big string and we will just pass it along).
> > > > >
> > > > > The python client does more than curl - it serializes data and parses
> > > > > output, that's at least 2 cpu intensive operations that need to happen. One
> > > > > of them you can eliminate.
> > > > >
> > > > > On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \<
> > > > > [joerg...@gmail.com](mailto:joerg...@gmail.com)\> wrote:
> > > > >
> > > > > > Also to mention, the number of shards and replica, which affect a lot
> > > > > > the indexing performance.
> > > > > >
> > > > > > Jörg
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com.
> > > > > >
> > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > > .
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > .
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [October 29, 2013, 3:30pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/12 "2013-10-29T15:30:29Z")
At least for http, if not also for thrift unless already included, I would
like to suggest gzip compression on the wire, but I'm not sure how the
python client can this enable.
Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)
#### Post date: [October 29, 2013, 4:01pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/13 "2013-10-29T16:01:17Z")
On Tue, Oct 29, 2013 at 4:02 PM, Mauro Farracha [farracha@gmail.com](mailto:farracha@gmail.com) wrote:
> Thanks Honza.
>
> I was importing the class. But not the right way 🙂 missing the connection
> part in from.
I thought it was the case, I update the code so that it should work in the
future.
> I changed the serializer (used ujson as Jorg mentioned) and I got an
> improvement from 2.66MB/s to 4.7MB/s.
ah, good to know, I will give it a try. I wanted to avoid additional
dependencies, but if it makes sense I will happily switch the client to
ujson. have you also tried just passing in a big string?
> Then I configured ThriftConnection and the write performance increased to
> 6.2MB/s.
>
> Not bad, but still far off from the 12MB/s from curl.
we still have to deserialize the response which curl doesn't need to do so
it will always have an advantage on us I am afraid, it shouldn't be this
big though.
> Have two questions:
>
> - using elasticsearch-py the index mapping is not the same on the server
> side as when using pyes. Am I missing something? With pyes all the
> properties were there, but using elasticsearch-py, only type appears on the
> server side and are not the ones I specified. On the server log, It shows
> "update\_mapping [accesslogs] (dynamic)" which doesn't happen with pyes. I'm
> sure I'm missing some property/config.
body should also include the doc\_type, so:
self.client.indices.put\_mapping(index=self.doc\_collection,
doc\_type=self.doc\_type,body={self.\_doc\_type:
{'properties':self.doc\_mapping}})
> - Also, can you guys share what's your performance on a single local node?
I haven't done any tests like this, it varies so much with different
HW/configuration/environment that there is little value in absolute
numbers, only thing that matters is the relative speed of python clients,
curl etc.
> As I mention on my first post, these are my non-default configurations,
> maybe there is still room for improvement? Not to mention of course, that
> these same settings were responsible for the 12MB/s on curl.
>
> indices.memory.index\_buffer\_\*\*size: 50%
> indices.memory.min\_index\_\*\*buffer\_size: 300mb
> index.translog.flush\_\*\*threshold: 30000
> index.store.type: mmapfs
> index.merge.policy.use\_\*\*compound\_file: false
these look reasonable though I am no expert. Also when using SSDs you might
benefit from switching the kernel IO scheduler to noop:
> **[Life After EC2](https://speakerdeck.com/elasticsearch/life-after-ec2)**
>
> A journey from slow recovery to realized potential.
> On Tuesday, 29 October 2013 14:32:12 UTC, Honza Král wrote:
>
> > you need to import it before you intend to use it:
> >
> > from elasticsearch.connection import ThriftConnection
> >
> > On Tue, Oct 29, 2013 at 3:02 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> >
> > > Ahhhh you are the source! 🙂
> > >
> > > As I mentioned on the first post, I wrote a python script using
> > > elasticsearch-py also and the performance was equals to pyes, but I
> > > couldn't get it working with Thrift. The documentation available for me was
> > > not detailed enough so I could understand how to fully use all the features
> > > and was a little bit confusing the Connection/Transport classes.
> > >
> > > Maybe you could help me out... the error was:
> > > self.client = Elasticsearch(hosts=self.\*_elasticsearch\_conn,connection\__
> > > \*class=ThriftConnection)
> > > NameError: global name 'ThriftConnection' is not defined
> > >
> > > I have ES thrift plugin installed (works on pyes), I have the thrift
> > > python module installed and I import the class. Don't know what I'm missing.
> > >
> > > On Tuesday, 29 October 2013 13:16:47 UTC, Honza Král wrote:
> > >
> > > > I am not familiar with pyes, I did however write elasticsearch-py and
> > > > made sure you can bypass the serialization by doing it yourself. If needed
> > > > you can even supply your own serializer - just create an instance that has
> > > > .dumps() and loads() methods and behaves the same as
> > > > elasticsearch.serializer. **JSONS** erializer. you can then pass it to
> > > > the Elasticsearch class as an argument (serializer=my\_faster\_\*\*serializ
> > > > \*\*er)
> > > >
> > > > On Tue, Oct 29, 2013 at 2:07 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > > >
> > > > > Hi Honza,
> > > > >
> > > > > Ok, that could be a problem. I'm passing a python dictionary to pyes
> > > > > driver. If I send a "string" json format I could pass the serialization?
> > > > > Are you familiar with pyes driver?
> > > > >
> > > > > I saw this method signature, but don't know what's the "header", and
> > > > > the document can it be one full string with several documents?
> > > > >
> > > > > index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
> > > > >
> > > > > Function helper for fast inserting
> > > > > Parameters:
> > > > >
> > > > > - _header_ – a string with the bulk header must be ended with a
> > > > > newline
> > > > > - _header_ – a json document string must be ended with a newline
> > > > >
> > > > > On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > and what was the bottle neck? Has the pyhton process maxed out the
> > > > > > CPU or was it waiting for network? You can try serializing the documents
> > > > > > yourself and passing json strings to the client's bulk() method to make
> > > > > > sure that's not the bottle neck (you can pass in list of strings or just
> > > > > > one big string and we will just pass it along).
> > > > > >
> > > > > > The python client does more than curl - it serializes data and parses
> > > > > > output, that's at least 2 cpu intensive operations that need to happen. One
> > > > > > of them you can eliminate.
> > > > > >
> > > > > > On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \<
> > > > > > [joerg...@gmail.com](mailto:joerg...@gmail.com)\> wrote:
> > > > > >
> > > > > > > Also to mention, the number of shards and replica, which affect a
> > > > > > > lot the indexing performance.
> > > > > > >
> > > > > > > Jörg
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > > > > Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > send an email to elasticsearc...@\*\*googlegroups. **c** om.
> > > > > > >
> > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)\*\*\*\*
> > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > > an email to elasticsearc...@\*\*googlegroups.\*\*com.
> > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > > .
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > .
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 4:13pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/14 "2013-10-29T16:13:28Z")
Hmmm... I'll try to investigate this lead, but compression adds cpu
processing and probably ES would need to decompress?
On Tuesday, 29 October 2013 15:30:29 UTC, Jörg Prante wrote:
> At least for http, if not also for thrift unless already included, I would
> like to suggest gzip compression on the wire, but I'm not sure how the
> python client can this enable.
>
> Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [October 29, 2013, 4:19pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/15 "2013-10-29T16:19:47Z")
It's a space/time tradeoff, compressed data needs a fraction of network
transport resources and saves memory overhead. If that is more gain than a
CPU spends in compressing/decompressing, it's a win. The larger the data,
the more compressing data wins.
Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 4:23pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/16 "2013-10-29T16:23:55Z")
Yep, makes sense when we have network in the middle.
On Tuesday, 29 October 2013 16:19:47 UTC, Jörg Prante wrote:
> It's a space/time tradeoff, compressed data needs a fraction of network
> transport resources and saves memory overhead. If that is more gain than a
> CPU spends in compressing/decompressing, it's a win. The larger the data,
> the more compressing data wins.
>
> Jörg
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 4:27pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/17 "2013-10-29T16:27:32Z")
Hi Honza,
Yep, I understand the issue around dependencies. The minimum you can do is
probably add this sort of information in documentation.
Regarding the mapping issue, you were right, adding the index type solved
the problem.
Since elasticsearch-py uses connection pooling with round-robin by default,
I was wondering if I could get more improvement if I had two nodes up,
since I would distribute the load between two servers, but using
ThriftConnection it throws an error which I don't understand why It happens
since Im pretty sure that Im passing the right configuration:
connection\_pool.py", line 60, in select
self.rr %= len(connections)
ZeroDivisionError: integer division or modulo by zero
Scenarios:
- two node, sniff\_\* properties =\> zerodivisionerror
- one node, sniff\_\* properties =\> zerodivisionerror (so it's an issue with
sniff properties?)
- one node, no sniff\_\* properties =\> no problems
- two node, no sniff\_\* properties =\> timeout connecting to ES.
I'm understanding that round-robin is used on each request, right? So I
would end up sending one bulk action to node1 and the second would go to
node2?
Thanks
On Tuesday, 29 October 2013 16:01:17 UTC, Honza Král wrote:
> On Tue, Oct 29, 2013 at 4:02 PM, Mauro Farracha \<[farr...@gmail.com](mailto:farr...@gmail.com)\
>
> > wrote:
>
> > Thanks Honza.
> >
> > I was importing the class. But not the right way 🙂 missing the
> > connection part in from.
>
> I thought it was the case, I update the code so that it should work in the
> future.
>
> > I changed the serializer (used ujson as Jorg mentioned) and I got an
> > improvement from 2.66MB/s to 4.7MB/s.
>
> ah, good to know, I will give it a try. I wanted to avoid additional
> dependencies, but if it makes sense I will happily switch the client to
> ujson. have you also tried just passing in a big string?
>
> > Then I configured ThriftConnection and the write performance increased to
> > 6.2MB/s.
> >
> > Not bad, but still far off from the 12MB/s from curl.
>
> we still have to deserialize the response which curl doesn't need to do so
> it will always have an advantage on us I am afraid, it shouldn't be this
> big though.
>
> > Have two questions:
> >
> > - using elasticsearch-py the index mapping is not the same on the server
> > side as when using pyes. Am I missing something? With pyes all the
> > properties were there, but using elasticsearch-py, only type appears on the
> > server side and are not the ones I specified. On the server log, It shows
> > "update\_mapping [accesslogs] (dynamic)" which doesn't happen with pyes. I'm
> > sure I'm missing some property/config.
>
> body should also include the doc\_type, so:
> self.client.indices.put\_mapping(index=self.doc\_collection,
> doc\_type=self.doc\_type,body={self.\_doc\_type:
> {'properties':self.doc\_mapping}})
>
> > - Also, can you guys share what's your performance on a single local node?
>
> I haven't done any tests like this, it varies so much with different
> HW/configuration/environment that there is little value in absolute
> numbers, only thing that matters is the relative speed of python clients,
> curl etc.
>
> > As I mention on my first post, these are my non-default configurations,
> > maybe there is still room for improvement? Not to mention of course, that
> > these same settings were responsible for the 12MB/s on curl.
> >
> > indices.memory.index\_buffer\_\*\*size: 50%
> > indices.memory.min\_index\_\*\*buffer\_size: 300mb
> > index.translog.flush\_\*\*threshold: 30000
> > index.store.type: mmapfs
> > index.merge.policy.use\_\*\*compound\_file: false
>
> these look reasonable though I am no expert. Also when using SSDs you
> might benefit from switching the kernel IO scheduler to noop:
> [https://speakerdeck.com/elasticsearch/life-after-ec2](https://speakerdeck.com/elasticsearch/life-after-ec2)
>
> > On Tuesday, 29 October 2013 14:32:12 UTC, Honza Král wrote:
> >
> > > you need to import it before you intend to use it:
> > >
> > > from elasticsearch.connection import ThriftConnection
> > >
> > > On Tue, Oct 29, 2013 at 3:02 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > >
> > > > Ahhhh you are the source! 🙂
> > > >
> > > > As I mentioned on the first post, I wrote a python script using
> > > > elasticsearch-py also and the performance was equals to pyes, but I
> > > > couldn't get it working with Thrift. The documentation available for me was
> > > > not detailed enough so I could understand how to fully use all the features
> > > > and was a little bit confusing the Connection/Transport classes.
> > > >
> > > > Maybe you could help me out... the error was:
> > > > self.client = Elasticsearch(hosts=self.\*\*elasticsearch\_conn,connection\_
> > > > \*\*class=ThriftConnection)
> > > > NameError: global name 'ThriftConnection' is not defined
> > > >
> > > > I have ES thrift plugin installed (works on pyes), I have the thrift
> > > > python module installed and I import the class. Don't know what I'm missing.
> > > >
> > > > On Tuesday, 29 October 2013 13:16:47 UTC, Honza Král wrote:
> > > >
> > > > > I am not familiar with pyes, I did however write elasticsearch-py and
> > > > > made sure you can bypass the serialization by doing it yourself. If needed
> > > > > you can even supply your own serializer - just create an instance that has
> > > > > .dumps() and loads() methods and behaves the same as
> > > > > elasticsearch.serializer. **JSONS** erializer. you can then pass it to
> > > > > the Elasticsearch class as an argument (serializer=my\_faster\_\*\*
> > > > > serializ\*\*er)
> > > > >
> > > > > On Tue, Oct 29, 2013 at 2:07 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > > > >
> > > > > > Hi Honza,
> > > > > >
> > > > > > Ok, that could be a problem. I'm passing a python dictionary to pyes
> > > > > > driver. If I send a "string" json format I could pass the serialization?
> > > > > > Are you familiar with pyes driver?
> > > > > >
> > > > > > I saw this method signature, but don't know what's the "header", and
> > > > > > the document can it be one full string with several documents?
> > > > > >
> > > > > > index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
> > > > > >
> > > > > > Function helper for fast inserting
> > > > > > Parameters:
> > > > > >
> > > > > > - _header_ – a string with the bulk header must be ended with a
> > > > > > newline
> > > > > > - _header_ – a json document string must be ended with a newline
> > > > > >
> > > > > > On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > and what was the bottle neck? Has the pyhton process maxed out the
> > > > > > > CPU or was it waiting for network? You can try serializing the documents
> > > > > > > yourself and passing json strings to the client's bulk() method to make
> > > > > > > sure that's not the bottle neck (you can pass in list of strings or just
> > > > > > > one big string and we will just pass it along).
> > > > > > >
> > > > > > > The python client does more than curl - it serializes data and
> > > > > > > parses output, that's at least 2 cpu intensive operations that need to
> > > > > > > happen. One of them you can eliminate.
> > > > > > >
> > > > > > > On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \<
> > > > > > > [joerg...@gmail.com](mailto:joerg...@gmail.com)\> wrote:
> > > > > > >
> > > > > > > > Also to mention, the number of shards and replica, which affect a
> > > > > > > > lot the indexing performance.
> > > > > > > >
> > > > > > > > Jörg
> > > > > > > >
> > > > > > > > --
> > > > > > > > You received this message because you are subscribed to the Google
> > > > > > > > Groups "elasticsearch" group.
> > > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > > send an email to elasticsearc...@\*\*googlegroups. **c** om.
> > > > > > > >
> > > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)\*\*\*\*
> > > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > > > > Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com.
> > > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > > > .
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > .
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 4:29pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/18 "2013-10-29T16:29:03Z")
It's not related to Thrift, using http also shares this behaviour.
On Tuesday, 29 October 2013 16:27:32 UTC, Mauro Farracha wrote:
> Hi Honza,
>
> Yep, I understand the issue around dependencies. The minimum you can do is
> probably add this sort of information in documentation.
>
> Regarding the mapping issue, you were right, adding the index type solved
> the problem.
>
> Since elasticsearch-py uses connection pooling with round-robin by
> default, I was wondering if I could get more improvement if I had two nodes
> up, since I would distribute the load between two servers, but using
> ThriftConnection it throws an error which I don't understand why It happens
> since Im pretty sure that Im passing the right configuration:
>
> connection\_pool.py", line 60, in select
> self.rr %= len(connections)
> ZeroDivisionError: integer division or modulo by zero
>
> Scenarios:
>
> - two node, sniff\_\* properties =\> zerodivisionerror
> - one node, sniff\_\* properties =\> zerodivisionerror (so it's an issue with
> sniff properties?)
> - one node, no sniff\_\* properties =\> no problems
> - two node, no sniff\_\* properties =\> timeout connecting to ES.
>
> I'm understanding that round-robin is used on each request, right? So I
> would end up sending one bulk action to node1 and the second would go to
> node2?
>
> Thanks
>
> On Tuesday, 29 October 2013 16:01:17 UTC, Honza Král wrote:
>
> > On Tue, Oct 29, 2013 at 4:02 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> >
> > > Thanks Honza.
> > >
> > > I was importing the class. But not the right way 🙂 missing the
> > > connection part in from.
> >
> > I thought it was the case, I update the code so that it should work in
> > the future.
> >
> > > I changed the serializer (used ujson as Jorg mentioned) and I got an
> > > improvement from 2.66MB/s to 4.7MB/s.
> >
> > ah, good to know, I will give it a try. I wanted to avoid additional
> > dependencies, but if it makes sense I will happily switch the client to
> > ujson. have you also tried just passing in a big string?
> >
> > > Then I configured ThriftConnection and the write performance increased
> > > to 6.2MB/s.
> > >
> > > Not bad, but still far off from the 12MB/s from curl.
> >
> > we still have to deserialize the response which curl doesn't need to do
> > so it will always have an advantage on us I am afraid, it shouldn't be this
> > big though.
> >
> > > Have two questions:
> > >
> > > - using elasticsearch-py the index mapping is not the same on the server
> > > side as when using pyes. Am I missing something? With pyes all the
> > > properties were there, but using elasticsearch-py, only type appears on the
> > > server side and are not the ones I specified. On the server log, It shows
> > > "update\_mapping [accesslogs] (dynamic)" which doesn't happen with pyes. I'm
> > > sure I'm missing some property/config.
> >
> > body should also include the doc\_type, so:
> > self.client.indices.put\_mapping(index=self.doc\_collection,
> > doc\_type=self.doc\_type,body={self.\_doc\_type:
> > {'properties':self.doc\_mapping}})
> >
> > > - Also, can you guys share what's your performance on a single local
> > > node?
> >
> > I haven't done any tests like this, it varies so much with different
> > HW/configuration/environment that there is little value in absolute
> > numbers, only thing that matters is the relative speed of python clients,
> > curl etc.
> >
> > > As I mention on my first post, these are my non-default configurations,
> > > maybe there is still room for improvement? Not to mention of course, that
> > > these same settings were responsible for the 12MB/s on curl.
> > >
> > > indices.memory.index\_buffer\_\*\*size: 50%
> > > indices.memory.min\_index\_\*\*buffer\_size: 300mb
> > > index.translog.flush\_\*\*threshold: 30000
> > > index.store.type: mmapfs
> > > index.merge.policy.use\_\*\*compound\_file: false
> >
> > these look reasonable though I am no expert. Also when using SSDs you
> > might benefit from switching the kernel IO scheduler to noop:
> > [https://speakerdeck.com/elasticsearch/life-after-ec2](https://speakerdeck.com/elasticsearch/life-after-ec2)
> >
> > > On Tuesday, 29 October 2013 14:32:12 UTC, Honza Král wrote:
> > >
> > > > you need to import it before you intend to use it:
> > > >
> > > > from elasticsearch.connection import ThriftConnection
> > > >
> > > > On Tue, Oct 29, 2013 at 3:02 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > > >
> > > > > Ahhhh you are the source! 🙂
> > > > >
> > > > > As I mentioned on the first post, I wrote a python script using
> > > > > elasticsearch-py also and the performance was equals to pyes, but I
> > > > > couldn't get it working with Thrift. The documentation available for me was
> > > > > not detailed enough so I could understand how to fully use all the features
> > > > > and was a little bit confusing the Connection/Transport classes.
> > > > >
> > > > > Maybe you could help me out... the error was:
> > > > > self.client = Elasticsearch(hosts=self.\*\*
> > > > > elasticsearch\_conn,connection\_\*\*class=ThriftConnection)
> > > > > NameError: global name 'ThriftConnection' is not defined
> > > > >
> > > > > I have ES thrift plugin installed (works on pyes), I have the thrift
> > > > > python module installed and I import the class. Don't know what I'm missing.
> > > > >
> > > > > On Tuesday, 29 October 2013 13:16:47 UTC, Honza Král wrote:
> > > > >
> > > > > > I am not familiar with pyes, I did however write elasticsearch-py and
> > > > > > made sure you can bypass the serialization by doing it yourself. If needed
> > > > > > you can even supply your own serializer - just create an instance that has
> > > > > > .dumps() and loads() methods and behaves the same as
> > > > > > elasticsearch.serializer. **JSONS** erializer. you can then pass it to
> > > > > > the Elasticsearch class as an argument (serializer=my\_faster\_\*\*
> > > > > > serializ\*\*er)
> > > > > >
> > > > > > On Tue, Oct 29, 2013 at 2:07 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > > > > >
> > > > > > > Hi Honza,
> > > > > > >
> > > > > > > Ok, that could be a problem. I'm passing a python dictionary to
> > > > > > > pyes driver. If I send a "string" json format I could pass the
> > > > > > > serialization? Are you familiar with pyes driver?
> > > > > > >
> > > > > > > I saw this method signature, but don't know what's the "header", and
> > > > > > > the document can it be one full string with several documents?
> > > > > > >
> > > > > > > index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
> > > > > > >
> > > > > > > Function helper for fast inserting
> > > > > > > Parameters:
> > > > > > >
> > > > > > > - _header_ – a string with the bulk header must be ended with a
> > > > > > > newline
> > > > > > > - _header_ – a json document string must be ended with a newline
> > > > > > >
> > > > > > > On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > and what was the bottle neck? Has the pyhton process maxed out the
> > > > > > > > CPU or was it waiting for network? You can try serializing the documents
> > > > > > > > yourself and passing json strings to the client's bulk() method to make
> > > > > > > > sure that's not the bottle neck (you can pass in list of strings or just
> > > > > > > > one big string and we will just pass it along).
> > > > > > > >
> > > > > > > > The python client does more than curl - it serializes data and
> > > > > > > > parses output, that's at least 2 cpu intensive operations that need to
> > > > > > > > happen. One of them you can eliminate.
> > > > > > > >
> > > > > > > > On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \<
> > > > > > > > [joerg...@gmail.com](mailto:joerg...@gmail.com)\> wrote:
> > > > > > > >
> > > > > > > > > Also to mention, the number of shards and replica, which affect a
> > > > > > > > > lot the indexing performance.
> > > > > > > > >
> > > > > > > > > Jörg
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > You received this message because you are subscribed to the Google
> > > > > > > > > Groups "elasticsearch" group.
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > > > send an email to elasticsearc...@\*\*googlegroups. **c** om.
> > > > > > > > >
> > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)\*\*\*\*
> > > > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > > > > >
> > > > > > > > --
> > > > > > > > You received this message because you are subscribed to the Google
> > > > > > > > Groups "elasticsearch" group.
> > > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com.
> > > > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > > > > .
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > > .
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Mauro\_Farracha](https://discuss.elastic.co/u/Mauro_Farracha)
#### Post date: [October 29, 2013, 4:39pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/19 "2013-10-29T16:39:57Z")
Regarding the IO Scheduler, I'm using a macos system. I don't think it
applies here. Although, great presentation on recovery shard performance!
To be in consideration in the future.
On Tuesday, 29 October 2013 16:29:03 UTC, Mauro Farracha wrote:
> It's not related to Thrift, using http also shares this behaviour.
>
> On Tuesday, 29 October 2013 16:27:32 UTC, Mauro Farracha wrote:
>
> > Hi Honza,
> >
> > Yep, I understand the issue around dependencies. The minimum you can do
> > is probably add this sort of information in documentation.
> >
> > Regarding the mapping issue, you were right, adding the index type solved
> > the problem.
> >
> > Since elasticsearch-py uses connection pooling with round-robin by
> > default, I was wondering if I could get more improvement if I had two nodes
> > up, since I would distribute the load between two servers, but using
> > ThriftConnection it throws an error which I don't understand why It happens
> > since Im pretty sure that Im passing the right configuration:
> >
> > connection\_pool.py", line 60, in select
> > self.rr %= len(connections)
> > ZeroDivisionError: integer division or modulo by zero
> >
> > Scenarios:
> >
> > - two node, sniff\_\* properties =\> zerodivisionerror
> > - one node, sniff\_\* properties =\> zerodivisionerror (so it's an issue
> > with sniff properties?)
> > - one node, no sniff\_\* properties =\> no problems
> > - two node, no sniff\_\* properties =\> timeout connecting to ES.
> >
> > I'm understanding that round-robin is used on each request, right? So I
> > would end up sending one bulk action to node1 and the second would go to
> > node2?
> >
> > Thanks
> >
> > On Tuesday, 29 October 2013 16:01:17 UTC, Honza Král wrote:
> >
> > > On Tue, Oct 29, 2013 at 4:02 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > >
> > > > Thanks Honza.
> > > >
> > > > I was importing the class. But not the right way 🙂 missing the
> > > > connection part in from.
> > >
> > > I thought it was the case, I update the code so that it should work in
> > > the future.
> > >
> > > > I changed the serializer (used ujson as Jorg mentioned) and I got an
> > > > improvement from 2.66MB/s to 4.7MB/s.
> > >
> > > ah, good to know, I will give it a try. I wanted to avoid additional
> > > dependencies, but if it makes sense I will happily switch the client to
> > > ujson. have you also tried just passing in a big string?
> > >
> > > > Then I configured ThriftConnection and the write performance increased
> > > > to 6.2MB/s.
> > > >
> > > > Not bad, but still far off from the 12MB/s from curl.
> > >
> > > we still have to deserialize the response which curl doesn't need to do
> > > so it will always have an advantage on us I am afraid, it shouldn't be this
> > > big though.
> > >
> > > > Have two questions:
> > > >
> > > > - using elasticsearch-py the index mapping is not the same on the
> > > > server side as when using pyes. Am I missing something? With pyes all the
> > > > properties were there, but using elasticsearch-py, only type appears on the
> > > > server side and are not the ones I specified. On the server log, It shows
> > > > "update\_mapping [accesslogs] (dynamic)" which doesn't happen with pyes. I'm
> > > > sure I'm missing some property/config.
> > >
> > > body should also include the doc\_type, so:
> > > self.client.indices.put\_mapping(index=self.doc\_collection,
> > > doc\_type=self.doc\_type,body={self.\_doc\_type:
> > > {'properties':self.doc\_mapping}})
> > >
> > > > - Also, can you guys share what's your performance on a single local
> > > > node?
> > >
> > > I haven't done any tests like this, it varies so much with different
> > > HW/configuration/environment that there is little value in absolute
> > > numbers, only thing that matters is the relative speed of python clients,
> > > curl etc.
> > >
> > > > As I mention on my first post, these are my non-default configurations,
> > > > maybe there is still room for improvement? Not to mention of course, that
> > > > these same settings were responsible for the 12MB/s on curl.
> > > >
> > > > indices.memory.index\_buffer\_\*\*size: 50%
> > > > indices.memory.min\_index\_\*\*buffer\_size: 300mb
> > > > index.translog.flush\_\*\*threshold: 30000
> > > > index.store.type: mmapfs
> > > > index.merge.policy.use\_\*\*compound\_file: false
> > >
> > > these look reasonable though I am no expert. Also when using SSDs you
> > > might benefit from switching the kernel IO scheduler to noop:
> > > [https://speakerdeck.com/elasticsearch/life-after-ec2](https://speakerdeck.com/elasticsearch/life-after-ec2)
> > >
> > > > On Tuesday, 29 October 2013 14:32:12 UTC, Honza Král wrote:
> > > >
> > > > > you need to import it before you intend to use it:
> > > > >
> > > > > from elasticsearch.connection import ThriftConnection
> > > > >
> > > > > On Tue, Oct 29, 2013 at 3:02 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > > > >
> > > > > > Ahhhh you are the source! 🙂
> > > > > >
> > > > > > As I mentioned on the first post, I wrote a python script using
> > > > > > elasticsearch-py also and the performance was equals to pyes, but I
> > > > > > couldn't get it working with Thrift. The documentation available for me was
> > > > > > not detailed enough so I could understand how to fully use all the features
> > > > > > and was a little bit confusing the Connection/Transport classes.
> > > > > >
> > > > > > Maybe you could help me out... the error was:
> > > > > > self.client = Elasticsearch(hosts=self.\*\*
> > > > > > elasticsearch\_conn,connection\_\*\*class=ThriftConnection)
> > > > > > NameError: global name 'ThriftConnection' is not defined
> > > > > >
> > > > > > I have ES thrift plugin installed (works on pyes), I have the thrift
> > > > > > python module installed and I import the class. Don't know what I'm missing.
> > > > > >
> > > > > > On Tuesday, 29 October 2013 13:16:47 UTC, Honza Král wrote:
> > > > > >
> > > > > > > I am not familiar with pyes, I did however write elasticsearch-py
> > > > > > > and made sure you can bypass the serialization by doing it yourself. If
> > > > > > > needed you can even supply your own serializer - just create an instance
> > > > > > > that has .dumps() and loads() methods and behaves the same as
> > > > > > > elasticsearch.serializer. **JSONS** erializer. you can then pass it
> > > > > > > to the Elasticsearch class as an argument (serializer=my\_faster\_\*\*
> > > > > > > serializ\*\*er)
> > > > > > >
> > > > > > > On Tue, Oct 29, 2013 at 2:07 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > > > > > >
> > > > > > > > Hi Honza,
> > > > > > > >
> > > > > > > > Ok, that could be a problem. I'm passing a python dictionary to
> > > > > > > > pyes driver. If I send a "string" json format I could pass the
> > > > > > > > serialization? Are you familiar with pyes driver?
> > > > > > > >
> > > > > > > > I saw this method signature, but don't know what's the "header",
> > > > > > > > and the document can it be one full string with several documents?
> > > > > > > >
> > > > > > > > index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
> > > > > > > >
> > > > > > > > Function helper for fast inserting
> > > > > > > > Parameters:
> > > > > > > >
> > > > > > > > - _header_ – a string with the bulk header must be ended with a
> > > > > > > > newline
> > > > > > > > - _header_ – a json document string must be ended with a newline
> > > > > > > >
> > > > > > > > On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > and what was the bottle neck? Has the pyhton process maxed out the
> > > > > > > > > CPU or was it waiting for network? You can try serializing the documents
> > > > > > > > > yourself and passing json strings to the client's bulk() method to make
> > > > > > > > > sure that's not the bottle neck (you can pass in list of strings or just
> > > > > > > > > one big string and we will just pass it along).
> > > > > > > > >
> > > > > > > > > The python client does more than curl - it serializes data and
> > > > > > > > > parses output, that's at least 2 cpu intensive operations that need to
> > > > > > > > > happen. One of them you can eliminate.
> > > > > > > > >
> > > > > > > > > On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \<
> > > > > > > > > [joerg...@gmail.com](mailto:joerg...@gmail.com)\> wrote:
> > > > > > > > >
> > > > > > > > > > Also to mention, the number of shards and replica, which affect a
> > > > > > > > > > lot the indexing performance.
> > > > > > > > > >
> > > > > > > > > > Jörg
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > You received this message because you are subscribed to the
> > > > > > > > > > Google Groups "elasticsearch" group.
> > > > > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > > > > send an email to elasticsearc...@\*\*googlegroups. **c** om.
> > > > > > > > > >
> > > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)\*\*\*\*
> > > > > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > You received this message because you are subscribed to the Google
> > > > > > > > > Groups "elasticsearch" group.
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > > > send an email to elasticsearc...@\*\*googlegroups.**com.
> > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)**
> > > > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > > > > Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > send an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > > > .
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)
#### Post date: [October 29, 2013, 4:46pm UTC](https://discuss.elastic.co/t/es-write-performance/14152/20 "2013-10-29T16:46:59Z")
Oops, there was a bug in the sniffing code, now fixed in master:
>
Can you please try again with master?
Thanks!
On Tue, Oct 29, 2013 at 5:29 PM, Mauro Farracha [farracha@gmail.com](mailto:farracha@gmail.com) wrote:
> It's not related to Thrift, using http also shares this behaviour.
>
> On Tuesday, 29 October 2013 16:27:32 UTC, Mauro Farracha wrote:
>
> > Hi Honza,
> >
> > Yep, I understand the issue around dependencies. The minimum you can do
> > is probably add this sort of information in documentation.
> >
> > Regarding the mapping issue, you were right, adding the index type solved
> > the problem.
> >
> > Since elasticsearch-py uses connection pooling with round-robin by
> > default, I was wondering if I could get more improvement if I had two nodes
> > up, since I would distribute the load between two servers, but using
> > ThriftConnection it throws an error which I don't understand why It happens
> > since Im pretty sure that Im passing the right configuration:
> >
> > connection\_pool.py", line 60, in select
> > self.rr %= len(connections)
> > ZeroDivisionError: integer division or modulo by zero
> >
> > Scenarios:
> >
> > - two node, sniff\_\* properties =\> zerodivisionerror
> > - one node, sniff\_\* properties =\> zerodivisionerror (so it's an issue
> > with sniff properties?)
> > - one node, no sniff\_\* properties =\> no problems
> > - two node, no sniff\_\* properties =\> timeout connecting to ES.
> >
> > I'm understanding that round-robin is used on each request, right? So I
> > would end up sending one bulk action to node1 and the second would go to
> > node2?
> >
> > Thanks
> >
> > On Tuesday, 29 October 2013 16:01:17 UTC, Honza Král wrote:
> >
> > > On Tue, Oct 29, 2013 at 4:02 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > >
> > > > Thanks Honza.
> > > >
> > > > I was importing the class. But not the right way 🙂 missing the
> > > > connection part in from.
> > >
> > > I thought it was the case, I update the code so that it should work in
> > > the future.
> > >
> > > > I changed the serializer (used ujson as Jorg mentioned) and I got an
> > > > improvement from 2.66MB/s to 4.7MB/s.
> > >
> > > ah, good to know, I will give it a try. I wanted to avoid additional
> > > dependencies, but if it makes sense I will happily switch the client to
> > > ujson. have you also tried just passing in a big string?
> > >
> > > > Then I configured ThriftConnection and the write performance increased
> > > > to 6.2MB/s.
> > > >
> > > > Not bad, but still far off from the 12MB/s from curl.
> > >
> > > we still have to deserialize the response which curl doesn't need to do
> > > so it will always have an advantage on us I am afraid, it shouldn't be this
> > > big though.
> > >
> > > > Have two questions:
> > > >
> > > > - using elasticsearch-py the index mapping is not the same on the
> > > > server side as when using pyes. Am I missing something? With pyes all the
> > > > properties were there, but using elasticsearch-py, only type appears on the
> > > > server side and are not the ones I specified. On the server log, It shows
> > > > "update\_mapping [accesslogs] (dynamic)" which doesn't happen with pyes. I'm
> > > > sure I'm missing some property/config.
> > >
> > > body should also include the doc\_type, so:
> > > self.client.indices.put\_\*\*mapping(index=self.doc\_\*\*collection,
> > > doc\_type=self.doc\_type,body={ **self._doc\_type: {'properties':self.doc_**
> > > mapping}})
> > >
> > > > - Also, can you guys share what's your performance on a single local
> > > > node?
> > >
> > > I haven't done any tests like this, it varies so much with different
> > > HW/configuration/environment that there is little value in absolute
> > > numbers, only thing that matters is the relative speed of python clients,
> > > curl etc.
> > >
> > > > As I mention on my first post, these are my non-default configurations,
> > > > maybe there is still room for improvement? Not to mention of course, that
> > > > these same settings were responsible for the 12MB/s on curl.
> > > >
> > > > indices.memory.index\_buffer\_ **si** ze: 50%
> > > > indices.memory.min\_index\_ **buffe** r\_size: 300mb
> > > > index.translog.flush\_ **threshold** : 30000
> > > > index.store.type: mmapfs
> > > > index.merge.policy.use\_ **compoun** d\_file: false
> > >
> > > these look reasonable though I am no expert. Also when using SSDs you
> > > might benefit from switching the kernel IO scheduler to noop:
> > > [https://speakerdeck.com/\*\*elasticsearch/life-after-ec2](https://speakerdeck.com/**elasticsearch/life-after-ec2)[https://speakerdeck.com/elasticsearch/life-after-ec2](https://speakerdeck.com/elasticsearch/life-after-ec2)
> > >
> > > > On Tuesday, 29 October 2013 14:32:12 UTC, Honza Král wrote:
> > > >
> > > > > you need to import it before you intend to use it:
> > > > >
> > > > > from elasticsearch.connection import ThriftConnection
> > > > >
> > > > > On Tue, Oct 29, 2013 at 3:02 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > > > >
> > > > > > Ahhhh you are the source! 🙂
> > > > > >
> > > > > > As I mentioned on the first post, I wrote a python script using
> > > > > > elasticsearch-py also and the performance was equals to pyes, but I
> > > > > > couldn't get it working with Thrift. The documentation available for me was
> > > > > > not detailed enough so I could understand how to fully use all the features
> > > > > > and was a little bit confusing the Connection/Transport classes.
> > > > > >
> > > > > > Maybe you could help me out... the error was:
> > > > > > self.client = Elasticsearch(hosts=self. **elast**
> > > > > > icsearch\_conn,connection\_ **class** =ThriftConnection)
> > > > > > NameError: global name 'ThriftConnection' is not defined
> > > > > >
> > > > > > I have ES thrift plugin installed (works on pyes), I have the thrift
> > > > > > python module installed and I import the class. Don't know what I'm missing.
> > > > > >
> > > > > > On Tuesday, 29 October 2013 13:16:47 UTC, Honza Král wrote:
> > > > > >
> > > > > > > I am not familiar with pyes, I did however write elasticsearch-py
> > > > > > > and made sure you can bypass the serialization by doing it yourself. If
> > > > > > > needed you can even supply your own serializer - just create an instance
> > > > > > > that has .dumps() and loads() methods and behaves the same as
> > > > > > > elasticsearch.serializer.**JSONS\*\*\*\*erializer. you can then pass it
> > > > > > > to the Elasticsearch class as an argument (serializer=my\_faster\_**
> > > > > > > serializ\*\*\*\*er)
> > > > > > >
> > > > > > > On Tue, Oct 29, 2013 at 2:07 PM, Mauro Farracha [farr...@gmail.com](mailto:farr...@gmail.com)wrote:
> > > > > > >
> > > > > > > > Hi Honza,
> > > > > > > >
> > > > > > > > Ok, that could be a problem. I'm passing a python dictionary to
> > > > > > > > pyes driver. If I send a "string" json format I could pass the
> > > > > > > > serialization? Are you familiar with pyes driver?
> > > > > > > >
> > > > > > > > I saw this method signature, but don't know what's the "header",
> > > > > > > > and the document can it be one full string with several documents?
> > > > > > > >
> > > > > > > > index\_raw\_bulk(_header_, _document_)[http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index\_raw\_bulk](http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ES.index_raw_bulk)
> > > > > > > >
> > > > > > > > Function helper for fast inserting
> > > > > > > > Parameters:
> > > > > > > >
> > > > > > > > - _header_ – a string with the bulk header must be ended with a
> > > > > > > > newline
> > > > > > > > - _header_ – a json document string must be ended with a newline
> > > > > > > >
> > > > > > > > On Tuesday, 29 October 2013 12:55:55 UTC, Honza Král wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > and what was the bottle neck? Has the pyhton process maxed out the
> > > > > > > > > CPU or was it waiting for network? You can try serializing the documents
> > > > > > > > > yourself and passing json strings to the client's bulk() method to make
> > > > > > > > > sure that's not the bottle neck (you can pass in list of strings or just
> > > > > > > > > one big string and we will just pass it along).
> > > > > > > > >
> > > > > > > > > The python client does more than curl - it serializes data and
> > > > > > > > > parses output, that's at least 2 cpu intensive operations that need to
> > > > > > > > > happen. One of them you can eliminate.
> > > > > > > > >
> > > > > > > > > On Tue, Oct 29, 2013 at 1:23 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \<
> > > > > > > > > [joerg...@gmail.com](mailto:joerg...@gmail.com)\> wrote:
> > > > > > > > >
> > > > > > > > > > Also to mention, the number of shards and replica, which affect a
> > > > > > > > > > lot the indexing performance.
> > > > > > > > > >
> > > > > > > > > > Jörg
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > You received this message because you are subscribed to the
> > > > > > > > > > Google Groups "elasticsearch" group.
> > > > > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*c\*\*\*\*om.
> > > > > > > > > >
> > > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)\*\*\*\*\*\*
> > > > > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > You received this message because you are subscribed to the Google
> > > > > > > > > Groups "elasticsearch" group.
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > > > send an email to elasticsearc...@**googlegroups.com.
> > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)**\*\*
> > > > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google
> > > > > > > Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com.
> > > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > > > .
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to elasticsearc...@googlegroups.\*\*com.
> > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > .
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
[Next page](https://discuss.elastic.co/t/es-write-performance/14152.md?page=2)