Hi,
I have a strange problem with ES. It's running on a cluster with 64 cores
etc, so I don't think the power of the hardware is the issue.
I want to index a lot of documents with elasticsearch-hadoop.
After some problems I now have everything into place and it seems to work
fine.
So I wrote a simple pig script which loads all the files (~500) and stores
them into an ES index.
However, after ~22h the job failed, because of connection problems between
the nodes.
But during that time, there wasn't any heavy usage of network bandwidth or
other ressources.
After that I tried to run the pig script only for one document so I know
what is indexed and what is missing.
After about 3 documents indexed well doing this, the jobs started to fail
again, due to network problems although there wasn't any significant load.
I observed that even after the indexing jobs stopped, there was stuff
happening with the index. The number of documents kept growing for quite
some time and the translog operations went up and down being mostly at
about half a million.
For me this looks like the index takes more time indexing than the pig
script takes for writing into the index and after some time somewhere a
buffer gets too full.
Is this possible? I would expect, that in this case elasticsearch-hadoop
should get throttled.
The only documentation about the translog is what I found here:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-translog.html
which I find a bit little. I still don't know what implications the number
of translog operations has.
On the linked page it says, I could increase the numbers when doing bulk
indexing but I don't understand how this would help.
Also what's TPS?
Best regards
Julius
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d17e1231-da99-4bc2-b019-806046ffd34e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.