Transport vs Node client for large bulk inserts?

Elastic Stack Elasticsearch
1

I am trying to determine which would be a better fit for a large bulk upload ( ~ 1 trillion items for a single index). I have tried with the http api, but its very slow and painful (it has taken a week and only inserted 112 billion items sofar). I imagine I would see a performance boost from using one of the native connectors. Which connector, Transport or Node, would give me the great performance and parallelism?

Appreciate the help.

2

This might be better in the ES category rather than the hadoop one, as it seems more general?

3

thanks. I'll try posting in the ES category.

4

You can move threads, just edit the topic and change the category :slight_smile: