How can use esrally import data to es cloud?

Elastic Stack Elasticsearch
1

how can use esrally import data to es cloud ?
can give a example.
the host connect internet with proxy
how can configure about it.

2 
2017-12-13 02:48:10,346 PID:22661 elasticsearch WARNING GET https://2588624b8d75420be2c1d6fb31b10fa7.ap-southeast-1.aws.found.io:9243/ 
[status:N/A request:42.529s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/usr/local/lib/python3.6/site-packages/esrally/client.py", line 47, in urlopen
    return self.pool.urlopen(method, url, body=body, retries=retries, headers=headers, **kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 333, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f8f5e9acef0>: Failed to establish a new
 connection: [Errno 101] Network is unreachable
2017-12-13 02:48:52,409 PID:22661 elasticsearch WARNING GET https://2588624b8d75420be2c1d6fb31b10fa7.ap-southeast-1.aws.found.io:9243/ 
[status:N/A request:42.061s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/usr/local/lib/python3.6/site-packages/esrally/client.py", line 47, in urlopen
    return self.pool.urlopen(method, url, body=body, retries=retries, headers=headers, **kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 333, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f8f5e9acf60>: Failed to establish a new
 connection: [Errno 101] Network is unreachable
2017-12-13 02:49:34,624 PID:22661 elasticsearch WARNING GET https://2588624b8d75420be2c1d6fb31b10fa7.ap-southeast-1.aws.found.io:9243/ 
[status:N/A request:42.214s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/usr/local/lib/python3.6/site-packages/esrally/client.py", line 47, in urlopen
    return self.pool.urlopen(method, url, body=body, retries=retries, headers=headers, **kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 333, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
3

Looking at the code for Rally, it appears that it uses its own connection_class with the elasticsearch-py client, so I don't think there is an existing way to do it.

@danielmitterdorfer does Rally currently have proxy support? If not I don't think it will be too hard to add since it just needs to be added as an option.

4

yes i don't find where to add the configure with proxy connect to internet.
thanks..

5

@zqc0512 there is an entire section in the docs about proxy configuration:

http://esrally.readthedocs.io/en/stable/configuration.html?highlight=proxy#proxy-configuration

6

about this ..

Note:
Rally will use this proxy server only for downloading benchmark-related data. It will not use this proxy for the actual benchmark.

it about down data ,but benchmark not use proxy..
@danielmitterdorfer

7

That's correct. As @dakrone has mentioned it is currently not possible to do it. I have raised to #382 in the Rally repo so we'll add support for that in a future release. Thanks for bringing that up.

8

thanks so much.
it about have a time ? when can output ?
i use this test the es cloud.
thanks..
@danielmitterdorfer

9

thanks so much.
it about have a time ? when can output ? i see it in Pre 1.0 and now is 0.8.1. that so bad.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.