The offline installation with esrally-dist-2.4.0 give incompatible error

keylane_sbaas · May 10, 2022, 2:45pm

I was installing esrally on RHEL7 machine. At the point of installing there was an problem, that not enough dependencies where installed at the time. That is not the problem for this topic, but the ERROR while installing esrally.

ERROR: botocore 1.13.50 has requirement urllib3<1.26,>=1.20; python_version >= "3.4", but you'll have urllib3 1.26.9 which is incompatible.

I used the latest version 2.4.0 of esrally, but the provided esrally-dist-2.4.0/bin/urllib3-1.26.9-py2.py3-none-any.whl is incompatible so it seems. Is this github bug worthy?

# python3 -m pip install esrally --no-index --find-links /data/scripts/elastic_rally/esrally -r /data/scripts/elastic_rally/esrally/requirements.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
Looking in links: /data/scripts/elastic_rally/esrally
Processing ./esrally/esrally-1.4.1-py3-none-any.whl
Requirement already satisfied: thespian==3.9.3 in /root/.local/lib/python3.8/site-packages (from esrally) (3.9.3)
Processing ./esrally/tabulate-0.8.5.tar.gz
Processing ./esrally/psutil-5.6.5.tar.gz
Requirement already satisfied: certifi in /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages (from esrally) (2021.10.8)
Processing ./esrally/elasticsearch-7.0.5-py2.py3-none-any.whl
Processing ./esrally/boto3-1.10.32-py2.py3-none-any.whl
Processing ./esrally/py-cpuinfo-3.2.0.tar.gz
Requirement already satisfied: Jinja2==2.10.3 in /root/.local/lib/python3.8/site-packages (from esrally) (2.10.3)
Requirement already satisfied: jsonschema==3.1.1 in /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages (from esrally) (3.1.1)
Requirement already satisfied: urllib3>=1.21.1 in /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages (from elasticsearch==7.0.5->esrally) (1.26.9)
Processing ./esrally/s3transfer-0.2.1-py2.py3-none-any.whl
Processing ./esrally/botocore-1.13.50-py2.py3-none-any.whl
Processing ./esrally/jmespath-0.10.0-py2.py3-none-any.whl
Requirement already satisfied: MarkupSafe>=0.23 in /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages (from Jinja2==2.10.3->esrally) (2.0.1)
Requirement already satisfied: six>=1.11.0 in /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages (from jsonschema==3.1.1->esrally) (1.16.0)
Requirement already satisfied: setuptools in /opt/rh/rh-python38/root/usr/lib/python3.8/site-packages (from jsonschema==3.1.1->esrally) (41.6.0)
Requirement already satisfied: importlib-metadata in /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages (from jsonschema==3.1.1->esrally) (4.11.3)
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages (from jsonschema==3.1.1->esrally) (0.18.1)
Requirement already satisfied: attrs>=17.4.0 in /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages (from jsonschema==3.1.1->esrally) (21.4.0)
Processing ./esrally/docutils-0.15.2-py3-none-any.whl
Processing ./esrally/python_dateutil-2.8.2-py2.py3-none-any.whl
Requirement already satisfied: zipp>=0.5 in /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages (from importlib-metadata->jsonschema==3.1.1->esrally) (3.7.0)
ERROR: botocore 1.13.50 has requirement urllib3<1.26,>=1.20; python_version >= "3.4", but you'll have urllib3 1.26.9 which is incompatible.
Installing collected packages: tabulate, psutil, elasticsearch, docutils, python-dateutil, jmespath, botocore, s3transfer, boto3, py-cpuinfo, esrally
  Found existing installation: tabulate 0.8.9
    Uninstalling tabulate-0.8.9:
      Successfully uninstalled tabulate-0.8.9
    Running setup.py install for tabulate ... done
  Found existing installation: psutil 5.8.0
    Uninstalling psutil-5.8.0:
      Successfully uninstalled psutil-5.8.0
    Running setup.py install for psutil ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/rh/rh-python38/root/usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wf8kdx35/psutil/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wf8kdx35/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-l830zfc6/install-record.txt --single-version-externally-managed --compile
         cwd: /tmp/pip-install-wf8kdx35/psutil/
    Complete output (41 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/psutil
    copying psutil/__init__.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_common.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_compat.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psaix.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psbsd.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_pslinux.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psosx.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psposix.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_pssunos.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_pswindows.py -> build/lib.linux-x86_64-3.8/psutil
    creating build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/__init__.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/__main__.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/runner.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_memory_leaks.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_process.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_system.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-3.8/psutil/tests
    running build_ext
    building 'psutil._psutil_linux' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/psutil
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python38/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python38/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=565 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/opt/rh/rh-python38/root/usr/include/python3.8 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.8/psutil/_psutil_common.o
    unable to execute 'gcc': No such file or directory
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
  Rolling back uninstall of psutil
  Moving to /opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/psutil-5.8.0.dist-info/
   from /opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/~sutil-5.8.0.dist-info
  Moving to /opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/psutil/
   from /opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/~sutil
ERROR: Command errored out with exit status 1: /opt/rh/rh-python38/root/usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wf8kdx35/psutil/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wf8kdx35/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-l830zfc6/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

michaelbaamonde · May 10, 2022, 8:23pm

Hi,

Thanks for your interest in Rally! I have a couple of questions that will hopefully get us on the right track.

First, did you attempt to run the install.sh script that's included in the offline installation package (available here) directly? The pip install command you've provided looks a bit different from what that script would execute. I'm specifically wondering about the requirements.txt file that you're passing via -r. Did you create this yourself? The output also looks off, with outdated package versions.

Second, you mention that there was an issue with dependencies at the time of installation. Can you tell me more about that? It may be relevant.

Overall, it'd help us diagnose the issue if you could run the install.sh script directly as-is and provide the output.

Thanks!

keylane_sbaas · May 20, 2022, 3:23pm

Hi Michael,

Yes i did try to run it with install.sh. Second I can use pip install on the offline system, so used other VM to gather all the needed files and ship it.
No, requirements.txt was created following the documentation. Not sure why the file was empty, but when with it.

Yes sure. Based on the output there was a problem. With googling around on dependencies for the packages, I came around to different pages that suggested different things. Install this package first or this package, but I understand that the installation did some compiling. So in the end I solved it by following theses steps:

Enable all the repos
yum groupinstall 'Development Tools' -y
yum install java-11-openjdk rh-python38 rh-python38-python-devel git19 -y
scl enable rh-python38 bash
scl enable git19 bash
./esrally-dist-2.4.0/install.sh

Is this worth adding to the documentation for the offline installation?
ps i worked on RHEL7 machine, maybe later testing it on RHEL8.

keylane_sbaas · May 20, 2022, 3:28pm

If it helps, there list of package the where install through the commands. The provide files for offline install didn't work so far I can see

Installing:
 python3                    x86_64         3.6.8-18.el7            rhel-7-server-rpms          70 k
Installing for dependencies:
 python3-libs               x86_64         3.6.8-18.el7            rhel-7-server-rpms         6.9 M
 python3-pip                noarch         9.0.3-8.el7             rhel-7-server-rpms         1.6 M
 python3-setuptools         noarch         39.2.0-10.el7           rhel-7-server-rpms         629 k
└──> yum install rh-python38 rh-python38-python-devel
Installing:
 rh-python38                                                      x86_64                              2.0-4.el7                                    rhel-server-rhscl-7-rpms                              3.3 k
Installing for dependencies:
 rh-python38-python                                               x86_64                              3.8.11-2.el7                                 rhel-server-rhscl-7-rpms                               72 k
 rh-python38-python-libs                                          x86_64                              3.8.11-2.el7                                 rhel-server-rhscl-7-rpms                              7.7 M
 rh-python38-python-pip                                           noarch                              19.3.1-2.el7                                 rhel-server-rhscl-7-rpms                              1.7 M
 rh-python38-python-pip-wheel                                     noarch                              19.3.1-2.el7                                 rhel-server-rhscl-7-rpms                              1.2 M
 rh-python38-python-setuptools                                    noarch                              41.6.0-7.el7                                 rhel-server-rhscl-7-rpms                              636 k
 rh-python38-python-setuptools-wheel                              noarch                              41.6.0-7.el7                                 rhel-server-rhscl-7-rpms                              300 k
 rh-python38-runtime                                              x86_64                              2.0-4.el7                                    rhel-server-rhscl-7-rpms                              1.1 M
 scl-utils                                                        x86_64                              20130529-19.el7                              rhel-7-server-rpms                                     24 k
└──> yum install rh-python38-python-devel 
 Installing:
 rh-python38-python-devel                                   x86_64                              3.8.11-2.el7                                    rhel-server-rhscl-7-rpms                                 276 k
Installing for dependencies:
 dwz                                                        x86_64                              0.11-3.el7                                      rhel-7-server-rpms                                        99 k
 iso-codes                                                  noarch                              3.46-2.el7                                      rhel-7-server-rpms                                       2.7 M
 perl-srpm-macros                                           noarch                              1-8.el7                                         rhel-7-server-rpms                                       4.7 k
 python-srpm-macros                                         noarch                              3-34.el7                                        rhel-7-server-rpms                                       8.8 k
 redhat-rpm-config                                          noarch                              9.1.0-88.el7                                    rhel-7-server-rpms                                        81 k
 rh-python38-python-rpm-macros                              noarch                              3.8.11-2.el7                                    rhel-server-rhscl-7-rpms                                  65 k
 scl-utils-build                                            x86_64                              20130529-19.el7                                 rhel-7-server-optional-rpms                               18 k
 xml-common                                                 noarch                              0.6.3-39.el7                                    rhel-7-server-rpms                                        26 k
└──> yum install gcc
Installing:
 gcc                                              x86_64                                   4.8.5-44.el7                                             rhel-7-server-rpms                                    16 M
Installing for dependencies:
 cpp                                              x86_64                                   4.8.5-44.el7                                             rhel-7-server-rpms                                   6.0 M
 glibc-devel                                      x86_64                                   2.17-325.el7_9                                           rhel-7-server-rpms                                   1.1 M
 glibc-headers                                    x86_64                                   2.17-325.el7_9                                           rhel-7-server-rpms                                   691 k
 kernel-headers                                   x86_64                                   3.10.0-1160.53.1.el7                                     rhel-7-server-rpms                                   9.0 M
 libmpc                                           x86_64                                   1.0.1-3.el7                                              rhel-7-server-rpms                                    51 k
 mpfr                                             x86_64                                   3.1.1-4.el7                                              rhel-7-server-rpms                                   203 k

system · June 17, 2022, 3:28pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.