OSS repo is not working

Hi team,

I wanted to report that OSS repo is not working properly, it has non oss packages instead of oss packages:
https://artifacts.elastic.co/packages/oss-7.x/yum

We need it, as we use the OSS packages through repo, not pulling down .rpm`s

[root@********]# cat /etc/yum.repos.d/elastic-oss.repo
[elastic-oss-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/oss-7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

[root@******]# yum repo-pkgs elastic-oss-7.x list
Loaded plugins: fastestmirror
Repository elasticsearch-7.x is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: centos-distro.cavecreek.net
 * epel: dl.fedoraproject.org
 * extras: mirror.phx1.us.spryservers.net
 * updates: mirror.san.fastserv.com
Available Packages
apm-server.i686                                                                                    7.6.2-1                                                                             elastic-oss-7.x
apm-server.x86_64                                                                                  7.6.2-1                                                                             elastic-oss-7.x
auditbeat.i686                                                                                     7.6.2-1                                                                             elastic-oss-7.x
auditbeat.x86_64                                                                                   7.6.2-1                                                                             elastic-oss-7.x
elasticsearch-oss.x86_64                                                                           7.6.2-1                                                                             elastic-oss-7.x
filebeat.i686                                                                                      7.6.2-1                                                                             elastic-oss-7.x
filebeat.x86_64                                                                                    7.6.2-1                                                                             elastic-oss-7.x
heartbeat-elastic.i686                                                                             7.6.2-1                                                                             elastic-oss-7.x
heartbeat-elastic.x86_64                                                                           7.6.2-1                                                                             elastic-oss-7.x
journalbeat.i686                                                                                   7.6.2-1                                                                             elastic-oss-7.x
journalbeat.x86_64                                                                                 7.6.2-1                                                                             elastic-oss-7.x
kibana-oss.x86_64                                                                                  7.6.2-1                                                                             elastic-oss-7.x
logstash.noarch                                                                                    1:7.1.1-1                                                                           elastic-oss-7.x
logstash-oss.noarch                                                                                1:7.6.2-1                                                                           elastic-oss-7.x
metricbeat.i686                                                                                    7.6.2-1                                                                             elastic-oss-7.x
metricbeat.x86_64                                                                                  7.6.2-1                                                                             elastic-oss-7.x
packetbeat.i686                                                                                    7.6.2-1                                                                             elastic-oss-7.x
packetbeat.x86_64                                                                                  7.6.2-1                                                                             elastic-oss-7.x

Seeing your output I saw this snippet

Can you make sure that there are no duplicates in your list, then update, then retry and we take it from there?

Ok, let me go this way. I put a repo on a fresh machine, downloaded repo file, with opendistro repo, elasticsearch 6.x oss repo, and elasticsearch 7.x oss repo, and here is output below:

[root@utils-ansible-01 yum.repos.d]# cat /etc/yum.repos.d/opendistroforelasticsearch-artifacts.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/oss-6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/oss-7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

[opendistroforelasticsearch-artifacts-repo]
name=Release RPM artifacts of OpenDistroForElasticsearch
baseurl=https://d3g5vo6xdbdb9a.cloudfront.net/yum/noarch/
enabled=1
gpgkey=https://d3g5vo6xdbdb9a.cloudfront.net/GPG-KEY-opendistroforelasticsearch
gpgcheck=1
repo_gpgcheck=1
autorefresh=1
type=rpm-md
[root@utils-ansible-01 yum.repos.d]# yum repo-pkgs elasticsearch-7.x list | grep filebeat
filebeat.i686                          7.6.2-1                 elasticsearch-7.x
filebeat.x86_64                        7.6.2-1                 elasticsearch-7.x
[root@utils-ansible-01 yum.repos.d]# yum repo-pkgs elasticsearch-6.x list | grep filebeat
filebeat.i686                            6.8.8-1               elasticsearch-6.x
filebeat.x86_64                          6.8.8-1               elasticsearch-6.x
filebeat-oss.i686                        6.3.2-1               elasticsearch-6.x
filebeat-oss.x86_64                      6.3.2-1               elasticsearch-6.x

As you can see from repo file definition, both 6.x and 7.x repo should be oss repo, but when you list packages you will see that oss filebeat exist in 6.x but not in 7.x repo. Below I will put list of both repo packages:

> [root@utils-ansible-01 yum.repos.d]# yum repo-pkgs elasticsearch-6.x list
> Loaded plugins: copr, fastestmirror
> Loading mirror speeds from cached hostfile
>  * base: repos.lax.quadranet.com
>  * epel: dl.fedoraproject.org
>  * extras: sjc.edge.kernel.org
>  * updates: mirror.keystealth.org
> Available Packages
> apm-server.i686                                                                                     6.8.8-1                                                                          elasticsearch-6.x
> apm-server.x86_64                                                                                   6.8.8-1                                                                          elasticsearch-6.x
> apm-server-oss.i686                                                                                 6.3.2-1                                                                          elasticsearch-6.x
> apm-server-oss.x86_64                                                                               6.3.2-1                                                                          elasticsearch-6.x
> auditbeat.i686                                                                                      6.8.8-1                                                                          elasticsearch-6.x
> auditbeat.x86_64                                                                                    6.8.8-1                                                                          elasticsearch-6.x
> auditbeat-oss.i686                                                                                  6.3.2-1                                                                          elasticsearch-6.x
> auditbeat-oss.x86_64                                                                                6.3.2-1                                                                          elasticsearch-6.x
> elasticsearch-oss.noarch                                                                            6.8.8-1                                                                          elasticsearch-6.x
> filebeat.i686                                                                                       6.8.8-1                                                                          elasticsearch-6.x
> filebeat.x86_64                                                                                     6.8.8-1                                                                          elasticsearch-6.x
> filebeat-oss.i686                                                                                   6.3.2-1                                                                          elasticsearch-6.x
> filebeat-oss.x86_64                                                                                 6.3.2-1                                                                          elasticsearch-6.x
> heartbeat-elastic.i686                                                                              6.8.8-1                                                                          elasticsearch-6.x
> heartbeat-elastic.x86_64                                                                            6.8.8-1                                                                          elasticsearch-6.x
> heartbeat-elastic-oss.i686                                                                          6.3.2-1                                                                          elasticsearch-6.x
> heartbeat-elastic-oss.x86_64                                                                        6.3.2-1                                                                          elasticsearch-6.x
> journalbeat.i686                                                                                    6.8.8-1                                                                          elasticsearch-6.x
> journalbeat.x86_64                                                                                  6.8.8-1                                                                          elasticsearch-6.x
> kibana-oss.x86_64                                                                                   6.8.8-1                                                                          elasticsearch-6.x
> logstash.noarch                                                                                     1:6.8.0-1                                                                        elasticsearch-6.x
> logstash-oss.noarch                                                                                 1:6.8.8-1                                                                        elasticsearch-6.x
> metricbeat.i686                                                                                     6.8.8-1                                                                          elasticsearch-6.x
> metricbeat.x86_64                                                                                   6.8.8-1                                                                          elasticsearch-6.x
> metricbeat-oss.i686                                                                                 6.3.2-1                                                                          elasticsearch-6.x
> metricbeat-oss.x86_64                                                                               6.3.2-1                                                                          elasticsearch-6.x
> packetbeat.i686                                                                                     6.8.8-1                                                                          elasticsearch-6.x
> packetbeat.x86_64                                                                                   6.8.8-1                                                                          elasticsearch-6.x
> packetbeat-oss.i686                                                                                 6.3.2-1                                                                          elasticsearch-6.x
> packetbeat-oss.x86_64                                                                               6.3.2-1                                                                          elasticsearch-6.x
> 
> 
> [root@utils-ansible-01 yum.repos.d]# yum repo-pkgs elasticsearch-7.x list
> Loaded plugins: copr, fastestmirror
> Loading mirror speeds from cached hostfile
>  * base: repos.lax.quadranet.com
>  * epel: dl.fedoraproject.org
>  * extras: sjc.edge.kernel.org
>  * updates: mirror.keystealth.org
> Available Packages
> apm-server.i686                                                                                   7.6.2-1                                                                            elasticsearch-7.x
> apm-server.x86_64                                                                                 7.6.2-1                                                                            elasticsearch-7.x
> auditbeat.i686                                                                                    7.6.2-1                                                                            elasticsearch-7.x
> auditbeat.x86_64                                                                                  7.6.2-1                                                                            elasticsearch-7.x
> elasticsearch-oss.x86_64                                                                          7.6.2-1                                                                            elasticsearch-7.x
> filebeat.i686                                                                                     7.6.2-1                                                                            elasticsearch-7.x
> filebeat.x86_64                                                                                   7.6.2-1                                                                            elasticsearch-7.x
> heartbeat-elastic.i686                                                                            7.6.2-1                                                                            elasticsearch-7.x
> heartbeat-elastic.x86_64                                                                          7.6.2-1                                                                            elasticsearch-7.x
> journalbeat.i686                                                                                  7.6.2-1                                                                            elasticsearch-7.x
> journalbeat.x86_64                                                                                7.6.2-1                                                                            elasticsearch-7.x
> kibana-oss.x86_64                                                                                 7.6.2-1                                                                            elasticsearch-7.x
> logstash.noarch                                                                                   1:7.1.1-1                                                                          elasticsearch-7.x
> logstash-oss.noarch                                                                               1:7.6.2-1                                                                          elasticsearch-7.x
> metricbeat.i686                                                                                   7.6.2-1                                                                            elasticsearch-7.x
> metricbeat.x86_64                                                                                 7.6.2-1                                                                            elasticsearch-7.x
> packetbeat.i686                                                                                   7.6.2-1                                                                            elasticsearch-7.x
> packetbeat.x86_64                                                                                 7.6.2-1                                                                            elasticsearch-7.x
1 Like

Hi @jovan,

Thank you for reporting this issue.

It seems the filebeat package is oss 7.x repo is the good one even it doesn't have -oss suffix:

[root@9b5e2813860c /]# cat /etc/yum.repos.d/elastic.repo
[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/oss-7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
[root@9b5e2813860c /]# yum info filebeat
Available Packages
Name        : filebeat
Arch        : i686
Version     : 7.6.2
Release     : 1
Size        : 18 M
Repo        : elasticsearch-7.x
Summary     : Filebeat sends log files to Logstash or directly to Elasticsearch.
URL         : https://www.elastic.co/products/beats/filebeat
License     : ASL-2.0
Description : Filebeat sends log files to Logstash or directly to Elasticsearch.

Name        : filebeat
Arch        : x86_64
Version     : 7.6.2
Release     : 1
Size        : 18 M
Repo        : elasticsearch-7.x
Summary     : Filebeat sends log files to Logstash or directly to Elasticsearch.
URL         : https://www.elastic.co/products/beats/filebeat
License     : ASL-2.0
Description : Filebeat sends log files to Logstash or directly to Elasticsearch.

The important things to check are the Version which should be 7.6.2 (it seems we have some old version packages hidden in the repo) and License which should be ASL-2.0 (Elastic License on non OSS versions).

Meanwhile we are investigating on it and we'll fix it very soon.

1 Like

So after some investigations, I can confirm you that Elasticsearch, Kibana and Logstash OSS packages have the -oss extensions while the other OSS packages (APM Server & Beats) don't have this extension anymore since 6.3.3.

That's why when you list the OSS 6.x repository packages with yum repo-pkgs elasticsearch-6.x list, you can see that apm-server and beats packages with -oss extension are 6.3.2 version.

I know that's pretty confusing but we prefer not removing them from existing repositories to avoid breaking installation of people still using these packages.

So for Filebeat OSS, you should use filebeat packages (without -oss) and in case of doubt use yum info to verify package license and version.

1 Like

Thanks for help!
I have used yumdownloader to download filebeat from repo, and wget to pull the filebeat-oss Elasticsearch official site download, compared md5sum, and they are same:

[root@utils-ansible-01 tmp]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.4.2-x86_64.rpm 
[root@utils-ansible-01 tmp]# mv filebeat-oss-7.4.2-x86_64.rpm oss.rpm
[root@utils-ansible-01 tmp]# yumdownloader filebeat-7.4.2
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos-distro.1gservers.com
 * epel: dl.fedoraproject.org
 * extras: mirrors.xtom.com
 * updates: mirror.phx1.us.spryservers.net
warning: /tmp/filebeat-oss-7.4.2-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY===============================================                  ]  14 MB/s |  14 MB  00:00:00 ETA
Public key for filebeat-oss-7.4.2-x86_64.rpm is not installed
filebeat-oss-7.4.2-x86_64.rpm                                                                                                                                                  |  19 MB  00:00:00
[root@utils-ansible-01 tmp]#
[root@utils-ansible-01 tmp]# ll
total 57504
-rw-r--r-- 1 root root 18961573 Mar 31 17:29 filebeat-oss-7.4.2-i686.rpm
-rw-r--r-- 1 root root 19931666 Mar 31 17:29 filebeat-oss-7.4.2-x86_64.rpm
-rw-r--r-- 1 root root 19931666 Mar 31 17:29 oss.rpm
[root@utils-ansible-01 tmp]# md5sum oss.rpm filebeat-oss-7.4.2-x86_64.rpm
6d7cd7b8b2feb631bee06ee614624d7b  oss.rpm
6d7cd7b8b2feb631bee06ee614624d7b  filebeat-oss-7.4.2-x86_64.rpm

Julien, just one more question, in the future, will naming convention stay the same that on OSS repo ELK packages will have -oss extension, and other packages like (APM Server & Beats) will not have -ossextension?

We may try to have same convention (-oss for all or none) at some point but we shouldn't do it before 8.x to not add change in existing 6.x and 7.x repositories.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.