Journalbeat can't create an alias and fails to work

Journalbeat is having issues getting started. It seems like it fails to create an alias?

journalctl -l --follow -u journalbeat eventually spits this out:

Oct 01 15:33:43 hostname journalbeat[30589]: 2019-10-01T15:33:43.068-0700        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): Connection marked as failed because the onConnect callback failed: failed to create alias: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Oct 01 15:33:43 hostname journalbeat[30589]: <html><head>
Oct 01 15:33:43 hostname journalbeat[30589]: <title>404 Not Found</title>
Oct 01 15:33:43 hostname journalbeat[30589]: </head><body>
Oct 01 15:33:43 hostname journalbeat[30589]: <h1>Not Found</h1>
Oct 01 15:33:43 hostname journalbeat[30589]: <p>The requested URL was not found on this server.</p>
Oct 01 15:33:43 hostname journalbeat[30589]: </body></html>
Oct 01 15:33:43 hostname journalbeat[30589]: : 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Oct 01 15:33:43 hostname journalbeat[30589]: <html><head>
Oct 01 15:33:43 hostname journalbeat[30589]: <title>404 Not Found</title>
Oct 01 15:33:43 hostname journalbeat[30589]: </head><body>
Oct 01 15:33:43 hostname journalbeat[30589]: <h1>Not Found</h1>
Oct 01 15:33:43 hostname journalbeat[30589]: <p>The requested URL was not found on this server.</p>
Oct 01 15:33:43 hostname journalbeat[30589]: </body></html>
Oct 01 15:33:43 hostname journalbeat[30589]: 2019-10-01T15:33:43.068-0700        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(elasticsearch(http://elasticsearch:9200)) with 3 reconnect attempt(s)

If I try to run the setup tasks manually:

root@hostname:/etc/journalbeat# journalbeat -c /etc/journalbeat/journalbeat.yml setup
Exiting: failed to create alias: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
: 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

Testing output and config works just fine.

root@hostname:/etc/journalbeat# journalbeat -c /etc/journalbeat/journalbeat.yml test output
elasticsearch: http://elasticsearch:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.7.0.57
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.4.0
root@hostname:/etc/journalbeat# journalbeat -c /etc/journalbeat/journalbeat.yml test config
Config OK

Annoyingly I have both metricbeat and filebeat running on the same host, with the same elasticsearch connection details, and they work just fine.

I've been running into this on 7.3.1, 7.3.2, and now 7.4.0....

Here is my journalbeat config export:

journalbeat:
  inputs:
  - paths: null
    seek: cursor
output:
  elasticsearch:
    hosts:
    - elasticsearch:9200
    password: password
    protocol: http
    username: username
path:
  config: /etc/journalbeat
  data: /var/lib/journalbeat
  home: /usr/share/journalbeat
  logs: /var/log/journalbeat
processors:
- add_host_metadata: null
- add_cloud_metadata: null
setup:
  dashboards:
    enabled: true
  kibana:
    host: localhost:5601/kibana
  template:
    settings:
      index:
        number_of_shards: 1

I'm running kibana and elasticsearch via docker-compose on a test desktop in my office. Whereever you see hostname in my config, that represents the dns address of that desktop.

Here is my docker-compose file:

version: '3.3'
services:
  esnode1:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
    container_name: esnode1
    environment:
      - cluster.name=hostname-cluster
      - node.name=esnode1
      - discovery.seed_hosts=esnode2
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - r3_cluster_esdata1:/usr/share/elasticsearch/data
#      - ./config/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - 127.0.0.1:9201:9200
    healthcheck:
      test: ["CMD", "curl","-s" ,"-f", "http://localhost:9200/_cat/health"]
    networks:
      - elknet
    restart: always
  esnode2:
    container_name: esnode2
    image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
    environment:
      - cluster.name=hostname-cluster
      - node.name=esnode2
      - discovery.seed_hosts=esnode1
      - cluster.initial_master_nodes=esnode1,esnode2
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - r3_cluster_esdata2:/usr/share/elasticsearch/data
#      - ./config/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
      test: ["CMD", "curl","-s" ,"-f", "http://localhost:9200/_cat/health"]
    ports:
      - 127.0.0.1:9202:9200
    networks:
      - elknet
    restart: always
  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.4.0
    environment:
            #SERVER_NAME: 127.0.0.1
      ELASTICSEARCH_HOSTS: http://esnode1:9200
      SERVER_HOST: 0.0.0.0
      SERVER_PORT: 5601
      SERVER_BASEPATH: /kibana
      SERVER_REWRITEBASEPATH: "true"
      
    ports:
      - 127.0.0.1:5601:5601
    networks:
      - elknet
    restart: always
    depends_on:
      - esnode1 
  

volumes:
  r3_cluster_esdata1:
    driver: local
  r3_cluster_esdata2:
    driver: local

networks:
  elknet:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: elknet

I do have Apache set up as a reverse proxy so that I can use http auth.

I do have remote machines successfully pushing filebeat and metricbeat data.

Any suggestions?

Ok, I just ran into the exact same issue when trying to set up heartbeat. When I run setup, and tail apache access logs, I see:

ip - r3elastic [02/Oct/2019:11:35:38 -0700] "PUT /_ilm/policy/heartbeat-7.4.0 HTTP/1.1" 200 210 "-" "Go-http-client/1.1"
ip - r3elastic [02/Oct/2019:11:35:38 -0700] "HEAD /_template/heartbeat-7.4.0 HTTP/1.1" 200 142 "-" "Go-http-client/1.1"
diff-ip - r3elastic [02/Oct/2019:11:35:38 -0700] "POST /_bulk HTTP/1.1" 200 612 "-" "Go-http-client/1.1"
ip - r3elastic [02/Oct/2019:11:35:38 -0700] "PUT /_template/heartbeat-7.4.0 HTTP/1.1" 200 210 "-" "Go-http-client/1.1"
ip - r3elastic [02/Oct/2019:11:35:38 -0700] "GET /_alias/heartbeat-7.4.0 HTTP/1.1" 404 252 "-" "Go-http-client/1.1"

So, why is /_alias/hearbeat-7.4.0 missing?

Curl also returns a 404.

# curl localhost:5601/_alias/heartbeat-7.4.0
{"statusCode":404,"error":"Not Found","message":"Not Found"}

And should that really matter? If I curl for filebeat or metricbeat, they 404 as well. But they still push data into elasticsearch.

# curl localhost:5601/_alias/journalbeat-7.4.0
{"statusCode":404,"error":"Not Found","message":"Not Found"}
# curl localhost:5601/_alias/metricbeat-7.4.0
{"statusCode":404,"error":"Not Found","message":"Not Found"}
# curl localhost:5601/_alias/filebeat-7.4.0
{"statusCode":404,"error":"Not Found","message":"Not Found"}

Could the issue be something with how Apache is proxying requests to Elasticsearch?

In my access logs, I see lines like the following for all of my beats.

ip - - [04/Oct/2019:13:40:31 -0700] "PUT /%3Cjournalbeat-7.4.0-%7Bnow%2Fd%7D-000001%3E HTTP/1.1" 404 341 "-" "Go-http-client/1.1"

This is what the vhost for elasticsearch looks like:

Listen 9200
<VirtualHost *:9200>
ServerAdmin email
ServerName hostname
ServerAlias hostname
<Location "/">
AuthType Basic
AuthName "Restricted Files"
# (Following line optional)
AuthBasicProvider file
AuthUserFile "/etc/apache2/passwords"
Require user elastic
</Location>
#ProxyPreserveHost On
#ProxyPass "http://127.0.0.1:9201/"
RewriteEngine On
RewriteRule "^/(.*)" "http://127.0.0.1:9201/$1" [P] [NE]
ProxyPassReverse "/" "http://127.0.0.1:9201/"
</VirtualHost>

I tried using the [B] apache flag and the [NE] flag to see if that would prevent apache from messing up the path. Assuming that it was messing up the path....

As you can see in my docker-compose.yml from earlier, esnode1 is listening on 127.0.0.1:9201.

Here's the vhost for kibana. I've tried using a couple different ways of proxying.

<VirtualHost *:80>
  ServerAdmin email
  ServerName hostname
  ServerAlias hostname
  <Location "/kibana">
AuthType Basic
AuthName "Restricted Files"
# (Following line optional)
AuthBasicProvider file
AuthUserFile "/etc/apache2/passwords"
Require user elastic

  AuthLDAPBindDN "binduser"
          AuthLDAPBindPassword "password"
          AuthLDAPURL "ldaps uri"
          AuthType Basic
          AuthBasicProvider ldap
          AuthName "Please authenticate"
          Require ldap-user blah

#  ProxyPreserveHost On
#  ProxyPass "/" "http://127.0.0.1:5601/kibana/"
#  ProxyPassReverse "/" "http://127.0.0.1:5601/kibana/"
  </Location>
  RewriteEngine On
  RewriteRule "^/kibana/(.*)" "http://127.0.0.1:5601/kibana/$1" [P]
  ProxyPassReverse "/kibana" "http://127.0.0.1:5601/kibana/"

</VirtualHost>

Any ideas?

And, here's some more data.

I successfully ran setup on a host running metricbeat 7.3.2. Then I upgraded metricbeat to 7.4.0. Setup fails with the same 404 as on my desktop that's running my test cluster.

[root@remotehost metricbeat]# metricbeat -c /etc/metricbeat/metricbeat.yml setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
[root@remotehost metricbeat]# yum install metricbeat
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.metrocast.net
 * epel: mirrors.sonic.net
 * extras: mirror.dal10.us.leaseweb.net
 * updates: mirrors.tripadvisor.com
Resolving Dependencies
--> Running transaction check
---> Package metricbeat.x86_64 0:7.3.1-1 will be updated
---> Package metricbeat.x86_64 0:7.4.0-1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================
 Package             Arch            Version            Repository            Size
===================================================================================
Updating:
 metricbeat          x86_64          7.4.0-1            elastic-7.x           30 M

Transaction Summary
===================================================================================
Upgrade  1 Package

Total size: 30 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : metricbeat-7.4.0-1.x86_64                                       1/2 
warning: /etc/metricbeat/metricbeat.yml created as /etc/metricbeat/metricbeat.yml.rpmnew
warning: /etc/metricbeat/modules.d/system.yml created as /etc/metricbeat/modules.d/system.yml.rpmnew
  Cleanup    : metricbeat-7.3.1-1.x86_64                                       2/2 
  Verifying  : metricbeat-7.4.0-1.x86_64                                       1/2 
  Verifying  : metricbeat-7.3.1-1.x86_64                                       2/2 

Updated:
  metricbeat.x86_64 0:7.4.0-1                                                      

Complete!
[root@remotehost metricbeat]# metricbeat -c /etc/metricbeat/metricbeat.yml setup
Exiting: failed to create alias: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
: 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

Adding AllowEncodedSlashes On to my vhost got me past the 404 error.

Now I'm getting this:

filebeat -c /etc/filebeat/filebeat.yml setup
Exiting: failed to create alias: {"error":"Incorrect HTTP method for uri [/%3Cfilebeat-7.4.0-%7Bnow/d%7D-000001%3E] and method [PUT], allowed: [POST]","status":405}: 405 Method Not Allowed: {"error":"Incorrect HTTP method for uri [/%3Cfilebeat-7.4.0-%7Bnow/d%7D-000001%3E] and method [PUT], allowed: [POST]","status":405}

I tried enabling mod_allowmethods in apache and setting AllowMethods GET POST PUT DELETE PATCH in my vhost, but it didn't help. I also tried AllowMethods reset.

Any ideas?

Per https://discuss.opendistrocommunity.dev/t/error-during-metricbeat-or-filebeat-setup/644/5 disabling ILM setup fixed the issue.

Am I the only one who doesn't see the connection between the 405 error and the fix? It makes no sense.....