Journalbeat is having issues getting started. It seems like it fails to create an alias?
journalctl -l --follow -u journalbeat eventually spits this out:
Oct 01 15:33:43 hostname journalbeat[30589]: 2019-10-01T15:33:43.068-0700 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): Connection marked as failed because the onConnect callback failed: failed to create alias: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Oct 01 15:33:43 hostname journalbeat[30589]: <html><head>
Oct 01 15:33:43 hostname journalbeat[30589]: <title>404 Not Found</title>
Oct 01 15:33:43 hostname journalbeat[30589]: </head><body>
Oct 01 15:33:43 hostname journalbeat[30589]: <h1>Not Found</h1>
Oct 01 15:33:43 hostname journalbeat[30589]: <p>The requested URL was not found on this server.</p>
Oct 01 15:33:43 hostname journalbeat[30589]: </body></html>
Oct 01 15:33:43 hostname journalbeat[30589]: : 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Oct 01 15:33:43 hostname journalbeat[30589]: <html><head>
Oct 01 15:33:43 hostname journalbeat[30589]: <title>404 Not Found</title>
Oct 01 15:33:43 hostname journalbeat[30589]: </head><body>
Oct 01 15:33:43 hostname journalbeat[30589]: <h1>Not Found</h1>
Oct 01 15:33:43 hostname journalbeat[30589]: <p>The requested URL was not found on this server.</p>
Oct 01 15:33:43 hostname journalbeat[30589]: </body></html>
Oct 01 15:33:43 hostname journalbeat[30589]: 2019-10-01T15:33:43.068-0700 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://elasticsearch:9200)) with 3 reconnect attempt(s)
If I try to run the setup tasks manually:
root@hostname:/etc/journalbeat# journalbeat -c /etc/journalbeat/journalbeat.yml setup
Exiting: failed to create alias: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
: 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
Testing output and config works just fine.
root@hostname:/etc/journalbeat# journalbeat -c /etc/journalbeat/journalbeat.yml test output
elasticsearch: http://elasticsearch:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 10.7.0.57
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
version: 7.4.0
root@hostname:/etc/journalbeat# journalbeat -c /etc/journalbeat/journalbeat.yml test config
Config OK
Annoyingly I have both metricbeat and filebeat running on the same host, with the same elasticsearch connection details, and they work just fine.
I've been running into this on 7.3.1, 7.3.2, and now 7.4.0....
Here is my journalbeat config export:
journalbeat:
inputs:
- paths: null
seek: cursor
output:
elasticsearch:
hosts:
- elasticsearch:9200
password: password
protocol: http
username: username
path:
config: /etc/journalbeat
data: /var/lib/journalbeat
home: /usr/share/journalbeat
logs: /var/log/journalbeat
processors:
- add_host_metadata: null
- add_cloud_metadata: null
setup:
dashboards:
enabled: true
kibana:
host: localhost:5601/kibana
template:
settings:
index:
number_of_shards: 1
I'm running kibana and elasticsearch via docker-compose on a test desktop in my office. Whereever you see hostname in my config, that represents the dns address of that desktop.
Here is my docker-compose file:
version: '3.3'
services:
esnode1:
image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
container_name: esnode1
environment:
- cluster.name=hostname-cluster
- node.name=esnode1
- discovery.seed_hosts=esnode2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- r3_cluster_esdata1:/usr/share/elasticsearch/data
# - ./config/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
ports:
- 127.0.0.1:9201:9200
healthcheck:
test: ["CMD", "curl","-s" ,"-f", "http://localhost:9200/_cat/health"]
networks:
- elknet
restart: always
esnode2:
container_name: esnode2
image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
environment:
- cluster.name=hostname-cluster
- node.name=esnode2
- discovery.seed_hosts=esnode1
- cluster.initial_master_nodes=esnode1,esnode2
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- r3_cluster_esdata2:/usr/share/elasticsearch/data
# - ./config/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
healthcheck:
test: ["CMD", "curl","-s" ,"-f", "http://localhost:9200/_cat/health"]
ports:
- 127.0.0.1:9202:9200
networks:
- elknet
restart: always
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:7.4.0
environment:
#SERVER_NAME: 127.0.0.1
ELASTICSEARCH_HOSTS: http://esnode1:9200
SERVER_HOST: 0.0.0.0
SERVER_PORT: 5601
SERVER_BASEPATH: /kibana
SERVER_REWRITEBASEPATH: "true"
ports:
- 127.0.0.1:5601:5601
networks:
- elknet
restart: always
depends_on:
- esnode1
volumes:
r3_cluster_esdata1:
driver: local
r3_cluster_esdata2:
driver: local
networks:
elknet:
driver: bridge
driver_opts:
com.docker.network.bridge.name: elknet
I do have Apache set up as a reverse proxy so that I can use http auth.
I do have remote machines successfully pushing filebeat and metricbeat data.
Any suggestions?