time="2018-01-29T22:23:01Z" level=info msg="Stopping signal distributor"
time="2018-01-29T22:23:01Z" level=info msg="Starting signal distributor"
time="2018-01-29T22:23:01Z" level=info msg="[filebeat] Starting (exec driver)"
time="2018-01-29T22:23:02Z" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 1/3."
time="2018-01-29T22:23:02Z" level=info msg="[filebeat] Stopping"
time="2018-01-29T22:23:04Z" level=info msg="[filebeat] Starting (exec driver)"
time="2018-01-29T22:23:05Z" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 2/3."
time="2018-01-29T22:23:05Z" level=info msg="[filebeat] Stopping"
time="2018-01-29T22:23:07Z" level=info msg="[filebeat] Starting (exec driver)"
time="2018-01-29T22:23:08Z" level=error msg="[filebeat] Backend finished unexpectedly, trying to restart 3/3."
time="2018-01-29T22:23:08Z" level=info msg="[filebeat] Stopping"
time="2018-01-29T22:23:10Z" level=info msg="[filebeat] Starting (exec driver)"
time="2018-01-29T22:23:11Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
time="2018-01-29T22:23:11Z" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!"
time="2018-01-29T22:23:21Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
My config file /etc/graylog/collector-sidecar/generated/filebeat.yml:
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/log/*/*.log
tags: ["json"]
output.elasticsearch:
hosts: ["http://0.0.0.0:9200"]
username: elastic
password: elastic
# A template is used to set the mapping in Elasticsearch
# By default template loading is disabled and no template is loaded.
# These settings can be adjusted to load your own template or overwrite existing ones
template:
# Template name. By default the template name is 'filebeat'.
name: "filebeat"
# Path to template file
path: /etc/filebeat/filebeat.template.json
# Overwrite existing template
overwrite: false
Am I missing something? Does the format look correct?
This is all running on the host machine. No containers on the system (yet). I have a single server running Graylog, Elasticsearch, and Collector-Sidecar.
Thank you for that tool tip! I resolved a few issues within Filebeat, but I am still receiving the error in /var/log/graylog/collector-sidecar/collector_sidecar.log:
time="2018-01-30T18:47:03Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
Running Filebeats, as you recommended, gave me this below. I assume this is normal:
2018/01/30 18:52:27.821829 client.go:214: DBG Publish: {
"@timestamp": "2018-01-30T18:52:27.714Z",
"beat": {
"hostname": "ip-172-31-2-210",
"name": "ip-172-31-2-210",
"version": "5.4.2"
},
"input_type": "log",
"message": "\tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]",
"offset": 2810082,
"source": "/var/log/graylog-server/server.log",
"tags": [
"json"
],
"type": "log"
}
2018/01/30 18:52:27.821862 output.go:109: DBG output worker: publish 50 events
2018/01/30 18:52:27.821875 client.go:633: DBG ES Ping(url=http://0.0.0.0:9200, timeout=1m30s)
2018/01/30 18:52:27.823255 client.go:657: DBG Ping status code: 200
2018/01/30 18:52:27.823272 client.go:658: INFO Connected to Elasticsearch version 5.6.2
2018/01/30 18:52:27.823312 output.go:317: INFO Trying to load template for client: http://0.0.0.0:9200
2018/01/30 18:52:27.823324 client.go:673: DBG HEAD http://0.0.0.0:9200/_template/filebeat <nil>
2018/01/30 18:52:27.823878 output.go:341: INFO Template already exists and will not be overwritten.
2018/01/30 18:52:27.836615 client.go:256: DBG PublishEvents: 50 events have been published to elasticsearch in 12.704166ms.
2018/01/30 18:52:27.836751 single.go:150: DBG send completed
2018/01/30 18:52:27.836763 output.go:109: DBG output worker: publish 50 events
2018/01/30 18:52:27.851124 client.go:256: DBG PublishEvents: 50 events have been published to elasticsearch in 14.341247ms.
I don't believe so. I want to say it is collector-sidecar. Anytime I try to hit the "Start Input" button in http://<my_graylog_ip>:9000/system/inputs, it returns an error, and notifies me to check logs. The only log file that seems to change is collector-sidecar.
Whenever I check the logs, the only log that seems to have any new info on it is collector_sidecar.log.<current_date>:
time="2018-01-30T20:22:00Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
time="2018-01-30T20:22:10Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
time="2018-01-30T20:22:20Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
time="2018-01-30T20:22:30Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
time="2018-01-30T20:22:40Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
time="2018-01-30T20:22:50Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
This very much sounds like an issue with graylog. Support to configure and start a collector from within graylog is provided by graylog, not beats. Graylog has it's own agent, waiting for configs from graylog, and applies those when you click the "Start Input" button.
I haven't tried the sidecar myself, but I guess you configured it to fetch configs for a given 'machine role/tag' + the agent is complaining about not being able to fetch/find said configs. I guess you misconfigured some tag.
Note: it seems the sidecar is attempting to overwrite your local configs.
Also check the graylog forum if you can find something related.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.