Filebeat 6.7.0 not pushing logs to Elasticsearch 6.7.0 after upgrade

Elastic Stack Beats
1

Filebeat stopped working after upgrading from 6.6.2 to 6.7.0

My Filebeat configuration is:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/www/current/log/production.log
    - /var/www/current/log/api_v2.production.log
  multiline.pattern: '^[EIWDF]\, '
  multiline.negate: true
  multiline.match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.elasticsearch:
  hosts: ["ip:29200"]

But /var/log/filebeat/filebeat shows:

2019-03-28T14:12:11.091Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://ip:9200)): Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please upgrade to the default distribution of Elasticsearch from elastic.co, or downgrade to the oss-only distribution of beats
2019-03-28T14:12:11.091Z	INFO	[publish]	pipeline/retry.go:189	retryer: send unwait-signal to consumer
2019-03-28T14:12:11.091Z	INFO	[publish]	pipeline/retry.go:191	  done
2019-03-28T14:12:11.091Z	INFO	[publish]	pipeline/retry.go:166	retryer: send wait signal to consumer
2019-03-28T14:12:11.091Z	INFO	[publish]	pipeline/retry.go:168	  done
2019-03-28T14:12:11.091Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://ip:9200)) with 5 reconnect attempt(s)
2019-03-28T14:12:11.125Z	INFO	elasticsearch/client.go:739	Attempting to connect to Elasticsearch version 6.7.0

However when i hit curl ip:9200 it return:

{
  "name" : "30KRsiU",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "xim_BCzFSXWrAO_kMO3TQA",
  "version" : {
    "number" : "6.7.0",
    "build_flavor" : "oss",
    "build_type" : "docker",
    "build_hash" : "8453f77",
    "build_date" : "2019-03-21T15:32:29.844721Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

How do I get this working? I am out of ideas. Thanks for any help.

2

I'm not sure what's broken but why not just revert Filebeat back to v6.6.2?

3

Hello,

We are having exactly the same problem. All of our Beats stopped sending metrics/logs to Elastic Cloud.
For us the error messages are different though. Here is a part of the error.

ERROR instance/beat.go:907 Exiting: Couldn't connect to any of the configured Elasticsearch hosts. [...] 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] requires authentication","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"action [cluster:monitor/main] requires authentication","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}]

And, I have checked all the authentication credentials. There is nothing different from the previous version. I purged and reinstalled the Beats but it was a no go.

The only way to fix it was to do a downgrade.

4

Very similar issue here as well. It seems that 6.7 is badly broken. Filebeat is not pushing anything to Logstash, Metricbeat is pushing info just fine though. And when I try to push the Filebeat modules to Logstash using:

sudo filebeat setup --pipelines --modules apache2,system

I end up with the following error:

Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://192.168.4.130:5044: Get http://192.168.4.130:5044: read tcp 192.168.1.248:40992->192.168.4.130:5044: read: connection reset by peer]

Trying to connect to Logstash via curl:

[user@host] % curl http://my.ip.address:5044

Gives me the error:

curl: (56) Recv failure: Connection reset by peer

Interestingly enough if I run netstat on the Elastic Stack server I can see that port 5044 has connections to it from two of the three systems that are sending logs, but it doesn't look like the logs are getting sent.

Had a heck of a time getting Kibana and Elasticsearch to stay running, a reboot of my Elastic Stack server finally seems to have fixed that issue, but still not shipping logs to Logstash. I really don't want to downgrade to 6.6 as I'm looking forward to a couple of the features in 6.7, but considering that it is unusable at this point I may have to.

1 Like
5

Hello,

I see the same problem since upgrading from 6.6.2 to 6.7.

Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please upgrade to the default distribution of Elasticsearch from elastic.co, or downgrade to the oss-only distribution of beats

Checking with tcpdump I see

GET /_xpack?human=false HTTP/1.1
Host: ip:9200
User-Agent: Go-http-client/1.1
Accept: application/json
Accept-Encoding: gzip

With the response

HTTP/1.1 400 Bad Request
Warning: 299 Elasticsearch-6.7.0-8453f77 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 167

Analysing further on Github, I think it might be related to this commit that adds licence check:

6

I solve the problem uninstalling filebeat and installing the OSS version downloaded from

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-6.7.0-amd64.deb

I couldn't find the OSS version from the official APT repository.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.