How to disable xpack check in Filebeat

Elastic Stack Beats
1

hello, I am a newbie to use ES and Kibana in Amazon Elasticsearch Service, which do not support xpack plugin.
I use Filebeat to send nginx log to ES directly. However, when run sudo filebeat setup -e, error happen like below:

ERROR   instance/beat.go:691    Exiting: 2 errors: Error checking if xpack is available: 401 Unauthorized: {"Message":"Your request: '/_xpack' is not allowed."}; Error checking if xpack is available: 401 Unauthorized: {"Message":"Your request: '/_xpack' is not allowed."}

the error happened because elasticsearch in Amazzon Elasticsearch Service don't support xpack plugin, and user can't install manually.

So, I add below config to /etc/filebeat/filebeat.yml in order to disable xpack and forbidden filebeat to check xpack availability. But the error still exists.

xpack.security.enabled: false
xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: false
xpack.reporting.enabled: false

Is there any way to disable filebeat xpack check?

Environments:
Filebeat: 6.4.0
Elasticsearch: 6.3 in Amazon ElasticSearch Service

2

You have two options;

  1. Upgrade to Elastic Cloud and get access to all the X-Pack functionality included
  2. Use https://www.elastic.co/downloads/beats/filebeat-oss
3

Can you share your complete filebeat.yml config file?

According to the docs the setting to disable monitoring is xpack.monitoring.enabled: false. The default value is false.

4

I installed the latest filebeat-sso, the error still exsit.

Error checking if xpack is available: 401 Unauthorized: {"Message":"Your request: '/_xpack' is not allowed."};
5

Can you share your complete filebeat.yml config file?

6

this is my filebeat.yml


#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*



filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

setup.kibana:
  host: "https://search-server-log-MY-AMAZON-ID.us-west-1.es.amazonaws.com:443/_plugin/kibana/"

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://search-server-log-MY-AMAZON-ID.us-west-1.es.amazonaws.com:443"]

xpack.security.enabled: false
xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: false
xpack.reporting.enabled: false
7

These settings do not exist in beats:

xpack.security.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: false
xpack.reporting.enabled: false

The problem is setup tries to install ML modules. Normally Elasticsearch gets us an error message telling us ML is not available and beats continue with setup phase. The Amazon Elasticsearch Service is not by us, we don't test against it and we can not guarantee compatibility.
You can try to run setup only with functionality you want to enable. This will setup dashboards, ingest node pipelines and the index templates:
$ sudo filebeat setup -e --dashboards --pipelines --template

8

Thanks steffens. This helped.

Facing the next issue on ingest-geoip. No access to AWS elasticsearch to install a plugin. (or I don't know if that is possible)

    2018-08-31T03:08:18.312Z        INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.3.1
    2018-08-31T03:08:18.437Z        INFO    fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-system-syslog-pipeline' loaded
    2018-08-31T03:08:18.471Z        ERROR   instance/beat.go:691    Exiting: Error loading pipeline for fileset system/auth: This module requires the ingest-geoip plugin to be installed in Elasticsearch. You can install it using the following command in the Elasticsearch home directory:
        sudo bin/elasticsearch-plugin install ingest-geoip
    Exiting: Error loading pipeline for fileset system/auth: This module requires the ingest-geoip plugin to be installed in Elasticsearch. You can install it using the following command in the Elasticsearch home directory:
        sudo bin/elasticsearch-plugin install ingest-geoip
    ```
9

It's not.

10

I found a post what worked for me

  1. Stop the filebeat service
  2. delete existing pipeline
  3. start the filebeat service
  4. Log
instance# systemctl stop filebeat.service 
instance# curl -XDELETE "https://vpc-<domain-name-123123412341.us-west-1.es.amazonaws.com>:443/_ingest/pipeline/filebeat-*"
instance# systemctl restart filebeat.service
11

@warkolm thanks for the reply. But I don't fully understand what you mean with "It's not"

You mean: I can install plugins.
or
"that is not the problem"

12

You cannot install plugins on the AWS service.

You should upgrade to Elastic Cloud.

13

I would love to. However I must have the data installed within my VPC for security reasons.
It was my understanding that Elastic Cloud currently does not offer to host the solution on my own VPC

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.