Watcher and X-Pack do not appear to run or even be installed despite using Elastic stack v6.3.2

My setup

I'm using the master branch of docker-elk via Docker for Mac v18.06.0-ce-mac70. The docker-elk repo is currently based on the Elastic stack v6.3.2, and specifically uses the following images (links to GitHub repositories included):

For convenience: link to the docker.elastic.co repository.

Necessity of installing of X-Pack

I'm a bit confused here about whether X-Pack is installed already or not.

The Elastic 'downloads' page for X-Pack clarifies:

In versions 6.3 and later, X-Pack is included with the default distributions of Elastic Stack, with all free features enabled by default.

The 'Installing X-Pack' documentation for ELK 6.3 clarifies:

By default, when you install Elasticsearch, Kibana, and Logstash, X-Pack is installed.
By default, all basic X-Pack features are enabled. You can enable or disable specific X-Pack features in the elasticsearch.yml , kibana.yml , and logstash.yml configuration files.

Installing X-Pack in Elasticsearch says:

By default, when you install Elasticsearch, X-Pack is installed.

Installing X-Pack in Kibana says:

By default, when you install Kibana, X-Pack is installed.

This all sounds like X-Pack should be installed and set up for me for this version of the Elastic stack, i.e. it is ready-to-use. However, that is not my experience, as I shall detail below.

Inability to use Watcher

The Info API page specifies the following request:

GET /_xpack

For me, invoking that command as-is in the Kibana console (or as a curl: curl -XGET 'http://localhost:9200/_xpack') returns:

{
  "error": {
    "root_cause": [
      {
        "type": "invalid_index_name_exception",
        "reason": "Invalid index name [_xpack], must not start with _.",
        "index_uuid": "_na_",
        "index": "_xpack"
      }
    ],
    "type": "invalid_index_name_exception",
    "reason": "Invalid index name [_xpack], must not start with _.",
    "index_uuid": "_na_",
    "index": "_xpack"
  },
  "status": 400
}

The Stats API specifies:

GET _xpack/watcher/stats

And for me, this fires an 'index not found' exception:

{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "resource.type": "index_expression",
        "resource.id": "_xpack",
        "index_uuid": "_na_",
        "index": "_xpack"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "resource.type": "index_expression",
    "resource.id": "_xpack",
    "index_uuid": "_na_",
    "index": "_xpack"
  },
  "status": 404
}

I have also come across this command:

GET _watcher/stats?pretty

For me, this returns:

{
  "error" : "Incorrect HTTP method for uri [/_watcher/stats?pretty] and method [GET], allowed: [POST]",
  "status" : 405
}

I also cannot perform PUTs (please disregard that the 'actions' field is empty for now; the response is the important bit):

PUT _watcher/watch/filebeat-error-alert
{
  "trigger" : {
    "schedule": "1m"
  },
  "input": {
    "search": {
      "request": {
        "indices": "filebeat-*",
        "body": {
          "query": {
            "match": {
              "level": "error"
            } 
          }
        }
      }
    }
  },
  "actions": {

  }
}

... This returns:

{
  "error": {
    "root_cause": [
      {
        "type": "invalid_index_name_exception",
        "reason": "Invalid index name [_watcher], must not start with '_', '-', or '+'",
        "index_uuid": "_na_",
        "index": "_watcher"
      }
    ],
    "type": "invalid_index_name_exception",
    "reason": "Invalid index name [_watcher], must not start with '_', '-', or '+'",
    "index_uuid": "_na_",
    "index": "_watcher"
  },
  "status": 400
}

Of note, my Elastic stack does not start up properly when specifying the key xpack.watcher.enabled (mentioned in the Elasticsearch v6.3 docs) in elasticsearch.yml, giving the error "unknown setting" (much like this post concerning xpack.notification.email for v6.1.1). To clarify, all of the above details I've supplied have been with a successfully started-up Elastic stack, not specifying that key in the elasticsearch.yml file.

Help needed

I would like to get Watcher working to set up my first watch, and X-Pack as well for other features such as notifications. However, I've been unable to make any progress. The documentation suggests that there is no setup required in order for Watcher to work from v6.3.x of the Elastic stack, but that does not match up with my experience. Could you please clarify what I may be missing. Thank you!

The images you are linking to are the oss images, and these do not come bundled with X-Pack. If you look at the docker.elastic.so repository, you can find links to the default images, which do contain X-Pack.

Thank you for the quick and decisive help, Christian. I can confirm now that this was the root of the problem.

Using the oss images (note how no X-Pack modules are loaded):

elasticsearch_1  | [2018-08-24T09:48:06,191][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [aggs-matrix-stats]
elasticsearch_1  | [2018-08-24T09:48:06,191][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [analysis-common]
elasticsearch_1  | [2018-08-24T09:48:06,191][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [ingest-common]
elasticsearch_1  | [2018-08-24T09:48:06,193][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [lang-expression]
elasticsearch_1  | [2018-08-24T09:48:06,193][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [lang-mustache]
elasticsearch_1  | [2018-08-24T09:48:06,195][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [lang-painless]
elasticsearch_1  | [2018-08-24T09:48:06,196][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [mapper-extras]
elasticsearch_1  | [2018-08-24T09:48:06,196][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [parent-join]
elasticsearch_1  | [2018-08-24T09:48:06,198][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [percolator]
elasticsearch_1  | [2018-08-24T09:48:06,200][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [rank-eval]
elasticsearch_1  | [2018-08-24T09:48:06,200][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [reindex]
elasticsearch_1  | [2018-08-24T09:48:06,201][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [repository-url]
elasticsearch_1  | [2018-08-24T09:48:06,201][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [transport-netty4]
elasticsearch_1  | [2018-08-24T09:48:06,202][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded module [tribe]
elasticsearch_1  | [2018-08-24T09:48:06,206][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded plugin [ingest-geoip]
elasticsearch_1  | [2018-08-24T09:48:06,209][INFO ][o.e.p.PluginsService     ] [AKyXfOo] loaded plugin [ingest-user-agent]

Using the default images:

elasticsearch_1  | [2018-08-24T09:50:43,070][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [aggs-matrix-stats]
elasticsearch_1  | [2018-08-24T09:50:43,070][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [analysis-common]
elasticsearch_1  | [2018-08-24T09:50:43,070][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [ingest-common]
elasticsearch_1  | [2018-08-24T09:50:43,072][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [lang-expression]
elasticsearch_1  | [2018-08-24T09:50:43,074][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [lang-mustache]
elasticsearch_1  | [2018-08-24T09:50:43,074][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [lang-painless]
elasticsearch_1  | [2018-08-24T09:50:43,075][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [mapper-extras]
elasticsearch_1  | [2018-08-24T09:50:43,076][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [parent-join]
elasticsearch_1  | [2018-08-24T09:50:43,076][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [percolator]
elasticsearch_1  | [2018-08-24T09:50:43,077][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [rank-eval]
elasticsearch_1  | [2018-08-24T09:50:43,077][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [reindex]
elasticsearch_1  | [2018-08-24T09:50:43,078][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [repository-url]
elasticsearch_1  | [2018-08-24T09:50:43,078][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [transport-netty4]
elasticsearch_1  | [2018-08-24T09:50:43,078][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [tribe]
elasticsearch_1  | [2018-08-24T09:50:43,078][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-core]
elasticsearch_1  | [2018-08-24T09:50:43,078][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-deprecation]
elasticsearch_1  | [2018-08-24T09:50:43,078][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-graph]
elasticsearch_1  | [2018-08-24T09:50:43,078][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-logstash]
elasticsearch_1  | [2018-08-24T09:50:43,079][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-ml]
elasticsearch_1  | [2018-08-24T09:50:43,080][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-monitoring]
elasticsearch_1  | [2018-08-24T09:50:43,080][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-rollup]
elasticsearch_1  | [2018-08-24T09:50:43,080][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-security]
elasticsearch_1  | [2018-08-24T09:50:43,083][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-sql]
elasticsearch_1  | [2018-08-24T09:50:43,083][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-upgrade]
elasticsearch_1  | [2018-08-24T09:50:43,083][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded module [x-pack-watcher]
elasticsearch_1  | [2018-08-24T09:50:43,085][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded plugin [ingest-geoip]
elasticsearch_1  | [2018-08-24T09:50:43,086][INFO ][o.e.p.PluginsService     ] [FPo-E0v] loaded plugin [ingest-user-agent]
...
elasticsearch_1  | [2018-08-24T09:51:04,066][INFO ][o.e.c.m.MetaDataIndexTemplateService] [FPo-E0v] adding template [.watches] for index patterns [.watches*]
elasticsearch_1  | [2018-08-24T09:51:04,296][INFO ][o.e.c.m.MetaDataIndexTemplateService] [FPo-E0v] adding template [.watch-history-7] for index patterns [.watcher-history-7*]
elasticsearch_1  | [2018-08-24T09:51:04,357][INFO ][o.e.c.m.MetaDataIndexTemplateService] [FPo-E0v] adding template [.triggered_watches] for index patterns [.triggered_watches*]
elasticsearch_1  | [2018-08-24T09:51:04,534][INFO ][o.e.c.m.MetaDataIndexTemplateService] [FPo-E0v] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
elasticsearch_1  | [2018-08-24T09:51:04,615][INFO ][o.e.c.m.MetaDataIndexTemplateService] [FPo-E0v] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
elasticsearch_1  | [2018-08-24T09:51:04,666][INFO ][o.e.c.m.MetaDataIndexTemplateService] [FPo-E0v] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
elasticsearch_1  | [2018-08-24T09:51:04,764][INFO ][o.e.c.m.MetaDataIndexTemplateService] [FPo-E0v] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
elasticsearch_1  | [2018-08-24T09:51:04,871][INFO ][o.e.c.m.MetaDataIndexTemplateService] [FPo-E0v] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]

Is there any resource that details the differences between the OSS and non-OSS images? And also the licence differences?

Thank you.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.