As an update:
I have found out that I get these kinds of errors if I have the following setting in my elasticsearch.yml:
xpack.monitoring.enabled: true
If I have it set to false, then everything starts just fine.
So this leads me to believe that the issue is related to X-pack and monitoring.
There's this other setting related to watches:
xpack.watcher.enabled: true
This also throws an exception, but at least it doesn't flood the logs like the monitoring errors. Example:
[2018-06-15T15:03:50,773][ERROR][o.e.x.w.WatcherService ] [elastic-host] couldn't load watch [ofkoEHUfTyOxi1yJz8Ylfg_kibana_version_mismatch], ignoring it...
java.lang.IllegalArgumentException: cannot execute scripts using [xpack_executable] context
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:305) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.(ScriptCondition.java:57) ~[?:?]
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.parse(ScriptCondition.java:67) ~[?:?]
at org.elasticsearch.xpack.watcher.Watcher.lambda$createComponents$2(Watcher.java:323) ~[?:?]
at org.elasticsearch.xpack.core.watcher.condition.ConditionRegistry.parseExecutable(ConditionRegistry.java:68) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:157) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:123) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:88) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherService.loadWatches(WatcherService.java:289) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherService.start(WatcherService.java:142) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:118) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.lambda$clusterChanged$3(WatcherLifeCycleService.java:174) ~[?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2018-06-15T15:03:50,778][ERROR][o.e.x.w.WatcherService ] [elastic-host] couldn't load watch [ofkoEHUfTyOxi1yJz8Ylfg_logstash_version_mismatch], ignoring it...
java.lang.IllegalArgumentException: cannot execute scripts using [xpack_executable] context
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:305) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.(ScriptCondition.java:57) ~[?:?]
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.parse(ScriptCondition.java:67) ~[?:?]
at org.elasticsearch.xpack.watcher.Watcher.lambda$createComponents$2(Watcher.java:323) ~[?:?]
at org.elasticsearch.xpack.core.watcher.condition.ConditionRegistry.parseExecutable(ConditionRegistry.java:68) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:157) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:123) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:88) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherService.loadWatches(WatcherService.java:289) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherService.start(WatcherService.java:142) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:118) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.lambda$clusterChanged$3(WatcherLifeCycleService.java:174) ~[?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2018-06-15T15:03:50,819][ERROR][o.e.x.w.WatcherService ] [elastic-host] couldn't load watch [ofkoEHUfTyOxi1yJz8Ylfg_elasticsearch_version_mismatch], ignoring it...
java.lang.IllegalArgumentException: cannot execute scripts using [xpack_executable] context
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:305) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.(ScriptCondition.java:57) ~[?:?]
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.parse(ScriptCondition.java:67) ~[?:?]
at org.elasticsearch.xpack.watcher.Watcher.lambda$createComponents$2(Watcher.java:323) ~[?:?]
at org.elasticsearch.xpack.core.watcher.condition.ConditionRegistry.parseExecutable(ConditionRegistry.java:68) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:157) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:123) ~[?:?]
at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:88) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherService.loadWatches(WatcherService.java:289) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherService.start(WatcherService.java:142) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:118) ~[?:?]
at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.lambda$clusterChanged$3(WatcherLifeCycleService.java:174) ~[?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
There's one more block similar to this but I'm over the character limit.