The problem seem to be multifaceted:
a) if elastic cannot initialize a single pipeline on startup, it doesn't even try the other pipelines - a single problem brings the house down
b) elastic seems to initialize things in a bad order, ie. pipelines before stored scripts - thus pipelines that use store scripts always fail on startup.
On all our elastic clusters we see the following type of error in our elastic logs, on every startup:
[2017-06-28T14:48:31,428][WARN ][o.e.c.s.ClusterService ] [UTf_V3O] failed to notify ClusterStateApplier
org.elasticsearch.ElasticsearchParseException: Error updating pipeline with id [xxx]
at org.elasticsearch.ingest.PipelineStore.innerUpdatePipelines(PipelineStore.java:85) ~[elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.ingest.PipelineStore.applyClusterState(PipelineStore.java:68) ~[elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:856) [elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:810) [elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:628) [elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:1112) [elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-5.2.1.jar:5.2.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_112]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_112]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_112]
Caused by: org.elasticsearch.ResourceNotFoundException: Unable to find script [xxx] in cluster state
at org.elasticsearch.script.ScriptService.getScriptFromClusterState(ScriptService.java:369) ~[elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.script.ScriptService.compileInternal(ScriptService.java:311) ~[elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:235) ~[elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:142) ~[?:?]
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:88) ~[?:?]
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:298) ~[elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:251) ~[elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.ingest.Pipeline$Factory.create(Pipeline.java:122) ~[elasticsearch-5.2.1.jar:5.2.1]
at org.elasticsearch.ingest.PipelineStore.innerUpdatePipelines(PipelineStore.java:81) ~[elasticsearch-5.2.1.jar:5.2.1]
... 11 more