X-pack breaks ES - several errors such as : java.io.FileNotFoundException: \\.\pipe\controller_log_8340 and Unknown Source errors

Hi, I did a clean install windows elastic 5.4.3 starts just dandy (so did Kibana start ok). Then I installed x-pack and ES wont start. Then I uninstall x-pack and all good again. So many messages but kinda looks like the inability to find a log is where it's breaking? Lots of messages around 1.8.0_131 Java - seems like a culprit?
Keen to get x-pack in Kibana working tons of disk (Alienware lappy - Win 7 Ultimate Sp1, 64 bit, 16GB ram).

Base install with 4 minimal YML parm changes. I have a full log, but too verbose to include all (I can email a helper with full logs)
cluster.name: ek-application
node.name: "ek.data"
path.data: ../ekdata
path.logs: ../ekdata

C:\Users\EK>java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

  1. inital good startup
    C:\EKDev\es5.4.3>bin\elasticsearch
    <snip - start up stuff removed for brevity>
    [2017-06-30T15:18:43,552][INFO ][o.e.h.n.Netty4HttpServerTransport] [ek.data] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
    [2017-06-30T15:18:43,559][INFO ][o.e.n.Node ] [ek.data] started
    [2017-06-30T15:19:01,271][INFO ][o.e.n.Node ] [ek.data] closed
    Terminate batch job (Y/N)? y
    ...all good...

  2. then x-pack
    C:\EKDev\es5.4.3>bin\elasticsearch-plugin install x-pack
    -> Downloading x-pack from elastic
    [=================================================] 100%

    Continue with installation? [y/N]y
    -> Installed x-pack
    ...still all good...

  3. then elastic breaks (for brevity I've removed lines that look ok, and duplicate errors. hoping I don't remove a useful line?)
    C:\EKDev\es5.4.3>bin\elasticsearch
    [2017-06-30T15:21:41,309][INFO ][o.e.n.Node ] [ek.data] initializing ...
    [2017-06-30T15:21:41,560][INFO ][o.e.e.NodeEnvironment ] [ek.data] using [1] data paths, mounts [[OS (C:)]], net usable_space [393.6gb], net total_space [524.9gb], spins? [unknown], types [NTFS]
    [2017-06-30T15:21:41,562][INFO ][o.e.e.NodeEnvironment ] [ek.data] heap size [1.9gb], compressed ordinary object pointers [true]
    [2017-06-30T15:21:41,624][INFO ][o.e.n.Node] [ek.data] node name [ek.data], node ID [QDFe7VyaSa-zZxjtlgeP6Q]
    [2017-06-30T15:21:41,626][INFO ][o.e.n.Node ] [ek.data] version[5.4.3], pid[8340], build[eed30a8/2017-06-22T00:34:03.743Z], OS[Windows 7/6.1/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
    [2017-06-30T15:21:41,628][INFO ][o.e.n.Node ] [ek.data] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:
    +UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -
    Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -
    Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutO
    fMemoryError, -Delasticsearch, -Des.path.home=C:\EKDev\es5.4.3]

    [2017-06-30T15:21:44,774][INFO ][o.e.p.PluginsService] [ek.data] loaded plugin [x-pack]
    [2017-06-30T15:21:49,394][DEBUG][o.e.a.ActionModule] Using REST wrapper from plugin org.elasticsearch.xpack.XPackPlugin
    [2017-06-30T15:22:00,219][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [ek.data] uncaught exception in thread [main]
    org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to create native process factories for Machine Learning]; nested: FileNotFoundException[\.\pipe\controller_log_8340 (The system cannot find the file specified)];
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.4.3.jar:5.4.3]

    Caused by: org.elasticsearch.ElasticsearchException: Failed to create native process factories for Machine Learning
    at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:290) ~[?:?]
    at org.elasticsearch.xpack.XPackPlugin.createComponents(XPackPlugin.java:288) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$7(Node.java:411) ~[elasticsearch-5.4.3.jar:5.4.3]
    at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:1.8.0_131]


    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.3.jar:5.4.3]
    ... 6 more
    Caused by: java.io.FileNotFoundException: \.\pipe\controller_log_8340 (The system cannot find the file specified)
    at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_131]

    at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:297) ~[?:?]
    at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:286) ~[?:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_131]
    at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:139) ~[?:?]
    at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:106) ~[?:?]
    at org.elasticsearch.xpack.ml.job.process.ProcessPipes.connectStreams(ProcessPipes.java:140) ~[?:?]
    at org.elasticsearch.xpack.ml.job.process.NativeController.(NativeController.java:65) ~[?:?]
    at org.elasticsearch.xpack.ml.job.process.NativeControllerHolder.getNativeController(NativeControllerHolder.java:50) ~[?:?]
    at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:280) ~[?:?]
    at org.elasticsearch.xpack.XPackPlugin.createComponents(XPackPlugin.java:288) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$7(Node.java:411) ~[elasticsearch-5.4.3.jar:5.4.3]
    at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:1.8.0_131]


    ... 6 more

now no good...darn it...

  1. but uninstall x-pack
    C:\EKDev\es5.4.3>bin\elasticsearch-plugin remove x-pack
    -> removing [x-pack]...
    -> preserving plugin config files [C:\EKDev\es5.4.3\config\x-pack] in case of upgrade; delete manually if not needed

  2. then all good again....
    C:\EKDev\es5.4.3>bin\elasticsearch
    [2017-06-30T15:23:34,828][INFO ][o.e.n.Node] [ek.data] initializing ...
    [
    [2017-06-30T15:23:50,924][INFO ][o.e.n.Node] [ek.data] started

    [2017-06-30T15:24:08,640][INFO ][o.e.n.Node] [ek.data] closed
    Terminate batch job (Y/N)? y

help/pointers gratefully received. I saw a thread on permissions (Debian?) but if good, then no good, then good again after 'remove' , I'm hoping security is not a prob? Perhaps I need to manually make log files?
Kind regards,
Emmet (elastic newbee)

It looks like this issue is due to a problem with the Machine Learning component of X-Pack.

I'll ask one of the ML team to check on it for you, but in the meantime can you try disabling ML and see if that helps.

Add this:

xpack.ml.enabled: false

to your config/elasticsearch.yml file.

Brilliant Tim. It didn't work straight off. Tried and same error...

but... I uninstalled and reinstalled x-pack (with YML set) and all good. I used your name so that I could see it was getting the correct YML.

Steps

  1. modify as per your YML suggestion and to ensure the parms stick I used
    cluster.name: emmet-application-tim
    node.name: "emmet.data-tim"
    xpack.ml.enabled: false

started good
C:\EKDev\es5.4.3\bin>elasticsearch
[2017-06-30T18:31:25,198][INFO ][o.e.n.Node ] [emmet.data-tim] initializing ...
[2017-06-30T18:32:46,544][INFO ][o.e.c.m.MetaDataCreateIndexService] [emmet.data-tim] [.monitoring-alerts-2] creating index, cause [auto(bulk api)], templates [.monitoring-alerts-2], shards [1]/[1], mappings [doc]
[2017-06-30T18:32:47,052][INFO ][o.e.c.m.MetaDataCreateIndexService] [emmet.data-tim] [.watcher-history-3-2017.06.30] creating index, cause [auto(bulk api)], templates [watch_history_3], shards [1]/[1], mappings [watch_record]
[2017-06-30T18:32:47,977][INFO ][o.e.c.m.MetaDataMappingService] [emmet.data-tim] [.watcher-history-3-2017.06.30/shYkcpNFRf-cK7MZ6-4XAg] update_mapping [watch_record]
[2017-06-30T18:32:48,711][INFO ][o.e.c.m.MetaDataMappingService] [emmet.data-tim] [.watcher-history-3-2017.06.30/shYkcpNFRf-cK7MZ6-4XAg] update_mapping [watch_record]

and
http://localhost:9200/
requests the login as per the very good demo on the starter pack...

https://www.elastic.co/content-pack

elastic changeme <<--- PW

..it's 6:44pm in NZ.. I think we've earned a glass of wine. I'll have one for the both of us.
cheers,
Emmet

I see that 'getting started with elastic does not default' (starter pack) unless one has subscribed. Anyway - its an excellent webinar watch and has already saved me time by seeing the architecture and other useful parts of the presentation (40 mins..but once done you are way more informed)... kind regards, Emmet

go here...and register

https://www.elastic.co/webinars/getting-started-elasticsearch

@E_K do you see a line similar to this in your Elasticsearch log file?

[2017-06-30T01:13:47,694][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/3224] [Main.cc@128] controller (64 bit): Version 5.4.3 (Build 7ee6a31ee5db75) Copyright (c) 2017 Elasticsearch BV

If not, the next step in troubleshooting would be to find out why this program isn't being run.

It should be located at %ES_HOME%\plugins\x-pack\platform\windows-x86_64\bin\controller.exe (where %ES_HOME is the top level directory of Elasticsearch - you probably won't actually have an environment variable for this).

If you have such a program, please try running it. If it works it will appear to do nothing, and you can just kill it. But if it pops up any errors about missing DLLs or anything else then that would be interesting to know.

The other possible problem I can think of would be directory permissions. Could it be that the user you ran Elasticsearch as cannot see one of the directories somewhere along the path x-pack\platform\windows-x86_64\bin due to the way permissions are set?

thanks - Tim's advice was on the money. Sorted. cheers

here's the log David

left hand side is the non x-pack, rhs is the failed x-pack

Tim's disable tip means its working - but was late Fri night and a family
priority means I prob wont look at this until Monday - but appreciate the
quick reply

cheers,
Emmet

I was able to run controller.exe David and is showing ok under task manager

so that bit's dandy.

cheers,
Emmet

managed to run it fine David
cheers,
Emmet

There was no attachment on this comment. Maybe it was too big for the forum to accept.

But maybe there's no need to post the whole log file. The key thing I would look for in the log is whether there is a line similar to this when X-Pack is installed?

[2017-06-30T01:13:47,694][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/3224] [Main.cc@128] controller (64 bit): Version 5.4.3 (Build 7ee6a31ee5db75) Copyright (c) 2017 Elasticsearch BV

If there isn't then it suggests that the ES JVM cannot see the controller.exe program for some reason.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.