# Troubleshooting with machine learning **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [July 28, 2017, 7:00am UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910 "2017-07-28T07:00:02Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![echai0](https://avatars.discourse-cdn.com/v4/letter/e/958977/32.png) [@echai0](https://discuss.elastic.co/u/echai0)\ **Post date:** [July 28, 2017, 7:00am UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/1 "2017-07-28T07:00:02Z")
Hello, I'm a newbie who is learning machine learning of x-pack. I have a trouble with machine learning as I wrote![1|690x385] My goal is creating a single metric job. (Machine learning-\> create a new job -\> create a single metric job) My situation 1. I added ".monitoring-es-\*" with Time filter name "timestamp" on elasticsearch. (Management tab-\> Index pattern) 2. I made a single metric job and I got a graph for my ".monitoring-es-\*", I chose "mean" for aggregation and "node\_stats.os.cpu.load\_average.15m" for Field and "15m" for Bucket span. 3. However, My job didn't analyze the data which is different from the video of elastic's (the url for the video: [https://www.youtube.com/watch?v=DBRISS0UKcA](https://www.youtube.com/watch?v=DBRISS0UKcA)) Fore more detail, it does not analyze but nothing happend just made a job file and when I clicked "Machine learning " again and click "open count in Anomaly Explorer" It just show me "No results Found." I googled hundreds of time but I can't find any clue about this issue. Plz help me. ![1](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d91d90f84393f8bd60f8832483b02ef57fa29f15.png) ![4](https://us1.discourse-cdn.com/elastic/original/3X/e/6/e65237ad56c2195b58a5ab2ae19a6acc1b9e05c0.png) ![3](https://us1.discourse-cdn.com/elastic/original/3X/2/6/266181f8ad9a06e6d2c34dc13501e9bdf3629089.png) ![2](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8e11c16f8cd758c8aaec73bd5000ba87e4266254.png) ![5](https://us1.discourse-cdn.com/elastic/original/3X/9/2/92e5c882fb3418cc2d999f8d3cbc979f0d1f75d2.png) ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [July 31, 2017, 2:16am UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/2 "2017-07-31T02:16:45Z")
Hello, It's hard to tell exactly the sequence of things given your screenshots, but clearly you properly created a job (from what I can tell), however, the screenshot of the jobs page shows that the job has processed "0" records (i.e. it didn't analyze any of your data). So, when building the job in the Single Metric Wizard, make sure you select a good window of time for historical data. So either: 1. set the kibana time picker to something sensible (like last 7 days) and then click the button with the triangle on it: ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/9/89a110e1038bbe49abd46484cdf86d59754ebc8b.png) 2. Or, click the button that says "Use full .monitoring-es-\* data on it: ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/4/0479ee3a109f21f4b3d8ffd7abe3349580d8b7c1.png) Once you've done this, give your job a unique name (my screen looks like this): ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/2/62593d3e3378ffeacf88f0e7d234475909ad0793.png) Then click the "Create Job" button and you should see the data be analyzed with a little animation. Once finished, Click the "View Results" button: ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/7/975f98669d11584e5f854071b149b0ed8b7ecaf1.png) And then you'll see the results in the Single Metric Viewer: ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/a/ea780c144023e5a17ef43cb3f1d98d17b1154130.png) I hope this helps! ---
**Author:** ![echai0](https://avatars.discourse-cdn.com/v4/letter/e/958977/32.png) [@echai0](https://discuss.elastic.co/u/echai0)\ **Post date:** [July 31, 2017, 5:22am UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/3 "2017-07-31T05:22:37Z")
First of all, I really appreciate for your precious answer.! I didn't consider about the kibana time picker you mentioned. It seems work due to the graph is changing, when I change time peaker. However I switch the time peaker (last 24 hours or last 4hours and so on) , the analyzing process is still not working... and at the same time when i check the "job management", the processed records is still zero. I would be really happy if you can give me a hand more about it. If you need any imformation plz tell me. Have a nice day. ![cpu_full_data1](https://us1.discourse-cdn.com/elastic/original/3X/f/e/fedb6705713d11a7cb8b97b3a3005b52bc241579.png) ![cpu_full data2](https://us1.discourse-cdn.com/elastic/original/3X/c/d/cd7d1e3e5beb2e5ba73e9ad4c72ad88b4ecfee9a.png) ![cpu_4hours1](https://us1.discourse-cdn.com/elastic/original/3X/a/b/ab3f61b37ae9e39b5acbdea4c5cd62f7b532e0a7.png) ![cpu_4hours2](https://us1.discourse-cdn.com/elastic/original/3X/0/2/02a4bc0b81f8a7db95174f25c8ea74d48ab39a52.png) +ps) I followed exact way you show me and I got "stopped" for Datafeed state on Job management. This is picture for it. ![ps](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bb4a0cab40331e7e1ec82a9073fbb8caf2de539a.png) ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [July 31, 2017, 12:22pm UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/4 "2017-07-31T12:22:46Z")
Thanks, this is helpful information - because now it seems as if the problem you are having is likely not due to any configuration problem. So, what we need now is more detailed information as to what is happening behind the scenes. This information will be in the elasticsearch.log file. Please do the following: 1. Configure a job as you've done above but before clicking the "Create Job" button, keep track of where the end of the elasticsearch.log is. 2. Click the "Create Job" button in the UI and note the new messages that appear in the elasticsearch.log file. Please copy/paste these messages from the log here so that we can diagnose. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [July 31, 2017, 9:08pm UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/5 "2017-07-31T21:08:37Z")
Additionally, I would like to know the specs of the system you are running ML on (size of RAM, # of CPUs, etc.). Is it a single node setup for testing or is this a multi-node production system? ---
**Author:** ![echai0](https://avatars.discourse-cdn.com/v4/letter/e/958977/32.png) [@echai0](https://discuss.elastic.co/u/echai0)\ **Post date:** [August 1, 2017, 5:16am UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/6 "2017-08-01T05:16:56Z")
Always appreciate for your help. 1. spec of the system. I am running ML on CentOS7 installed on Server. the server spec is - system bit: 64bit - OS: CentOS7 - cpu: xeon(R) CPU E5-2609 v2 @2.5GHz number of cpu: 8 - memory: 65869812kB 1. elasticsearch.log -ps1. "파이프가 깨어짐" means "broken pipe error" (I think setting language into Korean makes me get this message.) -ps2. I paste the whole passage incase I could miss something. ![elasticsearch_logfile_image](https://us1.discourse-cdn.com/elastic/original/3X/8/d/8dbabfd0612821d5a2ea6033dc545c952f086768.png) (If you have a difficulty with seeing the code, please tell me your e-mail address. so I can send you the txt. fle) have a nice day ---
**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\ **Post date:** [August 1, 2017, 10:04am UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/10 "2017-08-01T10:04:58Z")
The Machine Learning feature is implemented using native processes that run outside of the JVM that runs Elasticsearch. In your case the native process called `autodetect` cannot be started. Usually this process would be started by another native process called `controller`, which is usually started when Elasticsearch is started. I can reproduce your sequence of error messages if I kill this `controller` process and then try to open a job. Please can you check what happens if you try and run the relevant native processes at the command prompt: ```auto $ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/controller --version $ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/autodetect --version ``` Do either of these commands complain about missing OS libraries or other OS-related problems? If not, the most likely scenario is that your `controller` process has been killed. Do you see it in the process list if you run `ps -e | grep controller`? How long has this node been running for? It would be helpful if you could check all the logs since the node started for messages containing either `controller` or `CppLogMessageHandler`. Something like: ```auto egrep 'controller|CppLogMessageHandler' $ES_HOME/logs/*log ``` Also, I will raise an issue for friendlier error reporting in the case where the `controller` process is not running for some reason. The way it's reported at the moment is impossible to understand. ---
**Author:** ![echai0](https://avatars.discourse-cdn.com/v4/letter/e/958977/32.png) [@echai0](https://discuss.elastic.co/u/echai0)\ **Post date:** [August 2, 2017, 1:41am UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/11 "2017-08-02T01:41:50Z")
1. I put $ES\_HOME/plugins/x-pack/platform/linux-x86\_64/bin/controller --version $ES\_HOME/plugins/x-pack/platform/linux-x86\_64/bin/autodetect --version in the terminal and I got following message for each of them. controller (64 bit): Version 5.5.0 (Build 9352b273163d45) Copyright (c) 2017 Elasticserach BV autodetect (64 bit): Version 5.5.0 (Build 9352b273163d45) Copyright (c) 2017 Elasticserach BV So as you say, I think it's ok with OS things. 1. "ps -ef |grep controller" I put that command to terminal and I got [2017-8-02|690x196] root 139969 55537 0 10:33 -pts/6 00:00:00 grep controller Isn't this message tell me that controller is on the process??? ![2017-8-02](https://us1.discourse-cdn.com/elastic/original/3X/8/2/827cd6084fec9bb99ecaef9917f19530a7d4d96e.jpg) ps) I made a new machine job again to check it would work, but same thing happened (No analysis) ps2) I tried to restart controller. but It didn't work. I used "kill -9 " command but it didn't work at all. ![kill controller](https://us1.discourse-cdn.com/elastic/original/3X/8/9/891ea111517f9b227c157447e718ab3125bbbe31.png) ps3) I tried to start controller and autodetect. However, controller didn't response at all and autodetect had a license problem. this is what i got on the terminal. ![autodect launch](https://us1.discourse-cdn.com/elastic/original/3X/7/5/75cdcc46c308f2ea24a24104baa972ba436ec4cd.png) 1. I checked log files and the log file which contains "controller" start from 2017-07-26. ![controller start](https://us1.discourse-cdn.com/elastic/original/3X/4/0/40ffbd44c7be61e2653f97e611342dae13659723.jpg) and I found a strange thing. It says [2017-07-26T17:48:46,834][INFO][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started ---
**Author:** ![echai0](https://avatars.discourse-cdn.com/v4/letter/e/958977/32.png) [@echai0](https://discuss.elastic.co/u/echai0)\ **Post date:** [August 2, 2017, 6:38am UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/12 "2017-08-02T06:38:36Z")
I finally i solved it. The main cause was that the elasticsearch was not actually runnung. ( I have to restart elasticserach) (when I restarted elasticserach, It did staop but not started) So I found out that I had written xpack.ml.enabled: true ml.enabled: true in elasticsearch.yml and I forgot about that. so, I delete "ml.enabled: true" and elasticsearch was now running. as you told me about, the controller is running when elasticsearch starts. Without your help (especially about controller and autodetect things) I can't solve it. Again, I really appreciate for your help.! Have a nice day. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 30, 2017, 6:38am UTC](https://discuss.elastic.co/t/troubleshooting-with-machine-learning/94910/13 "2017-08-30T06:38:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.