Being a newbie to ML.
Can anyone advice where I should start troubleshooting?
job-name:noise-reduction-system-metrics-consolidated
job state: opened. memory status: ok.
latest timestamp is updating but more than 2 months back
Thanks in advance!
screenshot of the jobs screen? Is that job actually running?
@richcollier The job is definitely running. Its the only job which appears to have a very large, processed record. Is there anything i can zoom into?
My apologies, I misread your original post. You said the latest timestamp is updating. I read that as is not.
Ok, so if it is updating but is more than 2 months back, that means that you asked the job to analyze a lot of historical data and it is attempting to do that. However, it appears to be taking a long time to go through that historical data either because a) it is a massive amount of data, and/or b) your cluster cannot manage what’s being asked of it.
You really need to determine if the pace at which the job is progressing in the effort of looking this far back over time is faster than “real-time”. In other words, watch the progression and assess the speed at which it will catch up to “now” (if it ever will).
You could consider not having the job look back in time, but rather just process real-time data and just wait a few weeks. You could also consider refactoring the ML job to make it more efficient