Hello,
I am working to create some real-time machine learning jobs and I am having a problem getting the jobs to continue running after their initial analysis. I have looked through some similar postings for troubleshooting but I have not yet found what is causing the issue.
After creation of a job (single or multiple metric) the job runs successfully and results are produced. I have checked the box to continue the job in real-time but after a short period I receive a job message stating "Datafeed has been retrieving no data for a while".
I have checked the processes and it appears that both the autodetect and controller processes are still running but I believe the autodetect process may be stalled. If I run gstack 3 times on the autodetect process the resulting files are identical. The output of S1 is below.
> #0 0x00007fa03e45049d in read () from /lib64/libpthread.so.0
> #1 0x00007fa03a16d89f in boost::iostreams::detail::file_descriptor_impl::read(char*, long) () from /opt/elasticsearch-5.5.1/plugins/x-pack/platform/linux-x86_64/bin/../lib/./l
> ibboost_iostreams-gcc62-mt-1_62.so.1.62.0
> #2 0x00007fa03d6f4919 in boost::iostreams::detail::indirect_streambuf<boost::iostreams::file_descriptor_source, std::char_traits<char>, std::allocator<char>, boost::iostreams::input_seekable>::underflow() () from /opt/elasticsearch-5.5.1/plugins/x-pack/platform/linux-x86_64/bin/../lib/libMlCore.so
> #3 0x00007fa03c0b5da5 in std::basic_streambuf<char, std::char_traits<char> >::xsgetn(char*, long) () from /opt/elasticsearch-5.5.1/plugins/x-pack/platform/linux-x86_64/bin/../lib/libstdc++.so.6
> #4 0x00007fa03c09025b in std::istream::read(char*, long) () from /opt/elasticsearch-5.5.1/plugins/x-pack/platform/linux-x86_64/bin/../lib/libstdc++.so.6
> #5 0x00007fa03c3f7052 in ml::api::CLengthEncodedInputParser::refillBuffer() () from /opt/elasticsearch-5.5.1/plugins/x-pack/platform/linux-x86_64/bin/../lib/libMlApi.so
> #6 0x00007fa03c3f7320 in ml::api::CLengthEncodedInputParser::parseStringFromStream(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) () from /opt/elasticsearch-5.5.1/plugins/x-pack/platform/linux-x86_64/bin/../lib/libMlApi.so
> #7 0x00007fa03c3f8730 in bool ml::api::CLengthEncodedInputParser::parseRecordFromStream<false, std::vector<boost::reference_wrapper<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<boost::reference_wrapper<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >(std::vector<boost::reference_wrapper<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<boost::reference_wrapper<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) () from /opt/elasticsearch-5.5.1/plugins/x-pack/platform/linux-x86_64/bin/../lib/libMlApi.so
> #8 0x00007fa03c3f76e3 in ml::api::CLengthEncodedInputParser::readStream(boost::function2<bool, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, boost::unordered::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&> const&) () from /opt/elasticsearch-5.5.1/plugins/x-pack/platform/linux-x86_64/bin/../lib/libMlApi.so
> #9 0x00007fa03c3a7751 in ml::api::CCmdSkeleton::ioLoop() () from /opt/elasticsearch-5.5.1/plugins/x-pack/platform/linux-x86_64/bin/../lib/libMlApi.so
> #10 0x000000000040b031 in ?? ()
> #11 0x00007fa03b6d1b35 in __libc_start_main () from /lib64/libc.so.6
> #12 0x0000000000409569 in ?? ()
> #13 0x00007ffeacb6b688 in ?? ()
> #14 0x000000000000001c in ?? ()
> #15 0x0000000000000015 in ?? ()
> #16 0x00007ffeacb6bd3e in ?? ()
> #17 0x00007ffeacb6bd4b in ?? ()
> #18 0x00007ffeacb6bd61 in ?? ()
> #19 0x00007ffeacb6bd85 in ?? ()
> #20 0x00007ffeacb6bd96 in ?? ()
> #21 0x00007ffeacb6bdab in ?? ()
> #22 0x00007ffeacb6bdc3 in ?? ()
> #23 0x00007ffeacb6bdda in ?? ()
> #24 0x00007ffeacb6bdf2 in ?? ()
> #25 0x00007ffeacb6be0e in ?? ()
> #26 0x00007ffeacb6be59 in ?? ()
> #27 0x00007ffeacb6be6c in ?? ()
> #28 0x00007ffeacb6bea3 in ?? ()
> #29 0x00007ffeacb6bed4 in ?? ()
> #30 0x00007ffeacb6bf05 in ?? ()
> #31 0x00007ffeacb6bf13 in ?? ()
> #32 0x00007ffeacb6bf46 in ?? ()
> #33 0x00007ffeacb6bf55 in ?? ()
> #34 0x00007ffeacb6bf8a in ?? ()
> #35 0x00007ffeacb6bf9a in ?? ()
> #36 0x00007ffeacb6bfcf in ?? ()
> #37 0x0000000000000000 in ?? ()
The commands below both return successfully without error.
$ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/controller --version
$ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/autodetect --version
Thank you for taking a look. let me know if there is any more information that I can provide, I am out of characters so I will post a reply with additional information.
