Xpack computation

[whiletruelearn (Krishna Sangeeth)]

Hi,

I read about the concept of ml nodes for xpack. But since xpack itself is a form of a plugin , i was curious to know where the actual compute is happening?

Is it internally some form of a rest service where data is pulled from elastic datastore and compute happening inside the ml nodes?

[Emanuil (Emanuil Tolev)]

@whiletruelearn

But since xpack itself is a form of a plugin , i was curious to know where the actual compute is happening?

It is in the plugins. Each node with the plugin does compute. I think you might like to look at the github repo of the C++ components and the Java plugin that uses the C++ core, it'll take you a while but you can see there how it integrates with the Java architecture.

EDIT: it's kind of general but you can see in this blog post [Elastic Stack Integration at the bottom] that the ML jobs are just Elasticsearch data that gets treated the same way as any other data in ES. So the plugin writes/reads its internal state in ES, and it accesses "target" data for analysis the same way.

@whiletruelearn

Is it internally some form of a rest service where data is pulled from elastic datastore and compute happening inside the ml nodes?

Not sure I get what you mean - plugins can access data in Elasticsearch.