I'm writing to you, because I need to your help / points of view about the architecture that I choose.
In fact, I just started a new project, I'm in preparation architecture stage.
Normally, I have, 4 stages: Collect data, storage, treatement data (We'll use Machine learning models also), data visualisation.
So, I will give you a vision about the architecture that we have choosen :
Industry phase ==> MQTT ==> Logstash ==> Elasticsearch ==> Kibana
My question is, how the data scientists will do their work ? I mean, because we don't have spark for example. Which framework is available to connect it with elastcisearch ? and to use by the data scientists. Because, our data scientist will use Python as program lanaguage ?
I need some clarification about the data science stage in our architecture from you the experts. Thanks a lot in advance