Best architecture

Hi all,

After reading some tutorials it's time to start working with elasticsearch.
Following the tutorial how to set-up was pretty easy.

First of all a little history from my side. I am used to program website's and used sql databases to work with. I am not used the way of thinking with these search machines.

Before i start i want to know whats the best way to set up my architecture.
We have around 50 systems in the field around the world that transport objects from a to b.
We see that logging and data analytics is getting more and more important. In the ideal situation we want a system warning even before the customer can call us with a problem, but thats for later.

Because we are young we have the ability to change things relatively fast and that makes us flexible.

Our logging contains mainly text, something like this:

time, [debug lvl], application name, message

The messages gives us more information about the system. It prints for example how long it took to get from a to b, with type object we are carrying and how we picked up the object.

Here some example lines:

2017-12-08 16:33:53,103 ;systemid XXX; [INFO]; Application:; Application started;
2017-12-08 16:34:12,1;systemid XXX; [INFO]; Application:; Configuration 11 started;
2017-12-08 16:34:32,30;systemid XXX; [INFO]; Application:; Gripper changed to IDX and type to TYPEX;
2017-12-08 16:34:56,23;systemid XXX; [INFO]; Application:; Object picked;
2017-12-08 16:35:53,56;systemid XXX; [INFO]; Application:; App pauze;
2017-12-08 16:36:12,103;systemid XXX; [INFO]; Application:; App resume;
2017-12-08 16:36:47,84;systemid XXX; [INFO]; Application:; Object placed;
2017-12-08 16:37:26,47;systemid XXX; [INFO]; Application:; Left full;
2017-12-08 16:34:56,23;systemid XXX; [INFO]; Application:; Object picked;
2017-12-08 16:36:47,84;systemid XXX; [INFO]; Application:; Object placed;
2017-12-08 16:34:56,23;systemid XXX; [INFO]; Application:; Object picked;
2017-12-08 16:36:47,84;systemid XXX; [INFO]; Application:; Object placed;
2017-12-08 16:34:56,23;systemid XXX; [INFO]; Application:; Object picked;
2017-12-08 16:36:47,84;systemid XXX; [INFO]; Application:; Object placed;
2017-12-08 16:34:56,23;systemid XXX; [INFO]; Application:; Object picked;
2017-12-08 16:36:47,84;systemid XXX; [INFO]; Application:; Object placed;
2017-12-08 16:38:25,18;systemid XXX; [INFO]; Application:; Waiting for object to arrive;
2017-12-08 16:34:56,23;systemid XXX; [INFO]; Application:; Object picked;
2017-12-08 16:36:47,84;systemid XXX; [INFO]; Application:; Object placed;
2017-12-08 16:38:51,25;systemid XXX; [INFO]; Application:; Waited for object to arrive;
2017-12-08 16:33:53,103;systemid XXX; [INFO]; Application:; Application stopped;
2017-12-08 16:34:12,1;systemid XXX; [INFO]; Application:; Configuration 1 started;
2017-12-08 16:34:32,30;systemid XXX; [INFO]; Application:; Gripper changed to IDX and type to TYPEX;
2017-12-08 16:34:56,23;systemid XXX; [INFO]; Application:; Object picked;
2017-12-08 16:36:47,84;systemid XXX; [INFO]; Application:; Object placed;
2017-12-08 16:39:31,76;systemid XXX; [INFO]; Application:; Both full;

(above is just an example, time etc are not correct)

There are a few basic things we want to analyse with elastic in combination with kibana:
Whats the average cycle time, whats the cycle time the past 24 hours(time vs cycle time). How many objects have been moved with a type of gripper. What are the latest errors, how long did we wait for products, etc.

I have two concepts of how the architecture can be made but i am not sure if that's the best way.
One of the concepts could something like this(i will do it in database structure because i am familiar with it):
id, timestamp, systemid XXX, debugtype, application, message

Or:

id, timestamp, systemid XXX, debug type, application, object picked (bool), object placed9bool), gripper type (integer), left full (bool), right full (bool), app state, configuration.

If you use the first one you need to filter all the messages, in the second example you only have to check for the bools are true.

What would be the best way to add in elastic? Perhaps there is a better way to make a table in elastic.
I hope you can help me out with this question.

A second question is; is it wise to put all the logs from all the systems together on one index or an index for every system apart.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.