How can I add a basetime to timestamps?

Elastic Stack Beats
1

Hi everyone,

we are new in Elastic framwork and we would like to monitor file logs from a robot control, with the following pattern. The problem is that timestamp are relative to a base time, so we would need to process the base time + line timestamp to obtain the absolute timestamp.

It is possible to do this?

Note: showed time are just example, we would need a milliseconds precission add

File log patter from Robot:

Name: Test0003
Simulation Start: 2018/04/23 12:38:40

Time     Data1     Data2
0.1    28,34      82,42
0.2    28,34      82,42
0.3    28,34      82,42
[...]

What we need in elastic:

Time                  Data1     Data2     Simulation
2018/04/23 12:38:41    28,34      82,42     Test0003
2018/04/23 12:38:42    28,34      82,42     Test0003
2018/04/23 12:38:43    28,34      82,42     Test0003
[...]

Describing the overall solution, we need to obtain logs from the robot and from other sources, but they must share the same time base (there will be a NTP v4. service to synchronize every source).

Is it possible to achieve this?

Thanks.

2

Hello @joe_bermejales,

This look more like a job for a custom plugin (or ruby filter) in Logstash, but you can still use filebeat to send the log to logstash and do the transformation there.

3

Thanks for your response @pierhugues.

The control system will produce about 500 to 1000 log lines per second, and the total time could reach 5 minutes or more.

Can logstash or filebeat manage these 300.000 log lines? It is too much? Where can I find a performance rate or max. limit for the Elastic ecosystem?

Thanks in advance.

4

5 mins or more,maybe it would make sense to run a query on your ES cluster and populate the missing time?

Can logstash or filebeat manage these 300.000 log lines? It is too much? Where can I find a performance rate or max. limit for the Elastic ecosystem?

That should not be a problem, you can scale your ES cluster or your Lostash instance horizontally if you need.