What is the best practice to convert my custom logs to ECS(Or not)

Elastic Stack Beats
1

Hi.

I am newbie on the ELK and i have done simple intergation with the cloud product(Very easy to use)
The problem started when i try to ship the logs with the new ECS(Elastic common schema) and i was lost..

I have custom logs.. Filebeat automatically added my log on the message property and filled in several fields automatically.
My question is if the best practice on my case is either create new log file that fits the ECS or use Logstash to convert my current log to be on the ECS .

I have on the same server several logs that i would like to send to elastic. which is another thing.. I have to use the same index or it is better to use different index for different logs.

I honestly don't know where to start.. Need some help here.

Thanks,
Aviad

2

Hi, thanks for the question about ECS. There is a great post about migrating to ECS.

After reading the article, feel free to let us know if you have further questions.

Thanks.

3

I read it and saw a few videos and webinar related to this topic and in general to elastic stack.
Unfortunately i still don't know what will be the best here.

Let's say i have a few log files:(I work with nodejs)

  1. Custom logs for all events in the system(Every entry point and the result) - let's call it myMessage.
    myMessage contain:
  • start time
  • time to process
  • request payload
  • response payload
  • ip and metadata
  1. Statistics - For example every 10 minutes we have the count of live sessions
  2. General logs

Now the questions is:

  1. Assuming i need some of the request payload fields to be indexes and not just as text because i need to to create dashboard from this details. The best here is to adjust it to the ECS schema all the fields? Is there any convention to do it? What about the data that is not best fit for ECS .. for example message in ECS is text.. or other fields

  2. General logs can be send to different index? for this use case, when i don't need to visulize i can just send all my logs as text to another index.

  3. Statistics logs should be sent in the same way? I do want to visulize them

Hope it make sense.. I really want to start with that and get the benefit of using ELK stack

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.