How to fit data from unchangeable source into ECS?

I am used to the very basic concepts of Elasticsearch but now facing an issue where I don't know how to proceed. I am indexing data in a fixed format and I am not able to change the format at the source. Now I want to use these data with SIEM. I know about ECS but I am not sure how to convert the data. As far as I understand index template are only usefull to map field types but not to restructure documents. Is that correct? I read about processors and pipelines. Is a pipeline combined with a processor, such as grok, the way to go? How to do that? Would that be applicable to already indexed data or just to new data?
Btw: I already indexed test data with filebeat to see if SIEM works in general. It did. I also know that SIEM doesn't recognize custom indices by default.
Thanks in advance.

Welcome to our community! :smiley:

Yes, that's right.

Your best best it to use the Grok Debugger in Kibana to build yourself a pattern, or just play with dissect if it's a simple format. Then expand from there.

Good hints. I will have a look at that. Thank you!

An experimental tool called ecs-mapper might also be useful. ECS-mapper takes a CSV file of your data source's custom fields mapped to ECS fields and helps generate starter pipeline configurations

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.