# How to fit data from unchangeable source into ECS?

**URL:** <https://discuss.elastic.co/t/how-to-fit-data-from-unchangeable-source-into-ecs/273126>\
**Category:** Elasticsearch\
**Tags:** ecs-elastic-common-schema\
**Created:** [May 17, 2021, 5:36am UTC](https://discuss.elastic.co/t/how-to-fit-data-from-unchangeable-source-into-ecs/273126 "2021-05-17T05:36:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheGerd](https://avatars.discourse-cdn.com/v4/letter/t/7bcc69/32.png) [@TheGerd](https://discuss.elastic.co/u/TheGerd)\
**Post date:** [May 17, 2021, 5:36am UTC](https://discuss.elastic.co/t/how-to-fit-data-from-unchangeable-source-into-ecs/273126/1 "2021-05-17T05:36:16Z")

</div>

Hello.  
I am used to the very basic concepts of Elasticsearch but now facing an issue where I don't know how to proceed. I am indexing data in a fixed format and I am not able to change the format at the source. Now I want to use these data with SIEM. I know about ECS but I am not sure how to convert the data. As far as I understand index template are only usefull to map field types but not to restructure documents. Is that correct? I read about processors and pipelines. Is a pipeline combined with a processor, such as grok, the way to go? How to do that? Would that be applicable to already indexed data or just to new data?  
Btw: I already indexed test data with filebeat to see if SIEM works in general. It did. I also know that SIEM doesn't recognize custom indices by default.  
Thanks in advance.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 17, 2021, 11:44pm UTC](https://discuss.elastic.co/t/how-to-fit-data-from-unchangeable-source-into-ecs/273126/2 "2021-05-17T23:44:39Z")

</div>

Welcome to our community! 😃

> [@TheGerd](#):
>
> Is a pipeline combined with a processor, such as grok, the way to go?

Yes, that's right.

Your best best it to use the Grok Debugger in Kibana to build yourself a pattern, or just play with [dissect](https://www.elastic.co/guide/en/elasticsearch/reference/current/dissect-processor.html) if it's a simple format. Then expand from there.

---

<div class="post-metadata">

**Author:** ![TheGerd](https://avatars.discourse-cdn.com/v4/letter/t/7bcc69/32.png) [@TheGerd](https://discuss.elastic.co/u/TheGerd)\
**Post date:** [May 18, 2021, 1:47pm UTC](https://discuss.elastic.co/t/how-to-fit-data-from-unchangeable-source-into-ecs/273126/3 "2021-05-18T13:47:20Z")

</div>

Good hints. I will have a look at that. Thank you!

---

<div class="post-metadata">

**Author:** ![ebeahan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebeahan/32/78989_2.png) [@ebeahan](https://discuss.elastic.co/u/ebeahan)\
**Post date:** [May 18, 2021, 8:07pm UTC](https://discuss.elastic.co/t/how-to-fit-data-from-unchangeable-source-into-ecs/273126/4 "2021-05-18T20:07:27Z")

</div>

An experimental tool called [`ecs-mapper`](https://github.com/elastic/ecs-mapper) might also be useful. ECS-mapper takes a CSV file of your data source's custom fields mapped to ECS fields and helps generate starter pipeline configurations

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 15, 2021, 8:08pm UTC](https://discuss.elastic.co/t/how-to-fit-data-from-unchangeable-source-into-ecs/273126/5 "2021-06-15T20:08:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.