# Moving my first logs to ECS **URL:** \ **Category:** Elasticsearch\ **Tags:** ecs-elastic-common-schema\ **Created:** [January 19, 2021, 3:09pm UTC](https://discuss.elastic.co/t/moving-my-first-logs-to-ecs/261567 "2021-01-19T15:09:00Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![leostereo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leostereo/32/74891_2.png) [@leostereo](https://discuss.elastic.co/u/leostereo)\ **Post date:** [January 19, 2021, 3:09pm UTC](https://discuss.elastic.co/t/moving-my-first-logs-to-ecs/261567/1 "2021-01-19T15:09:00Z")
Hi friends. After succesing submiting my custom events into elk and visalize ok kibana, I would like to improve my event login format. After googlin about creating custom ecs index and indexing its documents , I did not success. First , tryed to manually create an example index, doing: ``` PUT /ecs_sample { "mappings": { "timestamp": {"type": "date"}, "log.level": {"type": "keyword"}, "message": {"type": "keyword"}, "service": { "name": {"type": "keyword"} }, "event": { "severity": {"type": "short"}, "timezone": "Hora ARG", "created": {"type": "date"}, "category": {"type": "keyword"} }, "ecs": { "version": {"type": "keyword"} } } } ``` but elk yelds: ``` "root_cause" : [ { "type" : "mapper_parsing_exception", "reason" : "Root mapping definition has unsupported parameters: [ecs : {version={type=keyword}}] [service : {name={type=keyword}}] [log.level : {type=keyword}] [message : {type=keyword}] [event : {severity={type=short}, timezone=Hora ARG, created={type=date}, category={type=keyword}}] [timestamp : {type=date}]" } ``` So ... continue reading and install ECS Tooling from: [https://github.com/elastic/ecs/blob/master/USAGE.md#setup-and-install](https://github.com/elastic/ecs/blob/master/USAGE.md#setup-and-install) but ... after install can not make it run: ``` [root@devel ecs]# python scripts/generator.py Traceback (most recent call last): File "scripts/generator.py", line 7, in from generators import csv_generator File "/opt/ecs/scripts/generators/csv_generator.py", line 5, in from generator import ecs_helpers File "/opt/ecs/scripts/generator.py", line 7, in from generators import csv_generator ImportError: cannot import name csv_generator ``` Have not idea what to do , im not familiarized with python. As last attempt , I tryed to copy the mapping from an existing index. I enable system filebeat module, I can see data but when trying to analize its mapping doing: `GET filebeat-7.9.0/_mapping` Dont understand what is in response ... it is a very large document. So: Please if you can help me to accomplish this would be great. Do I need to ingest my logs with logtash or filebeat or can I continue using the api ? Can I copy the index mapping from an existing / template index and then add my fields? Is there an exisiting php / perl library to log data with ECS format ? Is there some working and not gigant doc to learn this ? Any idea would be wellcome, Leandro. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 25, 2021, 2:00am UTC](https://discuss.elastic.co/t/moving-my-first-logs-to-ecs/261567/2 "2021-01-25T02:00:50Z")
The request is malformed.... I will put it at the bottom but overall. You should probably read about [Mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) and [Index Template](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) which is how to apply mapping to a pattern of indexes. But lets focus on these _Do I need to ingest my logs with logtash or filebeat or can I continue using the api ?_ If your app is writing logs to a file (Especially ECS logs see below) ... I would start with Filebeat you will get a lot of the mapping etc for free. _Can I copy the index mapping from an existing / template index and then add my fields?_ Yes, but you need to be careful and format it correctly, which you did not. If you learn a bit and name your fields correct with the builtin in [dynamic templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html) most of it could be taken care of for you.... just using the base filebeat index . _Is there an exisiting php / perl library to log data with ECS format ?_ Yes Right [Here](https://github.com/elastic/ecs-logging-php) _Is there some working and not gigant doc to learn this ?_ Perhaps start with the many free webinars like this one > **[Introduction to logging with the ELK Stack: A primer for beginners](https://www.elastic.co/webinars/introduction-elk-stack)** > > What was the ELK Stack is now the Elastic Stack. In this video you will learn how combining the massively popular open source project Elasticsearch, Logstash, and Kibana delivers actionable insights in real time from almost any type of structured and... If it were me and I was learning.... 1. I would watch a couple of the Logging / Observability webinars and / or access our excellent free training like [this short quick start on logging](https://www.elastic.co/training/logging-quick-start) 2. I would use the php ECS logger and write to log files 3. Use filebeat to send the data to Elasticsearch 4. Learn and adjust from there.. 5. If you need to more specifics add an [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) to do additional parsing and put the fields in the existing ECS fields or name them properly so the dynamic template will take care of it for you..... or do what you are trying to do with a custom template which I gave you a sample below. BTW here is the correct mapping you were trying to create... I am not sure what you were trying to accomplish with the timezone fields so I took it out ... otherwise I would make it a keyword type. You can also create index templates via Kibana under Stack Management You should really create this as an index tempate ``` PUT /_index_template/ecs_sample { "index_patterns": [ "ecs-sample-*" ], "template": { "settings": { "number_of_shards": 1 }, "mappings": { "properties": { "@timestamp": { "type": "date" }, "log": { "properties": { "level": { "type": "keyword" } } }, "message": { "type": "keyword" }, "service": { "properties": { "name": { "type": "keyword" } } }, "event": { "properties": { "severity": { "type": "short" }, "created": { "type": "date" }, "category": { "type": "keyword" } } }, "ecs": { "properties": { "version": { "type": "keyword" } } } } } } } ``` ---
**Author:** ![leostereo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leostereo/32/74891_2.png) [@leostereo](https://discuss.elastic.co/u/leostereo)\ **Post date:** [January 25, 2021, 11:24am UTC](https://discuss.elastic.co/t/moving-my-first-logs-to-ecs/261567/3 "2021-01-25T11:24:42Z")
Thanks !!! you provided a lot of info ... I need to study. ---
**Author:** ![leostereo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leostereo/32/74891_2.png) [@leostereo](https://discuss.elastic.co/u/leostereo)\ **Post date:** [January 25, 2021, 11:27am UTC](https://discuss.elastic.co/t/moving-my-first-logs-to-ecs/261567/4 "2021-01-25T11:27:14Z")
> [@stephenb](#): > > If your app is writing logs to a file (Especially ECS logs see below) ... I would start with Filebeat you will get a lot of the mapping etc for free. I dont understand this: do you mean ; I should use filebeat on my app side to export logs or use filebeat on elk side to recibe logs ? just this. thanks again. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 25, 2021, 3:38pm UTC](https://discuss.elastic.co/t/moving-my-first-logs-to-ecs/261567/5 "2021-01-25T15:38:25Z")
App writes logs to file using php ECS logger Use Filebeat to read logs and send to elasticsearch. ---
**Author:** ![ebeahan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebeahan/32/78989_2.png) [@ebeahan](https://discuss.elastic.co/u/ebeahan)\ **Post date:** [January 25, 2021, 4:29pm UTC](https://discuss.elastic.co/t/moving-my-first-logs-to-ecs/261567/6 "2021-01-25T16:29:41Z")
> [@leostereo](#): > > So ... continue reading and install ECS Tooling from: > [https://github.com/elastic/ecs/blob/master/USAGE.md#setup-and-install](https://github.com/elastic/ecs/blob/master/USAGE.md#setup-and-install) > > but ... after install can not make it run I don't want to misdirect this discussion, but @leostereo, do feel free to open a separate topic if you need any help getting started with the ECS tooling. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 22, 2021, 4:29pm UTC](https://discuss.elastic.co/t/moving-my-first-logs-to-ecs/261567/7 "2021-02-22T16:29:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.