# How to create custom fields filein filebeat to preprocess before transporting to elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-create-custom-fields-filein-filebeat-to-preprocess-before-transporting-to-elasticsearch/118094>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 1, 2018, 7:41pm UTC](https://discuss.elastic.co/t/how-to-create-custom-fields-filein-filebeat-to-preprocess-before-transporting-to-elasticsearch/118094 "2018-02-01T19:41:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![akshaynshah](https://avatars.discourse-cdn.com/v4/letter/a/f05b48/32.png) [@akshaynshah](https://discuss.elastic.co/u/akshaynshah)\
**Post date:** [February 1, 2018, 7:41pm UTC](https://discuss.elastic.co/t/how-to-create-custom-fields-filein-filebeat-to-preprocess-before-transporting-to-elasticsearch/118094/1 "2018-02-01T19:41:15Z")

</div>

My Log Format:   
  
Time Hostname Severity Program Message  
  
My log line example:   
  
2018-02-01T21:1147+05:30 freebsd notice sshd Exiting on signal 15

I want filebeat to transfer these logs to elastic search (WITHOUT LOGSTASH) so that it gets saved in elastic search in the same format as: Time Hostname Severity Program and Message  
  
  
Instead, it gets stored in elastic search as offset, prospector.type, [beat.name](http://beat.name), beat.hostname etc  
  
  
I understand I need to change the fields.yml file but I am not understanding how to do that with my requirement i.e.  
  
Elastic Search should store data as:  
  
xyz.date xyz.hostname xyz.severity xyz.program xyz.message

---

<div class="post-metadata">

**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\
**Post date:** [February 2, 2018, 12:27pm UTC](https://discuss.elastic.co/t/how-to-create-custom-fields-filein-filebeat-to-preprocess-before-transporting-to-elasticsearch/118094/2 "2018-02-02T12:27:07Z")

</div>

This is not possible with filebeat alone as it doesn't have parsing capabilities (yet).

However, you can configure an `ingest pipeline` in Elasticsearch that does this field extraction for you, and have filebeat send the events through this pipeline.

Take a look at the [ingest node documentation](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html) to understand how to create an ingest pipeline. Add a [Grok Processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html) to the pipeline to parse the message and create the new fields.

Now you only need to add the pipeline name to the prospector configuration in `filebeat.yml` to have the logs pre-processed in the Elasticsearch ingest node:

```auto
filebeat.prospectors:
- type: log
  path: [...]
  pipeline: my_pipeline

```

If you prefer not to use an ingest pipeline, you can also develop your own filebeat module to parse your custom log files. See the [Beats Developer Guide](https://www.elastic.co/guide/en/beats/devguide/6.1/index.html), which contains a section for [Creating a New Filebeat Module](https://www.elastic.co/guide/en/beats/devguide/6.1/filebeat-modules-devguide.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 2, 2018, 12:27pm UTC](https://discuss.elastic.co/t/how-to-create-custom-fields-filein-filebeat-to-preprocess-before-transporting-to-elasticsearch/118094/3 "2018-03-02T12:27:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.