# Elasticsearch explicit field mappings in template not taken when data is logged from filebeat through ingest pipeline

**URL:** <https://discuss.elastic.co/t/elasticsearch-explicit-field-mappings-in-template-not-taken-when-data-is-logged-from-filebeat-through-ingest-pipeline/297946>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [February 22, 2022, 7:27pm UTC](https://discuss.elastic.co/t/elasticsearch-explicit-field-mappings-in-template-not-taken-when-data-is-logged-from-filebeat-through-ingest-pipeline/297946 "2022-02-22T19:27:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![shi](https://avatars.discourse-cdn.com/v4/letter/s/f14d63/32.png) [@shi](https://discuss.elastic.co/u/shi)\
**Post date:** [February 22, 2022, 7:27pm UTC](https://discuss.elastic.co/t/elasticsearch-explicit-field-mappings-in-template-not-taken-when-data-is-logged-from-filebeat-through-ingest-pipeline/297946/1 "2022-02-22T19:27:01Z")

</div>

sir,  
I have created an **index life cycle policy** with name

```auto
lp_ilp_fgnew1

```

Created **index template** lp\_templateforotherfgutmwaf7 as follows

```auto
PUT _index_template/lp_templateforotherfgutmwaf7
{
  "index_patterns": ["lp_index-otherfgutmwaf7-*"],                 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "lp_ilp_fgnew1",      
      "index.lifecycle.rollover_alias": "lp_index-otherfgutmwaf7"    
    }
  }
}  

```

Bootstrapped **initial indices for lp\_index-otherfgutmwaf** as follows  
I have tried to **explicitly map some of the fields** also in the mappings section here

```auto
PUT /%3Clp_index-otherfgutmwaf7-%7Bnow%2Fd%7D-000001%3E
{
"aliases": {
    "lp_index-otherfgutmwaf7": {
      "is_write_index": true
    }
  },
  "mappings" : {
      "properties" : {
        "action" : {
              "type" : "keyword"
        },
        "direction" : {
          "type" : "text"
        },
        "dstintf" : {
          "type" : "integer"
        }
      },
      "dynamic": true
    }
}

```

Created **ingest pipe line lp\_index-otherfgutmwaf7-pipeline** as follows

```auto
PUT _ingest/pipeline/lp_index-otherfgutmwaf7-pipeline
{
  "description": "Ingest pipeline",
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern" : "%{timestamp} _gateway %{kvmsg}"
      }
    },
    {
      "kv": {
    "field": "kvmsg",
    "field_split": " (?=[a-z\\_\\-]+=)",
    "value_split": "=",
    "trim_value": "\""
  }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": [
          "ISO8601"
        ],
        "output_format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSXXX"
      }
    },
    {
      "remove": {
        "field": "timestamp",
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": "message",
        "ignore_missing": true
      }
    },  
    {
      "remove": {
        "field": "kvmsg",
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": "devname",
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": "devid",
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": "type",
        "ignore_missing": true
      }
    }
  ]
}

```

Log data is populated from **filebeat** to the index through the pipeline.

Log data population to the index is also tested using direct **post** command as follows

```nohighlight
{
  "action": "block",
  "direction": "output",
  "testfield": "Testing",
  "dstintf": "6"
}

```

Log data from file gets loaded in index from file beat through pipeline **only when the explicit mappings section in index template is not there.**. Using direct **post** command also works.

**When explicit mappings section in index template is there** even when file data DOES NOT gets loaded in index from file beat through pipeline , it gets loaded  
using direct **post** command.

Why is it so ?  
How to load Log data from file in index from file beat through pipeline **when the explicit mappings section in index template** is there?

thanks and regards  
shini  
How to

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 23, 2022, 3:56am UTC](https://discuss.elastic.co/t/elasticsearch-explicit-field-mappings-in-template-not-taken-when-data-is-logged-from-filebeat-through-ingest-pipeline/297946/2 "2022-02-23T03:56:43Z")

</div>

Please share your filebeat.yml

---

<div class="post-metadata">

**Author:** ![shi](https://avatars.discourse-cdn.com/v4/letter/s/f14d63/32.png) [@shi](https://discuss.elastic.co/u/shi)\
**Post date:** [February 23, 2022, 6:28am UTC](https://discuss.elastic.co/t/elasticsearch-explicit-field-mappings-in-template-not-taken-when-data-is-logged-from-filebeat-through-ingest-pipeline/297946/3 "2022-02-23T06:28:43Z")

</div>

Sir,  
It is now understood that from filebeat though pipeline the issue was with

```auto
 "dstintf" : {
          "type" : "integer"
        }

```

It was my mistake ,  
actual value of **dstintf** from log file is **port6** and not **6** which was used while successfully testing using the **POST** command.  
So filebeat was throwing **integer parsing error** or this field and it failed. I could see all these error messaages in filebeat clearly after I commented out the modules section in filebeat which was also generating lots of logs.

The issue is solved  
Thanks and regards,  
shini

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2022, 6:28am UTC](https://discuss.elastic.co/t/elasticsearch-explicit-field-mappings-in-template-not-taken-when-data-is-logged-from-filebeat-through-ingest-pipeline/297946/4 "2022-03-23T06:28:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.