# Please help, Logstash can not parse field value to created dynamic index

**URL:** https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188
**Category:** Logstash
**Created:** [April 3, 2019, 12:08pm UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188 "2019-04-03T12:08:35Z")
**Posts on this page:** 14
**Page:** 1

<div class="post-metadata">

### Author: ![Worradorn\_Sirisang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/worradorn_sirisang/32/41290_2.png) [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)
#### Post date: [April 3, 2019, 12:08pm UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/1 "2019-04-03T12:08:36Z")

</div>

Hello team,  
I face a problem about logstash send data to missing elasticsearch index.

my architecture is outlined below.  
Filebeats -\> Logstash -\> Elasticsearch/AWS Elasticsearch

my machine have Filebeat agent for collecting application logs and send to Logstash. I add logappname and logapphostname as custom fields in "fields" in Filebeat configuration file.

```
- type: log
  enabled: true
  paths:
     - /home/user/app/current/var/log/p2c2p123.log
  multiline.pattern: '^\[[0-9]{4}-{0-9}{2}-{0-9}{2}\ [0-9]{2}:[0-9]{2}:[0-9]{2}\]'
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: '\[\]'
  fields:
       logappname: p2c2p123
       logapphostname: host1.testhostclub.local
  fields_under_root: true

```

on my Logstash, I match timestamp in message. my code is outlined below.

```
filter {
    grok {
           match =&gt; { "message" =&gt; "\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}(:%{WORD:logtype}:)?%{SPACE}%{WORD:CLASS}.%{WORD:loglevel}:%{DATA}" }
              }     
}

```

And I have another configuration file as "send-to-es.conf" for setting up Logstash output. my code is outlined below.

```
output {
      amazon_es{
             hosts => ["<ES Host>"]
             region => "ap-southeast-1"
             index => "%{logappname}-%{logapphostname}-%{+YYYY.MM.dd}"
      }
      #stdout { codec => ruby-debug }
 }

```

----\> in this point

on my elasticsearch, I see my data in the p2c2p123-host1.testhostclub.local-2019.04.03  
and some time, I see my data store in %{logappname}-%{logapphostname}-2019.04.03

I does not understand why this %{logappname}-%{logapphostname}-2019.04.03 is occur on my ES  
Please you help me to resolve my problem.

Regards

---

<div class="post-metadata">

### Author: ![Worradorn\_Sirisang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/worradorn_sirisang/32/41290_2.png) [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)
#### Post date: [April 4, 2019, 3:33am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/2 "2019-04-04T03:33:24Z")

</div>

any update?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 4, 2019, 5:20am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/3 "2019-04-04T05:20:17Z")

</div>

This typically mean that you have data coming in for which these fields are not defined. Look at the data in the strange index to identify where it is coming from.

---

<div class="post-metadata">

### Author: ![Worradorn\_Sirisang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/worradorn_sirisang/32/41290_2.png) [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)
#### Post date: [April 4, 2019, 6:37am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/4 "2019-04-04T06:37:17Z")

</div>

Hello Chrisitan,  
First of all thank you so much for reply my issue. As I see in %{logappname}-%{logapphostname}-2019.04.03. It contain data of p2c2p123.log like p2c2p123-host1.testhostclub.local-2019.04.03.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 4, 2019, 6:40am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/5 "2019-04-04T06:40:48Z")

</div>

Can you show a sample record? Does it have the fields used to create the index name parsed out?

---

<div class="post-metadata">

### Author: ![Worradorn\_Sirisang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/worradorn_sirisang/32/41290_2.png) [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)
#### Post date: [April 4, 2019, 6:56am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/6 "2019-04-04T06:56:06Z")

</div>

```
{
  "_index": "%{custom_field_logfilename}-%{custom_field_hostname}-2019.04.04-mxq1",
  "_type": "doc",
  "_id": "CxMX52kBND7ihUH8q7wL",
  "_version": 1,
  "_score": null,
  "_source": {
      "host": {
          "os": {
               "platform": "amzn",
               "version": "2018.03",
               "name": "Amazon Linux AMI",
               "family": ""
           },
          "architecture": "x86_64",
          "name": "ip-xxx-xxx-xxx-xxx",
          "containerized": true
      },
      "timestamp": "2019-03-22 12:57:57",
      "prospector": {
      "type": "log"
    },
    "@timestamp": "2019-04-04T06:44:34.596Z",
    "tags": [
      "p2c2p123",
      "beats_input_codec_plain_applied"
   ],
   "meta": {
      "cloud": {
         "provider": "ec2",
         "instance_id": "<instance_id>",
         "region": "<region>",
         "machine_type": "c5.2xlarge",
         "availability_zone": "<availability_zone>"
       }
    },
    "@version": "1",
    "message": "[2019-03-22 12:57:57] report.WARNING:",
    "input": {
       "type": "log"
    },
    "offset": 43071153,
    "beat": {
         "version": "6.6.1",
         "name": "ip-xxx-xxx-xxx-xxx",
         "hostname": "ip-xxx-xxx-xxx-xxx"
     },
     "source": "/home/ec2-user/app/current/var/log/p2c2p123.log",
     "log": {
           "file": {
               "path": "/home/ec2-user/app/current/var/log/p2c2p123.log"
            }
       },
       "CLASS": "report",
       "loglevel": "WARNING"
   },
  "fields": {
         "@timestamp": [
                 "2019-04-04T06:44:34.596Z"
          ]
    },
    "sort": [
          1554360274596
     ]
}
```

---

<div class="post-metadata">

### Author: ![Worradorn\_Sirisang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/worradorn_sirisang/32/41290_2.png) [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)
#### Post date: [April 4, 2019, 6:59am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/7 "2019-04-04T06:59:06Z")

</div>

```
{
   "_index": "p2c2p123-host1.testhostclub.local-2019.04.04-mxq1",
   "_type": "doc",
   "_id": "7Y4R52kB8zAPqq0q7qMM",
   "_version": 1,
   "_score": null,
   "_source": {
   "host": {
       "os": {
           "version": "2018.03",
          "platform": "amzn",
          "name": "Amazon Linux AMI",
          "family": ""
       },
       "architecture": "x86_64",
       "name": "ip-xxx-xxx-xxx-xxx",
       "containerized": true
      },
     "timestamp": "2019-04-04 06:38:07",
     "prospector": {
     "type": "log"
     },
     "@timestamp": "2019-04-04T06:38:08.853Z",
"tags": [
  "p2c2p123",
  "beats_input_codec_plain_applied"
],
"meta": {
  "cloud": {
    "provider": "ec2",
    "instance_id": "<instance_id>",
    "region": "<region>",
    "machine_type": "c5.2xlarge",
    "availability_zone": "<availability_zone>"
     }
  },
    "@version": "1",
    "message": "[2019-04-04 06:38:07] report.DEBUG:",
   "offset": 384412217,
   "custom_field_hostname": "host1.testhostclub.local",
    "input": {
      "type": "log"
    },
    "custom_field_logfilename": "p2c2p123",
    "log": {
       "file": {
        "path": "/home/ec2-user/app/current/var/log/p2c2p123.log"
      }
    },
     "beat": {
         "version": "6.6.1",
         "name": "ip-xxx-xxx-xxx-xxx",
        "hostname": "ip-xxx-xxx-xxx-xxx"
    },
       "source": "/home/ec2-user/app/current/var/log/p2c2p123.log",
       "CLASS": "report",
      "loglevel": "DEBUG"
   },
   "fields": {
      "@timestamp": [
      "2019-04-04T06:38:08.853Z"
    ]
 },
  "sort": [
     1554359888853
      ]
   }
```

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 4, 2019, 6:59am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/8 "2019-04-04T06:59:25Z")

</div>

That document does not have those fields defined, which is why you are seeing this issue.

It does look like this approach could result in a very large number of shards which is very inefficient and can cause problems down the line. I would recommend reading and following the guidelines outlines in [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster).

---

<div class="post-metadata">

### Author: ![Worradorn\_Sirisang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/worradorn_sirisang/32/41290_2.png) [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)
#### Post date: [April 4, 2019, 7:00am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/9 "2019-04-04T07:00:48Z")

</div>

Hello Christian,  
I update 2 sample finish

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 4, 2019, 7:05am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/10 "2019-04-04T07:05:10Z")

</div>

It looks like your Filebeat configs are not consistently setting the additional fields correctly. I would however recommend against this type of naming convention as it is likely to cause you problems with too many small shards as per the blog I linked to.

---

<div class="post-metadata">

### Author: ![Worradorn\_Sirisang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/worradorn_sirisang/32/41290_2.png) [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)
#### Post date: [April 4, 2019, 7:50am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/11 "2019-04-04T07:50:36Z")

</div>

Ok let me try 🙂

---

<div class="post-metadata">

### Author: ![Worradorn\_Sirisang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/worradorn_sirisang/32/41290_2.png) [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)
#### Post date: [April 4, 2019, 11:32am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/12 "2019-04-04T11:32:21Z")

</div>

Hello Chrisitian,  
I reduce master node from 3 to 2 node and my data node is 4 node. I still to face problem.  
as I see in logstash output (output { stdout { codec =\> rubydebug } }).

```
  I found output of my logstash does not contain
  logappname and logapphostname field 

  Please let me know how can I resolve this problem.
```

---

<div class="post-metadata">

### Author: ![Worradorn\_Sirisang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/worradorn_sirisang/32/41290_2.png) [@Worradorn\_Sirisang](https://discuss.elastic.co/u/Worradorn_Sirisang)
#### Post date: [April 5, 2019, 9:20am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/13 "2019-04-05T09:20:52Z")

</div>

Hello Chrisitian,  
I can resolve my problem by merge all file to one file.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 3, 2019, 9:20am UTC](https://discuss.elastic.co/t/please-help-logstash-can-not-parse-field-value-to-created-dynamic-index/175188/14 "2019-05-03T09:20:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.