Please help, Logstash can not parse field value to created dynamic index

Worradorn_Sirisang · April 3, 2019, 12:08pm

Hello team,
I face a problem about logstash send data to missing elasticsearch index.

my architecture is outlined below.
Filebeats -> Logstash -> Elasticsearch/AWS Elasticsearch

my machine have Filebeat agent for collecting application logs and send to Logstash. I add logappname and logapphostname as custom fields in "fields" in Filebeat configuration file.

- type: log
  enabled: true
  paths:
     - /home/user/app/current/var/log/p2c2p123.log
  multiline.pattern: '^\[[0-9]{4}-{0-9}{2}-{0-9}{2}\ [0-9]{2}:[0-9]{2}:[0-9]{2}\]'
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: '\[\]'
  fields:
       logappname: p2c2p123
       logapphostname: host1.testhostclub.local
  fields_under_root: true

on my Logstash, I match timestamp in message. my code is outlined below.

filter {
    grok {
           match =&gt; { "message" =&gt; "\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}(:%{WORD:logtype}:)?%{SPACE}%{WORD:CLASS}.%{WORD:loglevel}:%{DATA}" }
              }     
}

And I have another configuration file as "send-to-es.conf" for setting up Logstash output. my code is outlined below.

output {
      amazon_es{
             hosts =>  ["<ES Host>"]
             region => "ap-southeast-1"
             index =>  "%{logappname}-%{logapphostname}-%{+YYYY.MM.dd}"
      }
      #stdout { codec => ruby-debug }
 }

----> in this point

on my elasticsearch, I see my data in the p2c2p123-host1.testhostclub.local-2019.04.03
and some time, I see my data store in %{logappname}-%{logapphostname}-2019.04.03

I does not understand why this %{logappname}-%{logapphostname}-2019.04.03 is occur on my ES
Please you help me to resolve my problem.

Regards

Worradorn_Sirisang · April 4, 2019, 3:33am

any update?

Christian_Dahlqvist · April 4, 2019, 5:20am

This typically mean that you have data coming in for which these fields are not defined. Look at the data in the strange index to identify where it is coming from.

Worradorn_Sirisang · April 4, 2019, 6:37am

Hello Chrisitan,
First of all thank you so much for reply my issue. As I see in %{logappname}-%{logapphostname}-2019.04.03. It contain data of p2c2p123.log like p2c2p123-host1.testhostclub.local-2019.04.03.

Christian_Dahlqvist · April 4, 2019, 6:40am

Can you show a sample record? Does it have the fields used to create the index name parsed out?

Worradorn_Sirisang · April 4, 2019, 6:56am

{
  "_index": "%{custom_field_logfilename}-%{custom_field_hostname}-2019.04.04-mxq1",
  "_type": "doc",
  "_id": "CxMX52kBND7ihUH8q7wL",
  "_version": 1,
  "_score": null,
  "_source": {
      "host": {
          "os": {
               "platform": "amzn",
               "version": "2018.03",
               "name": "Amazon Linux AMI",
               "family": ""
           },
          "architecture": "x86_64",
          "name": "ip-xxx-xxx-xxx-xxx",
          "containerized": true
      },
      "timestamp": "2019-03-22 12:57:57",
      "prospector": {
      "type": "log"
    },
    "@timestamp": "2019-04-04T06:44:34.596Z",
    "tags": [
      "p2c2p123",
      "beats_input_codec_plain_applied"
   ],
   "meta": {
      "cloud": {
         "provider": "ec2",
         "instance_id": "<instance_id>",
         "region": "<region>",
         "machine_type": "c5.2xlarge",
         "availability_zone": "<availability_zone>"
       }
    },
    "@version": "1",
    "message": "[2019-03-22 12:57:57] report.WARNING:",
    "input": {
       "type": "log"
    },
    "offset": 43071153,
    "beat": {
         "version": "6.6.1",
         "name": "ip-xxx-xxx-xxx-xxx",
         "hostname": "ip-xxx-xxx-xxx-xxx"
     },
     "source": "/home/ec2-user/app/current/var/log/p2c2p123.log",
     "log": {
           "file": {
               "path": "/home/ec2-user/app/current/var/log/p2c2p123.log"
            }
       },
       "CLASS": "report",
       "loglevel": "WARNING"
   },
  "fields": {
         "@timestamp": [
                 "2019-04-04T06:44:34.596Z"
          ]
    },
    "sort": [
          1554360274596
     ]
}

Worradorn_Sirisang · April 4, 2019, 6:59am

{
   "_index": "p2c2p123-host1.testhostclub.local-2019.04.04-mxq1",
   "_type": "doc",
   "_id": "7Y4R52kB8zAPqq0q7qMM",
   "_version": 1,
   "_score": null,
   "_source": {
   "host": {
       "os": {
           "version": "2018.03",
          "platform": "amzn",
          "name": "Amazon Linux AMI",
          "family": ""
       },
       "architecture": "x86_64",
       "name": "ip-xxx-xxx-xxx-xxx",
       "containerized": true
      },
     "timestamp": "2019-04-04 06:38:07",
     "prospector": {
     "type": "log"
     },
     "@timestamp": "2019-04-04T06:38:08.853Z",
"tags": [
  "p2c2p123",
  "beats_input_codec_plain_applied"
],
"meta": {
  "cloud": {
    "provider": "ec2",
    "instance_id": "<instance_id>",
    "region": "<region>",
    "machine_type": "c5.2xlarge",
    "availability_zone": "<availability_zone>"
     }
  },
    "@version": "1",
    "message": "[2019-04-04 06:38:07] report.DEBUG:",
   "offset": 384412217,
   "custom_field_hostname": "host1.testhostclub.local",
    "input": {
      "type": "log"
    },
    "custom_field_logfilename": "p2c2p123",
    "log": {
       "file": {
        "path": "/home/ec2-user/app/current/var/log/p2c2p123.log"
      }
    },
     "beat": {
         "version": "6.6.1",
         "name": "ip-xxx-xxx-xxx-xxx",
        "hostname": "ip-xxx-xxx-xxx-xxx"
    },
       "source": "/home/ec2-user/app/current/var/log/p2c2p123.log",
       "CLASS": "report",
      "loglevel": "DEBUG"
   },
   "fields": {
      "@timestamp": [
      "2019-04-04T06:38:08.853Z"
    ]
 },
  "sort": [
     1554359888853
      ]
   }

Christian_Dahlqvist · April 4, 2019, 6:59am

That document does not have those fields defined, which is why you are seeing this issue.

It does look like this approach could result in a very large number of shards which is very inefficient and can cause problems down the line. I would recommend reading and following the guidelines outlines in this blog post.

Worradorn_Sirisang · April 4, 2019, 7:00am

Hello Christian,
I update 2 sample finish

Christian_Dahlqvist · April 4, 2019, 7:05am

It looks like your Filebeat configs are not consistently setting the additional fields correctly. I would however recommend against this type of naming convention as it is likely to cause you problems with too many small shards as per the blog I linked to.

Worradorn_Sirisang · April 4, 2019, 7:50am

Ok let me try

Worradorn_Sirisang · April 4, 2019, 11:32am

Hello Chrisitian,
I reduce master node from 3 to 2 node and my data node is 4 node. I still to face problem.
as I see in logstash output (output { stdout { codec => rubydebug } }).

  I found output of my logstash does not contain
  logappname and logapphostname field 

  Please let me know how can I resolve this problem.

Worradorn_Sirisang · April 5, 2019, 9:20am

Hello Chrisitian,
I can resolve my problem by merge all file to one file.

system · May 3, 2019, 9:20am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to use filebeat fields name value in logstash config Beats filebeat	5	14625	May 2, 2017
Missing fields Filebeat -> Logstash -> Elasticsearch Logstash	3	1628	October 27, 2017
Logstash match add_field using dynamic field Logstash	1	304	April 15, 2020
Unable to create dynamic index with conditional check in logstash Logstash	3	1019	October 19, 2018
Can't fix _grokparsefailure Logstash	6	471	January 9, 2019

Please help, Logstash can not parse field value to created dynamic index

Related topics