Filebeat - It is not creating index on Kibana

Hello!

I am using ELK stack version 5.2 latest.

I'm trying to configure the FIlebeat. My configuration is Filebeat > Elasticsearch > Kibana.
However there is something wrong and I couldn't figure it out. There are some errors in the log and the index is not created on Kibana.

I have configured index template file for Filebeat.

I have deleted data by curl -XDELETE 'http://localhost:9200/filebeat-*' and configured filebeat index template by using below command, but still index issue persists.
PS C:\Program Files\Filebeat> Invoke-WebRequest -Method Put -InFile filebeat.template.json -Uri http://localhost:9200/_template/filebeat?pretty

please find http://localhost:9200/_template/filebeat?pretty data below.

{
  "filebeat" : {
    "order" : 0,
    "template" : "filebeat-*",
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_default_" : {
        "_meta" : {
          "version" : "5.2.0"
        },
        "dynamic_templates" : [
          {
            "strings_as_keyword" : {
              "mapping" : {
                "ignore_above" : 1024,
                "type" : "keyword"
              },
              "match_mapping_type" : "string"
            }
          }
        ],
        "_all" : {
          "norms" : false
        },
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "offset" : {
            "type" : "long"
          },
          "meta" : {
            "properties" : {
              "cloud" : {
                "properties" : {
                  "machine_type" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "availability_zone" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "instance_id" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "project_id" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "provider" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "region" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  }
                }
              }
            }
          },
          "beat" : {
            "properties" : {
              "hostname" : {
                "ignore_above" : 1024,
                "type" : "keyword"
              },
              "name" : {
                "ignore_above" : 1024,
                "type" : "keyword"
              },
              "version" : {
                "ignore_above" : 1024,
                "type" : "keyword"
              }
            }
          },
          "input_type" : {
            "ignore_above" : 1024,
            "type" : "keyword"
          },
          "source" : {
            "ignore_above" : 1024,
            "type" : "keyword"
          },
          "message" : {
            "norms" : false,
            "type" : "text"
          },
          "type" : {
            "ignore_above" : 1024,
            "type" : "keyword"
          },
          "tags" : {
            "ignore_above" : 1024,
            "type" : "keyword"
          }
        }
      }
    },
    "aliases" : { }
  }
}

Please help us in creating filebeat index pattern.

What are the errors?

I am finidng below error in filebeat logs

2017-02-09T02:27:34-06:00 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.02.07_13.16.49.log
2017-02-09T02:27:34-06:00 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.02.07_13.16.49.log, offset: 1048570
2017-02-09T02:27:34-06:00 DBG File didn't change: D:\Team\logs\SystemOut_17.02.07_13.16.49.log
2017-02-09T02:27:34-06:00 DBG Check file for harvesting: D:\Team\logs\ARSGBCGLOBAL_SecurityUpdateEventHandler_2016-12-20.log
2017-02-09T02:27:34-06:00 DBG Update existing file for harvesting: D:\Team\logs\ARSGBCGLOBAL_SecurityUpdateEventHandler_2016-12-20.log, offset: 52117551
2017-02-09T02:27:34-06:00 DBG File didn't change: D:\Team\logs\ARSGBCGLOBAL_SecurityUpdateEventHandler_2016-12-20.log
2017-02-09T02:27:34-06:00 DBG Prospector states cleaned up. Before: 42, After: 42
2017-02-09T02:27:39-06:00 DBG Flushing spooler because of timeout. Events flushed: 0
2017-02-09T02:27:44-06:00 DBG Flushing spooler because of timeout. Events flushed: 0
2017-02-09T02:27:44-06:00 DBG Run prospector
2017-02-09T02:27:44-06:00 DBG Start next scan

I have deleted data using curl -XDELETE 'http://localhost:9200/filebeat-*'

and also configured template manually by using below command
PS C:\Program Files\Filebeat> Invoke-WebRequest -Method Put -InFile filebeat.template.json -Uri http://localhost:9200/_template/filebeat?pretty

still we could not able to index filebeat in kibana.

Before some days back we can able to and now we could not able to.

There are no errors in the log output you posted. Perhaps no new data is coming into the file?

Deleting the filebeat-* indicies does not cause filebeat to resend all data. Filebeat stores state info on the host in a file. You can stop filebeat, delete the file, then restart filebeat to make it resend all data.

Thank you Andrewkroh for your update. I ahve deleted old records and inserted new data so its fixed now. I can index filebeat now :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.