Filebeat - It is not creating index on Kibana


(Kumar) #1

Hello!

I am using ELK stack version 5.2 latest.

I'm trying to configure the FIlebeat. My configuration is Filebeat > Elasticsearch > Kibana.
However there is something wrong and I couldn't figure it out. There are some errors in the log and the index is not created on Kibana.

I have configured index template file for Filebeat.

I have deleted data by curl -XDELETE 'http://localhost:9200/filebeat-*' and configured filebeat index template by using below command, but still index issue persists.
PS C:\Program Files\Filebeat> Invoke-WebRequest -Method Put -InFile filebeat.template.json -Uri http://localhost:9200/_template/filebeat?pretty

please find http://localhost:9200/_template/filebeat?pretty data below.

{
  "filebeat" : {
    "order" : 0,
    "template" : "filebeat-*",
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_default_" : {
        "_meta" : {
          "version" : "5.2.0"
        },
        "dynamic_templates" : [
          {
            "strings_as_keyword" : {
              "mapping" : {
                "ignore_above" : 1024,
                "type" : "keyword"
              },
              "match_mapping_type" : "string"
            }
          }
        ],
        "_all" : {
          "norms" : false
        },
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "offset" : {
            "type" : "long"
          },
          "meta" : {
            "properties" : {
              "cloud" : {
                "properties" : {
                  "machine_type" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "availability_zone" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "instance_id" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "project_id" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "provider" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "region" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  }
                }
              }
            }
          },
          "beat" : {
            "properties" : {
              "hostname" : {
                "ignore_above" : 1024,
                "type" : "keyword"
              },
              "name" : {
                "ignore_above" : 1024,
                "type" : "keyword"
              },
              "version" : {
                "ignore_above" : 1024,
                "type" : "keyword"
              }
            }
          },
          "input_type" : {
            "ignore_above" : 1024,
            "type" : "keyword"
          },
          "source" : {
            "ignore_above" : 1024,
            "type" : "keyword"
          },
          "message" : {
            "norms" : false,
            "type" : "text"
          },
          "type" : {
            "ignore_above" : 1024,
            "type" : "keyword"
          },
          "tags" : {
            "ignore_above" : 1024,
            "type" : "keyword"
          }
        }
      }
    },
    "aliases" : { }
  }
}

Please help us in creating filebeat index pattern.


(Andrew Kroh) #2

What are the errors?


(Kumar) #3

I am finidng below error in filebeat logs

2017-02-09T02:27:34-06:00 DBG Check file for harvesting: D:\Team\logs\SystemOut_17.02.07_13.16.49.log
2017-02-09T02:27:34-06:00 DBG Update existing file for harvesting: D:\Team\logs\SystemOut_17.02.07_13.16.49.log, offset: 1048570
2017-02-09T02:27:34-06:00 DBG File didn't change: D:\Team\logs\SystemOut_17.02.07_13.16.49.log
2017-02-09T02:27:34-06:00 DBG Check file for harvesting: D:\Team\logs\ARSGBCGLOBAL_SecurityUpdateEventHandler_2016-12-20.log
2017-02-09T02:27:34-06:00 DBG Update existing file for harvesting: D:\Team\logs\ARSGBCGLOBAL_SecurityUpdateEventHandler_2016-12-20.log, offset: 52117551
2017-02-09T02:27:34-06:00 DBG File didn't change: D:\Team\logs\ARSGBCGLOBAL_SecurityUpdateEventHandler_2016-12-20.log
2017-02-09T02:27:34-06:00 DBG Prospector states cleaned up. Before: 42, After: 42
2017-02-09T02:27:39-06:00 DBG Flushing spooler because of timeout. Events flushed: 0
2017-02-09T02:27:44-06:00 DBG Flushing spooler because of timeout. Events flushed: 0
2017-02-09T02:27:44-06:00 DBG Run prospector
2017-02-09T02:27:44-06:00 DBG Start next scan


(Kumar) #4

I have deleted data using curl -XDELETE 'http://localhost:9200/filebeat-*'

and also configured template manually by using below command
PS C:\Program Files\Filebeat> Invoke-WebRequest -Method Put -InFile filebeat.template.json -Uri http://localhost:9200/_template/filebeat?pretty

still we could not able to index filebeat in kibana.

Before some days back we can able to and now we could not able to.


(Andrew Kroh) #5

There are no errors in the log output you posted. Perhaps no new data is coming into the file?

Deleting the filebeat-* indicies does not cause filebeat to resend all data. Filebeat stores state info on the host in a file. You can stop filebeat, delete the file, then restart filebeat to make it resend all data.


(Kumar) #6

Thank you Andrewkroh for your update. I ahve deleted old records and inserted new data so its fixed now. I can index filebeat now :slight_smile:


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.