Filebeat not creating index in ES 6.5.1

vishnujyothi · December 13, 2018, 4:59pm

Hi

Am using filebeat 6.5.1 connected to ES 6.5 , was trying/googling to find a fix to index creation problem. metricbeat and heartbeat installed in same machine are working fine (creating index and all)

ALL my ES/Kibana and beats are in same machine so configuration are pretty simple . Tried many steps given in similar issues and didn't succeeded any help would be much useful

**File beat log**

2018-12-13T16:56:16.043Z DEBUG [elasticsearch] elasticsearch/client.go:711 Ping status code: 200
2018-12-13T16:56:16.043Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.5.1
2018-12-13T16:56:16.043Z DEBUG [elasticsearch] elasticsearch/client.go:730 HEAD http://localhost:9200/_template/filebeat
2018-12-13T16:56:16.047Z INFO template/load.go:129 Template already exists and will not be overwritten.
2018-12-13T16:56:16.047Z DEBUG [modules] fileset/pipelines.go:45 Required processors:
2018-12-13T16:56:16.047Z DEBUG [elasticsearch] elasticsearch/client.go:730 GET http://localhost:9200/_ingest/pipeline/filebeat-6.5.1-logstash-log-pipeline-plain
2018-12-13T16:56:16.048Z DEBUG [modules] fileset/pipelines.go:71 Pipeline filebeat-6.5.1-logstash-log-pipeline-plain already loaded
2018-12-13T16:56:16.048Z DEBUG [modules] fileset/pipelines.go:45 Required processors:
2018-12-13T16:56:16.048Z DEBUG [elasticsearch] elasticsearch/client.go:730 GET http://localhost:9200/_ingest/pipeline/filebeat-6.5.1-logstash-slowlog-pipeline-plain

2018-12-13T16:56:16.049Z DEBUG [modules] fileset/pipelines.go:71 Pipeline filebeat-6.5.1-logstash-slowlog-pipeline-plain already loaded
2018-12-13T16:56:16.049Z DEBUG [modules] fileset/pipelines.go:45 Required processors:
2018-12-13T16:56:16.049Z DEBUG [fileset] fileset/fileset.go:220 Comparing ES version 6.5.1 with requirement of 6.1.0
2018-12-13T16:56:16.049Z DEBUG [elasticsearch] elasticsearch/client.go:730 GET http://localhost:9200/_ingest/pipeline/filebeat-6.5.1-system-syslog-pipeline
2018-12-13T16:56:16.050Z DEBUG [modules] fileset/pipelines.go:71 Pipeline filebeat-6.5.1-system-syslog-pipeline already loaded
2018-12-13T16:56:16.050Z DEBUG [modules] fileset/pipelines.go:45 Required processors:
2018-12-13T16:56:16.050Z DEBUG [fileset] fileset/fileset.go:220 Comparing ES version 6.5.1 with requirement of 6.1.0

Filebeat YML

filebeat.inputs:

    # Each - is an input. Most options can be set at the input level, so
    # you can use different inputs for various configurations.
    # Below are the input specific configurations.

    - type: log

      # Change to true to enable this input configuration.
      enabled: true

      # Paths that should be crawled and fetched. Glob based paths.
      paths:
       #- /var/log/*.log
        - /data/logs/grafana/*.log
        - /data/logs/metricbeat/*

        #- c:\programdata\elasticsearch\logs\*

    #==================== Elasticsearch template setting ==========================

    setup.template.name: "filebeat"
    setup.template.pattern: "filebeat-*"
    setup.template.overwrite: false
    setup.template.settings:
      index.number_of_shards: 1
      #index.codec: best_compression
      #_source.enabled: false

    #-------------------------- Elasticsearch output ------------------------------
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
      index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
      filebeat.registry_file: /var/lib/filebeat/registry

      # Optional protocol and basic auth credentials.
      #protocol: "https"
      #username: "elastic"
      #password: "changeme"

shaunak · December 14, 2018, 11:54pm

The filebeat-* index is created when Filebeat first tries to index a log event from one of your log files. I see you included the Filebeat log above. Is this the complete log? Are there any mentions of "Harvester" in your Filebeat log?

vishnujyothi · December 17, 2018, 9:53am

Its says Harvester for file still running

2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:494        Update existing file for harvesting: /var/log/messages, offset: 411624
2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:546        Harvester for file is still running: /var/log/messages
2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:195        input states cleaned up. Before: 1, After: 1, Pending: 0
2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:404        Check file for harvesting: /var/log/auth.log
2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:494        Update existing file for harvesting: /var/log/auth.log, offset: 0
2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:548        File didn't change: /var/log/auth.log
2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:404        Check file for harvesting: /var/log/secure
2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:494        Update existing file for harvesting: /var/log/secure, offset: 178251
2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:546        Harvester for file is still running: /var/log/secure
2018-12-17T09:51:12.920Z        DEBUG   [input] log/input.go:195        input states cleaned up. Before: 2, After: 2, Pending: 0
2018-12-17T09:51:18.401Z        DEBUG   [input] input/input.go:152      Run input
2018-12-17T09:51:18.401Z        DEBUG   [input] log/input.go:174        Start next scan
2018-12-17T09:51:18.401Z        DEBUG   [input] input/input.go:152      Run input
2018-12-17T09:51:18.401Z        DEBUG   [input] log/input.go:174        Start next scan
2018-12-17T09:51:18.401Z        DEBUG   [input] log/input.go:195        input states cleaned up. Before: 0, After: 0, Pending: 0
2018-12-17T09:51:18.401Z        DEBUG   [input] log/input.go:195        input states cleaned up. Before: 0, After: 0, Pending: 0
2018-12-17T09:51:22.920Z        DEBUG   [input] input/input.go:152      Run input
2018-12-17T09:51:22.920Z        DEBUG   [input] log/input.go:174        Start next scan
2018-12-17T09:51:22.920Z        DEBUG   [input] input/input.go:152      Run input
2018-12-17T09:51:22.920Z        DEBUG   [input] log/input.go:174        Start next scan
2018-12-17T09:51:22.920Z        DEBUG   [input] log/input.go:404        Check file for harvesting: /var/log/messages
2018-12-17T09:51:22.921Z        DEBUG   [input] log/input.go:494        Update existing file for harvesting: /var/log/messages, offset: 411624
2018-12-17T09:51:22.921Z        DEBUG   [input] log/input.go:546        Harvester for file is still running: /var/log/messages
2018-12-17T09:51:22.921Z        DEBUG   [input] log/input.go:195        input states cleaned up. Before: 1, After: 1, Pending: 0

system · January 14, 2019, 9:53am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat not able to create a ES Index Elasticsearch	2	417	October 24, 2018
Can't able to create index in ES via filebeat Beats filebeat	3	397	October 31, 2018
Filebeat - It is not creating index on Elasticsearch Beats filebeat	4	4488	July 5, 2017
Filebeat not creating index in Kibana Beats filebeat	1	481	December 14, 2018
Elasticsearch not creating index for the filebeat data Beats filebeat	2	1272	November 22, 2017

Filebeat not creating index in ES 6.5.1

Related topics