No indices match pattern "filebeat-*"

Hello, we have installed ElasticSearch, Filebeat and Kibana 6.3.2 (No Logstash). We have them on different machines because that's how the installation was required by our boss.

When I run curl /_cat/indices?v what I get is this:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .monitoring-kibana-6-2018.08.02 FWf-86uDTFmskvajdTC-og 1 0 8373 0 2.2mb 2.2mb
green open .monitoring-es-6-2018.08.03 AZUZdg9hRsGEcanbpDL8_A 1 0 51208 36 35.4mb 35.4mb
green open .monitoring-beats-6-2018.08.02 f2BEA-iQT_aYgSWls9dO3A 1 0 274 0 214.8kb 214.8kb
green open .monitoring-kibana-6-2018.08.03 v-spNFkZRyKq8xO1USWgQA 1 0 5084 0 2.7mb 2.7mb
green open .monitoring-es-6-2018.08.02 3jDRkluHSBCcFu11GlnLhw 1 0 74671 13 27.4mb 27.4mb
green open .kibana zw_XmgksQ-K7ePuyPEyokg 1 0 153 44 255.3kb 255.3kb

I need to create filebeat indices to match my filebeat-* index pattern so I can actually read data from the path I have configured in filebeat.yml. However I've been uninstalling and reinstalling these components and I end up with the same results.

When I open kibana on the browser, I can see that the "filebeat-*" is created, however on Elasticsearch it doesn't exist (and it should exist since filebeat is pointing to it).

Can anyone help me solve this issue?

Check Filebeat logs. Does seam like Filebeat -> Elasticsearch works if you get the Kibana Dashboards setup. You do need to have some logs for Filebeat to ship to Elasticsearch for any ES indices to be created :slight_smile:

make sure you order well: curl -X GET "IP: PORT/_cat/indices?v"

@A_B
I have already ran the command to setup the kibana dashboards, also i've run setup -e multiple times, which includes kibana dashboards too.

My filebeat.yml config

#============================= Filebeat inputs ===============================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /*/*/*/*/*.logs #Masked to protect company info

#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================
setup.dashboards.enabled: true

#============================== Kibana =====================================
setup.kibana:
  host: "*.*.*.*:5601" #Masked to protect company info

#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["*.*.*.*:9200"] #Masked to protect company info
  enabled: true
  index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  hosts: ["*.*.*.*:5044"] #Masked to protect company info
  enabled: false

#================================ Logging =====================================
logging.level: debug
logging.selectors: ["*"]

@Charaf_Ahmed


comment, maybe, all this part. but otherwise it looks good to me

did filebeat start correctly, without error. can we have more information on filebeat? log or others.

It started correctly

My logs

[azadmltsudev@azeuldsdevedg-1 ~]$ sudo cat /var/log/filebeat/filebeat.5
2018-08-03T14:33:50.034Z        INFO    instance/beat.go:492    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-08-03T14:33:50.035Z        DEBUG   [beat]  instance/beat.go:519    Beat metadata path: /var/lib/filebeat/meta.json
2018-08-03T14:33:50.035Z        INFO    instance/beat.go:499    Beat UUID: 7a46e24d-bc5c-4929-886c-47b643be8b35
2018-08-03T14:33:50.035Z        INFO    [beat]  instance/beat.go:716    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "7a46e24d-bc5c-4929-886c-47b643be8b35"}}}
2018-08-03T14:33:50.035Z        INFO    [beat]  instance/beat.go:725    Build info      {"system_info": {"build": {"commit": "45a9a9e1561b6c540e94211ebe03d18abcacae55", "libbeat": "6.3.2", "time": "2018-07-20T04:18:19.000Z", "version": "6.3.2"}}}
2018-08-03T14:33:50.035Z        INFO    [beat]  instance/beat.go:728    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":32,"version":"go1.9.4"}}}
2018-08-03T14:33:50.041Z        INFO    [beat]  instance/beat.go:732    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-08-03T11:27:06Z","containerized":true,"hostname":"azeuldsdevedg-1.spartamotor.dev","ips":["127.0.0.1/8","::1/128","*.*.*.*/24","fe80::20d:3aff:fe17:d1bc/64"],"kernel_version":"3.10.0-862.3.3.el7.x86_64","mac_addresses":["00:0d:3a:17:d1:bc","00:0d:3a:17:d1:bc"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":5,"patch":1804,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"6aaed308aa5a419f880c5e45eea65414"}}}
2018-08-03T14:33:50.041Z        INFO    [beat]  instance/beat.go:761    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 67322, "ppid": 1, "seccomp": {"mode":"disabled"}, "start_time": "2018-08-03T14:33:49.850Z"}}}

Logs continue like this (Made previous post too long):

2018-08-03T14:33:50.041Z        INFO    instance/beat.go:225    Setup Beat: filebeat; Version: 6.3.2
2018-08-03T14:33:50.041Z        DEBUG   [beat]  instance/beat.go:242    Initializing output plugins
2018-08-03T14:33:50.041Z        DEBUG   [processors]    processors/processor.go:49      Processors:
2018-08-03T14:33:50.042Z        INFO    elasticsearch/client.go:145     Elasticsearch url: http://*.*.*.*:9200
2018-08-03T14:33:50.042Z        DEBUG   [publish]       pipeline/consumer.go:120        start pipeline event consumer
2018-08-03T14:33:50.042Z        INFO    pipeline/module.go:81   Beat name: azeuldsdevedg-1.spartamotor.dev
2018-08-03T14:33:50.043Z        INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-08-03T14:33:50.043Z        INFO    elasticsearch/client.go:145     Elasticsearch url: http://*.*.*.*:9200
2018-08-03T14:33:50.043Z        DEBUG   [elasticsearch] elasticsearch/client.go:666     ES Ping(url=http://*.*.*.*:9200)
2018-08-03T14:33:50.049Z        DEBUG   [elasticsearch] elasticsearch/client.go:689     Ping status code: 200
2018-08-03T14:33:50.049Z        INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.3.2
2018-08-03T14:33:50.049Z        DEBUG   [dashboards]    dashboards/es_loader.go:309     Initialize the Elasticsearch 6.3.2 loader
2018-08-03T14:33:50.049Z        DEBUG   [dashboards]    dashboards/es_loader.go:309     Elasticsearch URL http://*.*.*.*:9200
2018-08-03T14:33:50.049Z        INFO    kibana/client.go:90     Kibana url: http://*.*.*.*:5601
2018-08-03T14:33:50.068Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Initialize the Kibana 6.3.2 loader
2018-08-03T14:33:50.068Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Kibana URL http://*.*.*.*:5601
2018-08-03T14:33:50.068Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Importing directory /usr/share/filebeat/kibana/6
2018-08-03T14:33:50.068Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import directory /usr/share/filebeat/kibana/6
2018-08-03T14:33:50.068Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import index-pattern from /usr/share/filebeat/kibana/6/index-pattern/filebeat.json
2018-08-03T14:33:50.579Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import directory /usr/share/filebeat/kibana/6
2018-08-03T14:33:50.580Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-Kafka-overview.json
2018-08-03T14:33:51.672Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-Mongodb-overview.json
2018-08-03T14:33:52.731Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-Postgresql-overview.json
2018-08-03T14:33:53.714Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-Postgresql-slowlogs.json
2018-08-03T14:33:54.810Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-apache2.json
2018-08-03T14:33:55.786Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-auditd.json
2018-08-03T14:33:56.796Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-auth-sudo-commands.json
2018-08-03T14:33:57.814Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-icinga-debug-log.json
2018-08-03T14:33:58.915Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-icinga-main-log.json
2018-08-03T14:33:59.950Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-icinga-startup-errors.json
2018-08-03T14:34:00.968Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-iis.json
2018-08-03T14:34:02.333Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-logstash-log.json
2018-08-03T14:34:03.365Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-logstash-slowlog.json
2018-08-03T14:34:04.353Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-mysql.json
2018-08-03T14:34:05.394Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-new-users-and-groups.json
2018-08-03T14:34:06.411Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-nginx-logs.json
2018-08-03T14:34:07.428Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-nginx-overview.json
2018-08-03T14:34:08.450Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-redis.json
2018-08-03T14:34:09.481Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-ssh-login-attempts.json
2018-08-03T14:34:10.515Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-syslog.json
2018-08-03T14:34:11.516Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/Filebeat-traefik-overview.json
2018-08-03T14:34:12.726Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/ml-nginx-access-remote-ip-count-explorer.json
2018-08-03T14:34:13.755Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/ml-nginx-remote-ip-url-explorer.json
2018-08-03T14:34:14.954Z        DEBUG   [dashboards]    dashboards/kibana_loader.go:121 Import dashboard from /usr/share/filebeat/kibana/6/dashboard/ml-traefik-access-remote-ip-count-explorer.json

Then this thing keeps repeating:

2018-08-03T14:34:19.061Z        INFO    instance/beat.go:607    Kibana dashboards successfully loaded.
2018-08-03T14:34:19.061Z        INFO    instance/beat.go:315    filebeat start running.
2018-08-03T14:34:19.061Z        DEBUG   [registrar]     registrar/registrar.go:97       Registry file set to: /var/lib/filebeat/registry
2018-08-03T14:34:19.061Z        INFO    registrar/registrar.go:117      Loading registrar data from /var/lib/filebeat/registry
2018-08-03T14:34:19.061Z        INFO    registrar/registrar.go:124      States Loaded from registrar: 9
2018-08-03T14:34:19.061Z        INFO    crawler/crawler.go:48   Loading Inputs: 1
2018-08-03T14:34:19.061Z        DEBUG   [registrar]     registrar/registrar.go:250      Starting Registrar
2018-08-03T14:34:19.061Z        DEBUG   [processors]    processors/processor.go:49      Processors:
2018-08-03T14:34:19.062Z        DEBUG   [input] log/config.go:178       recursive glob enabled
2018-08-03T14:34:19.062Z        DEBUG   [input] log/input.go:127        exclude_files: []. Number of stats: 9
2018-08-03T14:34:19.062Z        DEBUG   [input] log/input.go:148        input with previous states loaded: 0
2018-08-03T14:34:19.062Z        INFO    log/input.go:118        Configured paths: [/home/azadmltsudev/logs/T0/*.logs]
2018-08-03T14:34:19.062Z        INFO    input/input.go:88       Starting input of type: log; ID: 5963880881236787260
2018-08-03T14:34:19.062Z        DEBUG   [cfgfile]       cfgfile/reload.go:90    Checking module configs from: /etc/filebeat/modules.d/*.yml
2018-08-03T14:34:19.062Z        DEBUG   [input] log/input.go:154        Start next scan
2018-08-03T14:34:19.062Z        DEBUG   [cfgfile]       cfgfile/reload.go:104   Number of module configs found: 0
2018-08-03T14:34:19.062Z        INFO    crawler/crawler.go:82   Loading and starting Inputs completed. Enabled inputs: 1
2018-08-03T14:34:19.062Z        INFO    cfgfile/reload.go:122   Config reloader started
2018-08-03T14:34:19.062Z        DEBUG   [input] log/input.go:175        input states cleaned up. Before: 0, After: 0, Pending: 0
2018-08-03T14:34:19.062Z        DEBUG   [cfgfile]       cfgfile/reload.go:146   Scan for new config files
2018-08-03T14:34:19.062Z        DEBUG   [cfgfile]       cfgfile/reload.go:165   Number of module configs found: 0
2018-08-03T14:34:19.062Z        INFO    cfgfile/reload.go:214   Loading of config files completed.
2018-08-03T14:34:20.045Z        INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":44}},"total":{"ticks":120,"time":{"ms":131},"value":120},"user":{"ticks":80,"time":{"ms":87}}},"info":{"ephemeral_id":"d48741f4-ac12-48e0-8b7e-ae84bab9c1ce","uptime":{"ms":30018}},"memstats":{"gc_next":4194304,"memory_alloc":3046672,"memory_total":10581016,"rss":18522112}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":32},"load":{"1":0.28,"15":0.23,"5":0.2,"norm":{"1":0.0088,"15":0.0072,"5":0.0063}}}}}}
2018-08-03T14:34:29.062Z        DEBUG   [input] input/input.go:125      Run input
2018-08-03T14:34:29.062Z        DEBUG   [input] log/input.go:154        Start next scan
2018-08-03T14:34:29.063Z        DEBUG   [input] log/input.go:175        input states cleaned up. Before: 0, After: 0, Pending: 0
2018-08-03T14:34:39.063Z        DEBUG   [input] input/input.go:125      Run input
2018-08-03T14:34:39.063Z        DEBUG   [input] log/input.go:154        Start next scan
2018-08-03T14:34:39.063Z        DEBUG   [input] log/input.go:175        input states cleaned up. Before: 0, After: 0, Pending: 0
2018-08-03T14:34:49.063Z        DEBUG   [input] input/input.go:125      Run input
2018-08-03T14:34:49.064Z        DEBUG   [input] log/input.go:154        Start next scan
2018-08-03T14:34:49.064Z        DEBUG   [input] log/input.go:175        input states cleaned up. Before: 0, After: 0, Pending: 0

@Charaf_Ahmed

So yeah, I see nothing wrong in those logs

Oh, and btw I just did this too as sugested

Is this line from the log important? Number of module configs found: 0

What does sudo filebeat modules list return?

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["*.*.*.*:9200"] #Masked to protect company info
  enabled: true
  index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  hosts: ["*.*.*.*:5044"] #Masked to protect company info
  enabled: false

#================================ Logging =====================================

The output section looks a bit odd to me. output.logstash: is commented out but hosts and enabled is not. This would mean that for output.elasticsearch you have two contradicting hosts and enabled configurations. I would remove the Logstash output section completely. It is not possible (as far as I know) to ship logs to both Logstash and Elasticsearch at the same time from one Filebeat instance anyway...

@tdasch

This is my modules list:

image

@A_B

I already commented out the whole "output.logstash" section, meanining I commented the "hosts" and "enabled" part under it.

Me and my team are thinking there just maybe something wrong with version 6.3.2. Maybe I should uninstall and try with 6.0.1.

The only thing left that I can think of is that Filebeat does not have anything to ship.

Here's an example of one of my input configs

- type: log
  paths:
    - /var/log/auth.log
    - /var/log/faillog
    - /var/log/messages
    - /var/log/syslog
  encoding: plain
  fields:
    log_prefix: dc
    log_idx: syslog-beat
  fields_under_root: false
  document_type: syslog-beat
  scan_frequency: 10s
  harvester_buffer_size: 16384
  max_bytes: 10485760

I'm still running the old style (deprecated) config

/etc/filebeat # grep -B 2 conf.d filebeat.yml
  config.prospectors:
    enabled: true
    path: /etc/filebeat/conf.d/*.yml

I would double check that the path to the logs are correct and that Filebeat is allowed to read them.

@A_B

I just saw an error in my config, my path was pointing to a ".logs" when it was supposed to be ".log".

My file permissions are as follows:

So, it's more than whether or not it can read log files, it just won't.
And even though my "filebeat-*" index is automatically created with "filebeat setup -e", it still won't match anything in elasticsearch, which is the main issue here.

This is currently my curl to _cat/indices

So, even after you fixed the log path, you have no filebeat-* index?

filebeat-* is not created by filebeat setup -e. That sets up the Kibana saved Queries and Dashboards, as far as I know anyway.

The filebeat-* indices will be created by Filebeat when there are some logs to ship to Elasticsearch.

The index is created in kibana, but without any indices to match in elasticsearch, that index doesn't work