Kibana does not create a .kibana

Hello everybody,
Sorry to bother any of you with my basic question but i have a problem i cant go around anyway as all the solution i found online does not solve my problem.

I am a junior DevOps i have been working on my own on a High availability project based on amazon aws. I came to the point that i want to implement Elasticsearch,logstash and Kibana to visualise my data.

First thing first, i used chef as configuration tool to install all the required software and made sure that Elasticsearch and Kibana are on the same version. Than with packer and terraform i spin up my instances. The grate news is that from kibana ai can ping my elasticsearch and i dont have any problem to access kibana through the publicIP:5601.

When i access to kibana, as first message i am told that i dont have any index template loaded. and they ask me to create one. When i try to create a templare index, i cant find .Kibana...but only .Kibana_1 and Kibana_system

I am running elastic search,kibana and logstash on 3 different instances. and my configuration files for kibana are this

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address. ""

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes. "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: [""]

# When this setting's value is true Kibana uses the hostname specified in the
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana"
elasticsearch.password: "123"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/

# Enables you specify a file where Kibana stores log output.
logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

In my log i dont have any error at all, and in kibana i specified i want to create a .kibana but it just does not happen.
i am running the instances in ubuntu 18.04
I apology for the basic questions guys.

You shouldn't need to specify that as it's a default. However even if you don't have errors, can you post the logs from Kibana?

You are right, but even if i keep it as default it does not create it.

if i run `journalctl -u kibana'

i get this

-- Logs begin at Mon 2020-05-04 00:38:29 UTC, end at Mon 2020-05-04 01:38:42 UTC. --
May 04 00:40:35 ip-172-31-27-227 systemd[1]: Started Kibana.
May 04 00:40:37 ip-172-31-27-227 systemd[1]: Stopping Kibana...
May 04 00:40:37 ip-172-31-27-227 systemd[1]: Stopped Kibana.
-- Reboot --
May 04 01:38:20 ip-10-0-105-101 systemd[1]: Started Kibana.
May 04 01:38:42 ip-10-0-105-101 kibana[834]: {"type":"log","@timestamp":"2020-05-04T01:38:42Z","tags":["info","plugins-service"],"pid":834,"message":"Plugin \"case\" is disabled."}

Can you take a look at the _cat/indices API and post that output?

if i runt GET /_cat/indices nothing get printed.

Thank you very much for your time and help

now i get this

GET _cat/indices
Can't connect to _cat:80 (Temporary failure in name resolution)

Temporary failure in name resolution at /usr/share/perl5/LWP/Protocol/ line 50.

If you're running something from a shell, you need to use curl -H "Content-Type: application/json" host:9200/_cat/indices.

Still the same, i dont get anything back

sorry for the second text. I realise that i was wrong, because my security group were set wrong

this is the curl output:

curl -H "Content-Type: application/json"
green open .kibana_task_manager_1   zQi7CN_1TiCMb-hOD7Bvlg 1 0 2 6 33.5kb 33.5kb
green open .apm-agent-configuration Rrii0ZbYQKCJvIQ2m9ju6A 1 0 0 0   230b   230b
green open .kibana_1                LKeGfxxlRwG4PYQnhAvNwg 1 0 4 1 26.9kb 26.9kb

That is what you want. There should also be an alias to it, which you can check with _cat/aliases.

Earlier on kibana I saw that the aliases for. Kibana_1 it's. Kibana.
But when I select. Kibana_1 as index template.. I does not show any data

Why are you wanting to create an index template for that though?

My idea is this.
I have a Web application (which display only a home page" running on a separate instance.
I want to use kibana to display that instance if it's up or down, check the logs etc.
And the guides I found they all say to create a template index. Am I wrong?

Ah ok, we should have started there :slight_smile:

Yes, you can do that with the Elastic Stack. I am not sure what guides you are following, but you will need to use something like Heartbeat to poll the site, collect the response, then send it to Elasticsearch. Once that is done then you can visualise it in Kibana.

But the Kibana index pattern needs to point to the Heartbeat data. The .kibana index is a system one, it is used to store Kibana settings in there, not data collected from something like Heartbeat. So you will never be able to (meaningfully) visualise anything inside that index :slight_smile:

Oh ok. That make sense. For the heartbeat is something that I can run in a console. Any guide that you can suggest me to do so?
Thank you very much your help is a good place to get started.

One more thing.
I have a filebeat provisioned in my instance but it does not seems to work as expected.
For example, if I try to run the command . /filebeat modules enable nginx
It just tells me that filebeat does not exist, even if it does

I followed that guide, and I had the problem above mentioned

If you are running that exact command, you have a space between the dot and the forward slash. It should be ./filebeat modules enable nginx

Hi warklom,
I am sorry for earlier, and thank you again one more time for your patience.

I got some good progress. I realise that my kibana was not working because of an error during the downloads of my cookbooks. Now if i access my kibana, i can see my filebeat and metric beat.

I am just confused about one thing. As said before, i have my app running on a totally different instance but in the same VPC aws. when i log in my kibana, the only instance i see running, is the Kibana instance, how can i see my app instance? all the file i modified (kibana,filebeat,metricbeat, elasticsearch) are all related, so i can put just specific IPs for kibana etc, how can i check my app instance.

I hope my question is clear.