CLOSE THIS TOPIC - Failed to connect: 404 Not Found - Indexes are not being created

Hello
I installed ES Kibana and MB 6.2, started ES and started kibana so far, I can access KB dashboard, so fa so good , then I installed metricbeat with the following configuration

metricbeat.config.modules:

  path: ${path.config}/conf.d/*.yml
  reload.period: 60s
  reload.enabled: true
metricbeat.max_start_delay: 10

metricbeat.modules:
- module: system
  metricsets:
   - cpu             # CPU usage
   - memory          # Memory usage
- cpu             # CPU usage
     - filesystem      # File system usage for each mountpoint
     - fsstat          # File system summary metrics
     - load            # CPU load averages
     - memory          # Memory usage
    
     - process         # Per process metrics
  
     - uptime          # System Uptime
  
     - diskio         # Disk IO

  enabled: true
  period: 60s
  processes: ['.*']

name: Kibana
fields:
  env: aws-produciton
output.elasticsearch:
  enabled: true
  hosts: ["10.87.22.110:9200"]
  enabled: true
  path: "/var/log/metricbeat"
  filename: metricbeat.log
  rotate_every_kb: 10000
  number_of_files: 7

setup.template.settings: ( this seems to be know, no idea what it does )
setup.kibana:

  host: "10.87.22.110:5601"

logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat.log
  keepfiles: 7

In the logs Im getting

2018-03-12T14:45:26.818Z	ERROR	pipeline/output.go:74	Failed to connect: 404 Not Found: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"},"status":404}
2018-03-12T14:45:53.533Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":43},"total":{"ticks":100,"time":104,"value":100},"user":{"ticks":60,"time":61}},"info":{"ephemeral_id":"a2e59cb9-cde4-49d9-a017-49185daff66a","uptime":{"ms":60032}},"memstats":{"gc_next":6037568,"memory_alloc":3022080,"memory_total":11193784,"rss":258048}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":273},"write":{"bytes":142}},"pipeline":{"clients":1,"events":{"active":86}}},"system":{"load":{"1":0.02,"15":0.1,"5":0.05,"norm":{"1":0.01,"15":0.05,"5":0.025}}}}}}
2018-03-12T14:45:58.821Z	ERROR	pipeline/output.go:74	Failed to connect: 404 Not Found: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"},"status":404}

I do not see any indices being created either

curl  -v  'http://kibana.domain.lan:9200/_cat/indices?v'
* About to connect() to kibana.domain.lan port 9200 (#0)
*   Trying 10.87.22.110...
* Connected to kibana.domain.lan (10.87.22.110) port 9200 (#0)
> GET /_cat/indices?v HTTP/1.1
> User-Agent: curl/7.29.0
> Host: kibana.domain.lan:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/plain; charset=UTF-8
< content-length: 83
<
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
* Connection #0 to host kibana.domain.lan left intact

Any idea what's going on?

Thanks
Regards

It looks like the index doesn't exist and Metricbeat cannot create it, did you disable automatic index creation in Elasticsearch? https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docs-index_.html#index-creation

Best regards

Hello, thanks fort your reply

I added action.auto_create_index: true to elasticsearch.yml and restarted the service and still seeing the same issue, should I use the API instead the configuration file to set this value?

Thanks
Regards

These are the valued defined in the configuration file

[root@kibana elasticsearch]# grep -v '#' elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.87.22.110
action.auto_create_index: true

Thanks
Regards

I can see there are no templates loaded, could this be for the same reason that the index is not created?

And from kibana logs I see this

["warning","elasticsearch","admin"],"pid":21893,"message":"Unable to revive connection: http://kibana.domain.lan:9200/"}

Thanks

curl  -XGET   'http://kibana.domain.lan:9200/_search?pretty'  -v
* About to connect() to kibana.domain.lan port 9200 (#0)
*   Trying 10.87.22.110...
* Connected to kibana.domain.lan (10.87.22.110) port 9200 (#0)
> GET /_search?pretty HTTP/1.1
> User-Agent: curl/7.29.0
> Host: kibana.domain.lan:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 211
<
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

I also added to elasticsearch.yml the following.

node.master: true
node.data: true

Any idea what could be wrong?

Thanks for your time and support

Moved this issue to elasticsearch cuz I think is not a metricbeat issue.
I enabled debug I only see this error:

[2018-03-15T15:39:12,890][DEBUG][r.suppressed             ] path: /.kibana/_mappings, params: {index=.kibana}
org.elasticsearch.index.IndexNotFoundException: no such index
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:678) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:630) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:586) ~[elasticsearch-6.2.2.jar:6.2.2]

And metricbeat still complains about the index

2018-03-15T15:39:57.155Z ERROR pipeline/output.go:74 Failed to connect: 404 Not Found: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"},"status":404}

By the way SELINUX is disabled

getenforce
Disabled

As there was no more feedback please close this topic, I have no idea what's the problem with 6.2 that indexes are not being created I installed 5.6.8 and is working as expected.

Thanks
Regards

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.