CLOSE THIS TOPIC - Failed to connect: 404 Not Found - Indexes are not being created

Kernel_Panic · March 12, 2018, 2:48pm

Hello
I installed ES Kibana and MB 6.2, started ES and started kibana so far, I can access KB dashboard, so fa so good , then I installed metricbeat with the following configuration

metricbeat.config.modules:

  path: ${path.config}/conf.d/*.yml
  reload.period: 60s
  reload.enabled: true
metricbeat.max_start_delay: 10

metricbeat.modules:
- module: system
  metricsets:
   - cpu             # CPU usage
   - memory          # Memory usage
- cpu             # CPU usage
     - filesystem      # File system usage for each mountpoint
     - fsstat          # File system summary metrics
     - load            # CPU load averages
     - memory          # Memory usage
    
     - process         # Per process metrics
  
     - uptime          # System Uptime
  
     - diskio         # Disk IO

  enabled: true
  period: 60s
  processes: ['.*']

name: Kibana
fields:
  env: aws-produciton
output.elasticsearch:
  enabled: true
  hosts: ["10.87.22.110:9200"]
  enabled: true
  path: "/var/log/metricbeat"
  filename: metricbeat.log
  rotate_every_kb: 10000
  number_of_files: 7

setup.template.settings: ( this seems to be know, no idea what it does )
setup.kibana:

  host: "10.87.22.110:5601"

logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat.log
  keepfiles: 7

In the logs Im getting

2018-03-12T14:45:26.818Z	ERROR	pipeline/output.go:74	Failed to connect: 404 Not Found: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"},"status":404}
2018-03-12T14:45:53.533Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":43},"total":{"ticks":100,"time":104,"value":100},"user":{"ticks":60,"time":61}},"info":{"ephemeral_id":"a2e59cb9-cde4-49d9-a017-49185daff66a","uptime":{"ms":60032}},"memstats":{"gc_next":6037568,"memory_alloc":3022080,"memory_total":11193784,"rss":258048}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":273},"write":{"bytes":142}},"pipeline":{"clients":1,"events":{"active":86}}},"system":{"load":{"1":0.02,"15":0.1,"5":0.05,"norm":{"1":0.01,"15":0.05,"5":0.025}}}}}}
2018-03-12T14:45:58.821Z	ERROR	pipeline/output.go:74	Failed to connect: 404 Not Found: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"},"status":404}

I do not see any indices being created either

curl  -v  'http://kibana.domain.lan:9200/_cat/indices?v'
* About to connect() to kibana.domain.lan port 9200 (#0)
*   Trying 10.87.22.110...
* Connected to kibana.domain.lan (10.87.22.110) port 9200 (#0)
> GET /_cat/indices?v HTTP/1.1
> User-Agent: curl/7.29.0
> Host: kibana.domain.lan:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/plain; charset=UTF-8
< content-length: 83
<
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
* Connection #0 to host kibana.domain.lan left intact

Any idea what's going on?

Thanks
Regards

exekias · March 13, 2018, 10:56am

It looks like the index doesn't exist and Metricbeat cannot create it, did you disable automatic index creation in Elasticsearch? https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docs-index_.html#index-creation

Best regards

Kernel_Panic · March 13, 2018, 12:25pm

Hello, thanks fort your reply

I added action.auto_create_index: true to elasticsearch.yml and restarted the service and still seeing the same issue, should I use the API instead the configuration file to set this value?

Thanks
Regards

Kernel_Panic · March 13, 2018, 12:31pm

These are the valued defined in the configuration file

[root@kibana elasticsearch]# grep -v '#' elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.87.22.110
action.auto_create_index: true

Thanks
Regards

Kernel_Panic · March 13, 2018, 5:24pm

I can see there are no templates loaded, could this be for the same reason that the index is not created?

And from kibana logs I see this

["warning","elasticsearch","admin"],"pid":21893,"message":"Unable to revive connection: http://kibana.domain.lan:9200/"}

Thanks

Kernel_Panic · March 13, 2018, 6:14pm

curl  -XGET   'http://kibana.domain.lan:9200/_search?pretty'  -v
* About to connect() to kibana.domain.lan port 9200 (#0)
*   Trying 10.87.22.110...
* Connected to kibana.domain.lan (10.87.22.110) port 9200 (#0)
> GET /_search?pretty HTTP/1.1
> User-Agent: curl/7.29.0
> Host: kibana.domain.lan:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 211
<
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

I also added to elasticsearch.yml the following.

node.master: true
node.data: true

Kernel_Panic · March 14, 2018, 5:22pm

Any idea what could be wrong?

Thanks for your time and support

Kernel_Panic · March 15, 2018, 3:31pm

Moved this issue to elasticsearch cuz I think is not a metricbeat issue.
I enabled debug I only see this error:

[2018-03-15T15:39:12,890][DEBUG][r.suppressed             ] path: /.kibana/_mappings, params: {index=.kibana}
org.elasticsearch.index.IndexNotFoundException: no such index
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:678) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:630) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:586) ~[elasticsearch-6.2.2.jar:6.2.2]

And metricbeat still complains about the index

2018-03-15T15:39:57.155Z ERROR pipeline/output.go:74 Failed to connect: 404 Not Found: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"var","index_uuid":"_na_","index":"var"},"status":404}

By the way SELINUX is disabled

getenforce
Disabled

Kernel_Panic · March 17, 2018, 12:04am

As there was no more feedback please close this topic, I have no idea what's the problem with 6.2 that indexes are not being created I installed 5.6.8 and is working as expected.

Thanks
Regards

system · April 14, 2018, 12:04am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.