No Data Found error

ronchalant · January 25, 2017, 1:19pm

I have looked through many other threads on this issue, and I cannot seem to find anything that works for me.

I have an Elasticsearch 2.4.0 cluster w/ the appropriate marvel agent configured on all nodes. This has been verified through the GET _cat/plugins call, where all nodes have these configured:

Data02   license      2.4.0 j  
Data02   marvel-agent 2.4.0 j

My license is current and value through Sept 2017 - from GET _license:

  "license": {
    "status": "active",
    "type": "basic",
    "issue_date": "2016-07-29T00:00:00.000Z",
    "issue_date_in_millis": 1469750400000,
    "expiry_date": "2017-09-13T23:59:59.999Z",
    "expiry_date_in_millis": 1505347199999,
    "max_nodes": 100

(extraneous & identifying information was removed from the above response)

Marvel DID work for a while. We had some infrastructure issues a few weeks back however and it seems to have stopped working since for some reason.

There are still indexes being maintained:

GET _cat/indices/*marvel*
green open .marvel-2016.09.09      1 1  205521    0 853.9mb 426.9mb 
green open .marvel-2016.09.08      1 1  189022    0 810.8mb 405.4mb 
green open .marvel-2016.09.07      1 1 1017948    0   2.2gb   1.1gb 
green open .marvel-es-1-2017.01.25 1 1   32073    0  23.4mb   7.8mb 
green open .marvel-es-1-2017.01.24 1 1   55493    0  28.7mb  14.4mb 
green open .marvel-es-1-2017.01.19 1 1   59577    0  26.7mb  13.3mb 
green open .marvel-2016.09.10      1 1   36158    0 139.3mb  69.6mb 
green open .marvel-es-1-2017.01.23 1 1   56742    0  28.5mb  14.3mb 
green open .marvel-es-data-1       1 1      19 7486   681kb 340.5kb 
green open .marvel-es-1-2017.01.22 1 1   59341    0  24.9mb  12.4mb 
green open .marvel-es-1-2017.01.21 1 1   59518    0  24.5mb  12.3mb 
green open .marvel-es-1-2017.01.20 1 1   59576    0  26.5mb  13.3mb 
green open .marvel-kibana          1 1       1    0   6.5kb   3.2kb

We did upgrade from 1.7 -> 2.4 last september so some older indices are still lingering about, and we configured marvel to only roll 7 days of logs.

I added an alias .marvel-es-data which references .marvel-es-data-1 per one suggestion I found, but to no avail.

Looking at today's index (.marvel-es-1-2017.01.25) to see which nodes are reporting, I did note that two nodes are missing: one of the Data-only nodes and one of the Master nodes (the one that happens to be the current master). I don't know that this is the cause of the issue, but I'm going to look to restart both services off-hours. Maybe it is?

There are no errors reported in any logs that seem relevant, and the Sense UI via Kibana works fine. The ELASTICSEARCH_URL parameter looks good as well (Kibana & its plugins are in a docker container so I've been rebuilding that regularly trying to get it to work again).

Should the .marvel-es-1-{yyyy.mm.dd} nodes be part of some alias perhaps? I'm really baffled as to what else could be wrong.

ronchalant · January 25, 2017, 1:29pm

Of course right after this I found a thread on the main page with a similar issue (though with 2.0.0)

If this helps, on the Kibana instance this call returns an empty array:

/api/marvel/v1/clusters

I would note that we have a non-standard cluster name (it's not elasticsearch but something else)

pickypg · January 25, 2017, 2:42pm

Hi @ronchalant,

What version of Kibana / Marvel UI are you running? My guess is that this is the issue based on the index names.

Given that you're running ES 2.4.0, you should be running Kibana 4.6.x with the latest Marvel UI plugin for that version.

ronchalant · January 25, 2017, 3:14pm

I'm running Kibana 4.6 with Marvel 2.4.0. It's from a Dockerfile that we've used previously that simply loads in the plugins we want:

FROM kibana:4.6
RUN gosu kibana /opt/kibana/bin/kibana plugin --install elastic/sense
RUN gosu kibana /opt/kibana/bin/kibana plugin --install elasticsearch/marvel/2.4.0

ronchalant · January 25, 2017, 3:20pm

FWIW I updated to 2.4.4 and same error. As far as I can tell it doesn't seem to find cluster(s).

pickypg · January 26, 2017, 3:46am

Can you post the output of the .marvel-es-data-1 index settings and mappings?

curl -XGET localhost:9200/.marvel-es-data-1

Thanks

ronchalant · January 26, 2017, 11:57am

The size is too big .. I have to split it across a few posts..

{
  ".marvel-es-data-1" : {
    "aliases" : {
      ".marvel-es-data" : { }
    },
    "mappings" : {
      "cluster_event" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "type" : "multi_field",
              "fields" : {
                "{name}" : {
                  "index" : "analyzed",
                  "omit_norms" : true,
                  "type" : "string"
                },
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        } ]
      },
      "routing_event" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "type" : "multi_field",
              "fields" : {
                "{name}" : {
                  "index" : "analyzed",
                  "omit_norms" : true,
                  "type" : "string"
                },
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        } ]
      },
      "node" : {
        "enabled" : false,
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "type" : "multi_field",
              "fields" : {
                "{name}" : {
                  "index" : "analyzed",
                  "omit_norms" : true,
                  "type" : "string"
                },
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        } ]
      },
      "cluster_info" : {
        "enabled" : false,
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "type" : "multi_field",
              "fields" : {
                "{name}" : {
                  "index" : "analyzed",
                  "omit_norms" : true,
                  "type" : "string"
                },
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        } ]
      },
      "node_event" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "type" : "multi_field",
              "fields" : {
                "{name}" : {
                  "index" : "analyzed",
                  "omit_norms" : true,
                  "type" : "string"
                },
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        } ]
      },
      "cluster_state" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "type" : "multi_field",
              "fields" : {
                "{name}" : {
                  "index" : "analyzed",
                  "omit_norms" : true,
                  "type" : "string"
                },
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        } ],
        "properties" : {
          "blocks" : {
            "type" : "object",
            "enabled" : false
          },
          "nodes" : {
            "type" : "object",
            "enabled" : false
          },
          "routing_nodes" : {
            "type" : "object",
            "enabled" : false
          },
          "routing_table" : {
            "type" : "object",
            "enabled" : false
          }
        }
      },
      "indices_stats" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "type" : "multi_field",
              "fields" : {
                "{name}" : {
                  "index" : "analyzed",
                  "omit_norms" : true,
                  "type" : "string"
                },
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        } ],

ronchalant · January 26, 2017, 11:58am

"properties" : {
  "primaries" : {
    "properties" : {
      "docs" : {
        "properties" : {
          "count" : {
            "type" : "long"
          }
        }
      },
      "indexing" : {
        "properties" : {
          "index_total" : {
            "type" : "long"
          }
        }
      }
    }
  },
  "total" : {
    "properties" : {
      "search" : {
        "properties" : {
          "query_total" : {
            "type" : "long"
          }
        }
      }
    }
  }
}
  },
  "node_stats" : {
"dynamic_templates" : [ {
  "string_fields" : {
    "mapping" : {
      "type" : "multi_field",
      "fields" : {
        "{name}" : {
          "index" : "analyzed",
          "omit_norms" : true,
          "type" : "string"
        },
        "raw" : {
          "ignore_above" : 256,
          "index" : "not_analyzed",
          "type" : "string"
        }
      }
    },
    "match" : "*",
    "match_mapping_type" : "string"
  }
} ],
"properties" : {
  "breakers" : {
    "properties" : {
      "fielddata" : {
        "properties" : {
          "estimated_size_in_bytes" : {
            "type" : "long"
          },
          "limit_size_in_bytes" : {
            "type" : "long"
          },
          "tripped" : {
            "type" : "long"
          }
        }
      },
      "parent" : {
        "properties" : {
          "estimated_size_in_bytes" : {
            "type" : "long"
          },
          "limit_size_in_bytes" : {
            "type" : "long"
          },
          "tripped" : {
            "type" : "long"
          }
        }
      },
      "request" : {
        "properties" : {
          "estimated_size_in_bytes" : {
            "type" : "long"
          },
          "limit_size_in_bytes" : {
            "type" : "long"
          },
          "tripped" : {
            "type" : "long"
          }
        }
      }
    }
  },
  "fs" : {
    "properties" : {
      "total" : {
        "properties" : {
          "disk_io_op" : {
            "type" : "long"
          },
          "disk_io_size_in_bytes" : {
            "type" : "long"
          },
          "disk_read_size_in_bytes" : {
            "type" : "long"
          },
          "disk_reads" : {
            "type" : "long"
          },
          "disk_write_size_in_bytes" : {
            "type" : "long"
          },
          "disk_writes" : {
            "type" : "long"
          }
        }
      }
    }
  },
  "indices" : {
    "properties" : {
      "indexing" : {
        "properties" : {
          "throttle_time_in_millis" : {
            "type" : "long"
          }
        }
      },
      "percolate" : {
        "properties" : {
          "memory_size_in_bytes" : {
            "type" : "long"
          },
          "queries" : {
            "type" : "long"
          },
          "time_in_millis" : {
            "type" : "long"
          },
          "total" : {
            "type" : "long"
          }
        }
      },
      "query_cache" : {
        "properties" : {
          "evictions" : {
            "type" : "long"
          },
          "hit_count" : {
            "type" : "long"
          },
          "memory_size_in_bytes" : {
            "type" : "long"
          },
          "miss_count" : {
            "type" : "long"
          }
        }
      },
      "segments" : {
        "properties" : {
          "index_writer_max_memory_in_bytes" : {
            "type" : "long"
          },
          "index_writer_memory_in_bytes" : {
            "type" : "long"
          },
          "version_map_memory_in_bytes" : {
            "type" : "long"
          }
        }
      }
    }
  },
  "jvm" : {
    "properties" : {
      "buffer_pools" : {
        "properties" : {
          "direct" : {
            "properties" : {
              "used_in_bytes" : {
                "type" : "long"
              }
            }
          },
          "mapped" : {
            "properties" : {
              "used_in_bytes" : {
                "type" : "long"
              }
            }
          }
        }
      },

ronchalant · January 26, 2017, 12:00pm

              "gc" : {
            "properties" : {
              "collectors" : {
                "properties" : {
                  "old" : {
                    "properties" : {
                      "collection_count" : {
                        "type" : "long"
                      },
                      "collection_time_in_millis" : {
                        "type" : "long"
                      }
                    }
                  },
                  "young" : {
                    "properties" : {
                      "collection_count" : {
                        "type" : "long"
                      },
                      "collection_time_in_millis" : {
                        "type" : "long"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      },
      "os" : {
        "properties" : {
          "load_average" : {
            "properties" : {
              "15m" : {
                "type" : "float"
              },
              "1m" : {
                "type" : "float"
              },
              "5m" : {
                "type" : "float"
              }
            }
          }
        }
      },
      "thread_pool" : {
        "properties" : {
          "listener" : {
            "properties" : {
              "completed" : {
                "type" : "long"
              },
              "largest" : {
                "type" : "long"
              },
              "queue" : {
                "type" : "long"
              },
              "rejected" : {
                "type" : "long"
              },
              "threads" : {
                "type" : "long"
              }
            }
          }
        }
      }
    }
  },
  "cluster_stats" : {
    "dynamic_templates" : [ {
      "string_fields" : {
        "mapping" : {
          "type" : "multi_field",
          "fields" : {
            "{name}" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "raw" : {
              "ignore_above" : 256,
              "index" : "not_analyzed",
              "type" : "string"
            }
          }
        },
        "match" : "*",
        "match_mapping_type" : "string"
      }
    } ]
  },
  "index_stats" : {
    "dynamic_templates" : [ {
      "string_fields" : {
        "mapping" : {
          "type" : "multi_field",
          "fields" : {
            "{name}" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "raw" : {
              "ignore_above" : 256,
              "index" : "not_analyzed",
              "type" : "string"
            }
          }
        },
        "match" : "*",
        "match_mapping_type" : "string"
      }
    } ],
    "properties" : {
      "index" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        },
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed",
            "ignore_above" : 256
          }
        }
      },
      "primaries" : {
        "properties" : {
          "docs" : {
            "properties" : {
              "count" : {
                "type" : "long"
              }
            }
          },
          "indexing" : {
            "properties" : {
              "index_total" : {
                "type" : "long"
              }
            }
          }
        }
      },
      "total" : {
        "properties" : {
          "fielddata" : {
            "properties" : {
              "memory_size_in_bytes" : {
                "type" : "long"
              }
            }
          },
          "indexing" : {
            "properties" : {
              "throttle_time_in_millis" : {
                "type" : "long"
              }
            }
          },
          "merges" : {
            "properties" : {
              "total_size_in_bytes" : {
                "type" : "long"
              }
            }
          },
          "percolate" : {
            "properties" : {
              "memory_size_in_bytes" : {
                "type" : "long"
              },
              "queries" : {
                "type" : "long"
              },
              "time_in_millis" : {
                "type" : "long"
              },
              "total" : {
                "type" : "long"
              }
            }
          },
          "query_cache" : {
            "properties" : {
              "evictions" : {
                "type" : "long"
              },
              "hit_count" : {
                "type" : "long"
              },
              "memory_size_in_bytes" : {
                "type" : "long"
              },
              "miss_count" : {
                "type" : "long"
              }
            }
          },

ronchalant · January 26, 2017, 12:00pm

              "search" : {
            "properties" : {
              "query" : {
                "properties" : {
                  "query_total" : {
                    "type" : "long"
                  }
                }
              }
            }
          },
          "segments" : {
            "properties" : {
              "index_writer_max_memory_in_bytes" : {
                "type" : "long"
              },
              "index_writer_memory_in_bytes" : {
                "type" : "long"
              },
              "version_map_memory_in_bytes" : {
                "type" : "long"
              }
            }
          }
        }
      }
    }
  },
  "index_event" : {
    "dynamic_templates" : [ {
      "string_fields" : {
        "mapping" : {
          "type" : "multi_field",
          "fields" : {
            "{name}" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "raw" : {
              "ignore_above" : 256,
              "index" : "not_analyzed",
              "type" : "string"
            }
          }
        },
        "match" : "*",
        "match_mapping_type" : "string"
      }
    } ]
  },
  "shard_event" : {
    "dynamic_templates" : [ {
      "string_fields" : {
        "mapping" : {
          "type" : "multi_field",
          "fields" : {
            "{name}" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "raw" : {
              "ignore_above" : 256,
              "index" : "not_analyzed",
              "type" : "string"
            }
          }
        },
        "match" : "*",
        "match_mapping_type" : "string"
      }
    } ]
  },
  "_default_" : {
    "dynamic_templates" : [ {
      "string_fields" : {
        "mapping" : {
          "type" : "multi_field",
          "fields" : {
            "{name}" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "raw" : {
              "ignore_above" : 256,
              "index" : "not_analyzed",
              "type" : "string"
            }
          }
        },
        "match" : "*",
        "match_mapping_type" : "string"
      }
    } ]
  }
},
"settings" : {
  "index" : {
    "codec" : "best_compression",
    "marvel" : {
      "plugin" : {
        "version" : "2.4.0"
      },
      "template" : {
        "version" : "1"
      },
      "index_format" : "6"
    },
    "number_of_shards" : "1",
    "mapper" : {
      "dynamic" : "false"
    },
    "creation_date" : "1473479904156",
    "analysis" : {
      "analyzer" : {
        "default" : {
          "type" : "standard",
          "stopwords" : "_none_"
        }
      }
    },
    "number_of_replicas" : "1",
    "uuid" : "w8Vz9x4MQ8OMYgNokgQ7zA",
    "version" : {
      "created" : "2040099"
    }
  }
},
"warmers" : { }

}
}

pickypg · January 26, 2017, 4:28pm

This is most likely related to the issue. This is my local version (in 5.1.2, so I have two extra types, _default_ and kibana):

{
  ".monitoring-data-2": {
    "aliases": {},
    "mappings": {
      "node": {
        "enabled": false
      },
      "cluster_info": {
        "enabled": false,
        "_meta": {
          "xpack.version": "5.1.2"
        }
      },
      "_default_": {
        "enabled": false
      },
      "kibana": {
        "enabled": false
      }
    },
    "settings": {
      "index": {
        "codec": "best_compression",
        "number_of_shards": "1",
        "provided_name": ".monitoring-data-2",
        "creation_date": "1485231084460",
        "number_of_replicas": "1",
        "uuid": "VFLJgf5_RqyrBO5M0c-Q_w",
        "version": {
          "created": "5010299"
        }
      }
    }
  }
}

Yours should be along the lines of

{
  ".marvel-es-data-1": {
    "aliases": {},
    "mappings": {
      "node": {
        "enabled": false
      },
      "cluster_info": {
        "enabled": false,
        "_meta": {
          "marvel.version": "2.4.4"
        }
      }
    },
    "settings": {
      "index": {
        "codec": "best_compression",
        "number_of_shards": "1",
        "provided_name": ".marvel-es-data-1",
        "creation_date": "1485231084460",
        "number_of_replicas": "1",
        "uuid": "VFLJgf5_RqyrBO5M0c-Q_w",
        "version": {
          "created": "2040499"
        }
      }
    }
  }
}

Note that things aren't actually mapped. We're explicitly using that index as a cache.

This suggests that you have a global template (a template whose value is "template": "*"). Can you check to see if that's the case and remove it, then delete your .marvel-* indices and allow them to be recreated without any global template?

ronchalant · January 26, 2017, 4:44pm

The only other template with a potentially conflicting wildcard was an older one - one just called 'marvel' that appears to have been leftover, and it had a wildcard of .marvel-*

I removed that, deleted all marvel indexes and it just recreated the one for today, but still no dice

In developer tools on the kibana marvel page it's still POSTing to /api/marvel/v1/clusters every few moments and getting an empty array.

If I delete the remaining marvel templates would the agent automatically recreate them?

system · February 23, 2017, 4:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.