New data not showing in dashboard

Elastic Stack Kibana
1

I have new data loading into an index.
I can see the data under the discover tab.

But I cannot see the data in my dashboard.

Anyone ever experience this?

Here is the discover tab:

I am positive I am not seeing this new data in the dashboard.
Do I need to rebuild the dashboard?

All this is happening after upgrading from:

  • Elasticsearch: 2.0.0 to 2.3.1

  • Kibana: 4.0 to 4.5

  • logstash: 2.0.0 to 2.3.1

2

I think I figured out the problem.
The field my visualization needs is now ANALYZED.

I guess when I did a restore from snapshot, that setting didn't carry over?

3

Ok, I didnt figure it out.
I cannot see any data beyond April 15th, today is April 27th.

4

Is it also correct to presume that you can't see the data on visualize as well?

Do other dates not work? Is your data consistent with fields and such?

5

Correct, I cannot see new data within the visualizations.
I can see new data only in the 'Discover' tab.

6

Could it be that the new data doesn't fit some filter or query parameters of your visualizations? Or did you try creating new visualizations and the data still doesn't appear?

You could check the "Request" of the Visualization and paste it here.

7

Here is the entire Request, and part of the Response.

Request:

{
  "size": 0,
  "query": {
    "filtered": {
      "query": {
        "query_string": {
          "analyze_wildcard": true,
          "query": "*"
        }
      },
      "filter": {
        "bool": {
          "must": [
            {
              "range": {
                "@timestamp": {
                  "gte": 1459483200000,
                  "lte": 1462075199999,
                  "format": "epoch_millis"
                }
              }
            }
          ],
          "must_not": []
        }
      }
    }
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "12h",
        "time_zone": "America/New_York",
        "min_doc_count": 1,
        "extended_bounds": {
          "min": 1459483200000,
          "max": 1462075199999
        }
      },
      "aggs": {
        "3": {
          "terms": {
            "field": "Member Type",
            "size": 16,
            "order": {
              "_count": "desc"
            }
          }
        }
      }
    }
  }
}

Response:

{
  "took": 25,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 284958,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "2": {
      "buckets": [
        {
          "3": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "Full ATS Membership (US Only)",
                "doc_count": 6927
              },
              {
                "key": "Full ATS Membership (International Class A)",
                "doc_count": 2255
              },
              {
                "key": "Full ATS Membership (International Class C)",
                "doc_count": 1711
              },
              {
                "key": "Trainee ATS Membership - First Year Free (US Only)",
                "doc_count": 1224
              },
              {
                "key": "Trainee ATS Membership (US Only)",
                "doc_count": 1073
              },
              {
                "key": "Trainee ATS Membership - First Year Free (International)",
                "doc_count": 540
              },
              {
                "key": "Affiliate ATS Membership (US Only)",
                "doc_count": 408
              },
              {
                "key": "Emeritus ATS Membership",
                "doc_count": 383
              },
              {
                "key": "Trainee ATS Membership (International)",
                "doc_count": 383
              },
              {
                "key": "Senior ATS Membership (US Only)",
                "doc_count": 326
              },
              {
                "key": "Senior ATS Membership (International)",
                "doc_count": 248
              },
              {
                "key": "Full ATS Membership (International Class B)",
                "doc_count": 212
              },
              {
                "key": "Corporate ATS Membership",
                "doc_count": 71
              },
              {
                "key": "Honorary ATS Membership",
                "doc_count": 36
              },
              {
                "key": "Full ATS Membership (Complimentary Membership)",
                "doc_count": 22
              },
              {
                "key": "Affiliate ATS Membership (Complimentary Membership)",
                "doc_count": 17
              }
            ]
          },
          "key_as_string": "2016-04-01T00:00:00.000-04:00",
          "key": 1459483200000,
          "doc_count": 15836
        },
8

I'm guessing one of the upgrades might have changed the mapping of one of your fields. Maybe in one of the fields used in your Visualization like "Member Type".

You can check the mapping of an index where you do get data, and one where you do not get data to see. You can use something like this;
http://localhost:9200/logstash-2016.04.22/_mapping

It might be easiest to save an old and new mapping and diff them with the tool of your choice. If you find a difference we can figure out how to fix it.

Regards,
Lee

9

Actually, now that I think about it, I had this problem before the upgrade.
I was hoping the upgrade would fix the problem, but no.

10

I think I figured this out.

Somehow, my Member Type field became lower case!
Not sure why that is the case.

11

When I first created the index on the older server, I used this:
curl -XPOST http://localhost:9200/member_type_history -d '{"mappings" : {"Member Type" : {"properties":{"Member Type":{"type":"string","index":"not_analyzed"}}}}}'

I tried restoring this data with a snapshot, but new data would not show.

So, on the new server, I used this mapping instead:
curl -XPOST http://localhost:9200/member_type_history -d '{"mappings" : {"Member Type" : {"properties":{"member type":{"type":"string","index":"not_analyzed"}}}}}'

I can now see new data! thanks to the new mapping!
Only difference is the letter case. Member Type vs member type

So, now I need to get that old data into my new Elasticsearch.
Any suggestions?, especially with the field name case being different.