URGENT: Kibana not discovering data - tried everything out there and still doesn't work

[kirev95 (Veselin Kirev)]

Hello everyone,

I was using Kibana for a project which I had to present today and unfortunately it didn't discover any data from in Elasticsearch.

I use the following command in Kibana to create the Elasticsearch index:

PUT tweets
{
"mappings": {
"tweet": {
  "properties": {
    "createdAt": {
      "type":   "date",
      "format": "yyyyMMdd'T'HHmmssZ"
    },
    "sentiment":{
    	"type": "text"
    },
"approximateTotalEngagement":{
	"type": "long"
},
"approximateRetweetResponseTime":{
	"type": "integer"
},
    "favoriteCount": {
        "type": "long"
      },
      "hashtagEntities": {
        "properties": {
          "end": {
            "type": "long"
          },
          "start": {
            "type": "long"
.............. and  so on

Then from a Spark program I perform the following:

JavaEsSparkStreaming.saveJsonToEs(processedTweets, "tweets/tweet");

The data is saved in ES which I check by discovering the indices using curl, but nothing is shown in Kibana. Checked the fields, and tried to refresh these when data is available. Everything looks ok, but it just doesn't work. No idea where the bug is, as I haven't changed anything from the Spark App, since last time it was working. The problem is somewhere in Kibana/Elasticsearch configurations, but cannot find it.

I would be really happy if someone has some suggestions! Thanks!

[BigFunger (Jim Unger)]

What do you mean when you say "Nothing is shown in Kibana"? It looks like Kibana is picking up your index's fields at least, and they are showing up as searchable and aggregatable.

Is nothing showing up in discover?

Are you sure that the timepicker (in the upper right) is set to a window that should cover the values in your createdAt field?

When you search using Dev Tools, are you able to get results? Are the createdAt values what you expect them to be?

[kirev95 (Veselin Kirev)]

Hello and thanks for your answer!

I am ashamed of what the problem was. Indeed as you suggested the problem was with the time picker. It appears that my program had a bug, which was not visible until the UK time changed to British Summer Time (BST), which adds 1 additional hour to the original time of each tweet. This means that I wasn't able to see the tweets, because I was searching with the time picker always in the past, e.g. last 1h, 2h, week, etc. whereas the tweets amazingly were "in the future" (1h ahead of my clock), which I didn't expect at all. It's just funny how such small things can cause so many problems!

[BigFunger (Jim Unger)]

We've all been there. Happy to help!