[6.1.1]delete the index and reindex it,kibana could not discover the data


(Foolcage) #1

delete the index and reindex it,kibana could not discover the data?
I could use curl to get the data,but in kibana discover page,nothing show,any one meet this problem?
➜ ~ curl -XGET 'localhost:9200/_cat/indices?v&s=index&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana uHzsyU6_R8S_HON5E0O9Gg 1 1 7 0 44kb 44kb
yellow open cash_flow_statement 3EKlnpUiR8m8WQBlG4yXbA 5 1 3375 0 5.1mb 5.1mb
yellow open income_statement 93t024BBTWacZNsKeGDvMw 5 1 3379 0 2.6mb 2.6mb
yellow open index_china_day_kdata wib80Y-mQ1GB0GnEKx_oXg 5 1 30845 0 8.6mb 8.6mb
yellow open stock_china_day_kdata gkr2KcvYRrGPXTMUrGKDQQ 5 1 8227016 0 2.2gb 2.2gb
yellow open stock_meta -hO16JILTCW_Ab6dK8B_cw 5 1 3445 0 444.1kb 444.1kb


(Foolcage) #2

I see something strage in kibana:
Time reportEventDate timestamp
January 1st 1970, 13:36:10.930 October 25th 2017, 08:00:00.000 -
January 1st 1970, 13:36:10.930 October 28th 2017, 08:00:00.000

why the time is 1970?
And my timefield is reportEventDate...


(Bhavya R M) #3

Hi,

Have you created index patterns in Kibana for your indices?
This is where you create index patterns: https://www.elastic.co/guide/en/kibana/6.2/index-patterns.html

Thanks,
Bhavya


(Foolcage) #4

yes,I have created the index pattern.
In fact I could found the data, but it seems the time in kibana is very strage.

➜  ~ curl -XPOST 'localhost:9200/income_statement/_search?pretty' -H 'Content-Type: application/json' -d'
{
  "query": {
    "constant_score": {
      "filter": {
        "term": {
          "id": "stock_sh_603707_20171231"
        }
      }
    }
  }
}
'
{
  "took" : 14,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "income_statement",
        "_type" : "doc",
        "_id" : "stock_sh_603707_20171231",
        "_score" : 1.0,
        "_source" : {
          "nonOperatingIncome" : 4063079.89,
          "code" : "603707",
          "sellingExpenses" : 4.696877972E7,
          "OperatingTotalCosts" : 7.6705641194E8,
          "id" : "stock_sh_603707_20171231",
          "totalProfits" : 3.5620431768E8,
          "operatingProfit" : 3.5244199594E8,
          "exchangeGains" : 0.0,
          "attributableToOwnersOfParentCompany" : 3.1390973003E8,
          "financingExpenses" : -4342459.58,
          "investmentIncomeFromRelatedEnterpriseAndJointlyOperating" : "526941.55",
          "reportDate" : "20171231",
          "disposalLossOnNonCurrentLiability" : 0.0,
          "incomeTaxExpense" : 4.198191831E7,
          "EPS" : 0.8131,
          "nonOperatingExpenditure" : 300758.15,
          "incomeFromChangesInFairValue" : 0.0,
          "attributableToMinorityShareholders" : 0.0,
          "reportEventDate" : "2018-01-27",
          "netProfitAttributedToParentCompanyOwner" : 3.1422239937E8,
          "accumulatedOtherComprehensiveIncome" : 3.1390973003E8,
          "businessTaxesAndSurcharges" : 3815442.94,
          "investmentIncome" : 6772404.66,
          "operatingRevenue" : 1.11272600322E9,
          "OperatingCosts" : 5.9668439996E8,
          "securityId" : "stock_sh_603707",
          "minorityInterestIncome" : 0.0,
          "ManagingCosts" : 1.2296594498E8,
          "assetsDevaluation" : 964303.92,
          "dilutedEPS" : 0.8131,
          "netProfit" : 3.1422239937E8,
          "otherComprehensiveIncome" : -312669.34
        }
      }
    ]
  }
}

you could see the time filed:
"reportDate" : "20171231"

But in kibana:

the time filed:
"reportDate"January 1st 1970, 13:36:11.231

In fact, when i index the data,the date format is different between reportDate(yyyyMMdd) and reportEventDate(yyyy-MM-dd).
And from the dsl query directly, all of themn are correct as you can see in the curl result.

Thanks for your reply.


(Bhavya R M) #5

Hi,

When you are creating your index pattern in Kibana - which field did you use for your timefilter? From the screenshot - it looks it's "reportEventDate" . And for this particular index - reportEventDate has the value of "reportEventDate" : "2018-01-27", and that is getting displayed correctly. You can read this for more information: https://www.elastic.co/guide/en/kibana/current/tutorial-define-index.html

I don't know whats happening with "reportDate" : "20171231". You might have to transform it using an ingest filter before ingesting the data into Elastic. I will find that out for you in a few hours.

Thanks,
Bhavya


(Foolcage) #6

Thanks!
more details:
the first time I index the data to es, evety thing is correct in kibana.
then I delete all of the index and reindex them again.
the strange things happen.
um..I used reportDate for the time field in kibana.


(Foolcage) #7

correct it:
it shoud be the time=reportDate and all of them is :1970...(so the problem is here)


(Foolcage) #8

I think es treat 20171231 as the seconds that have elapsed since January 1, 1970...

curl -XPOST 'localhost:9200/income_statement/doc/_search?pretty&filter_path=hits.hits._source' -H 'Content-Type: application/json' -d'
{
  "query": {
    "range" : {
                "reportDate" : {
                    "gte" : "2017-06-30",
                    "lte" :  "2017-06-30"
                }
            }
  },
  "size": 10,
  "sort": [
    {
      "netProfit": {
        "order": "desc"
      }
    }
  ]
}
'
{ }

the result is empty

and

curl -XPOST 'localhost:9200/income_statement/doc/_search?pretty&filter_path=hits.hits._source' -H 'Content-Type: application/json' -d'
{
  "query": {
    "range" : {
                "reportDate" : {
                    "gte" : 20170630,
                    "lte" :  20170630
                }
            }
  },
  "size": 10,
  "sort": [
    {
      "netProfit": {
        "order": "desc"
      }
    }
  ]
}
'
{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "nonOperatingIncome" : 8.33E8,
          "code" : "600028",
          "sellingExpenses" : 2.5955E10,
          "OperatingTotalCosts" : 1.126122E12,
          "id" : "stock_sh_600028_20170630",
          "totalProfits" : 4.5032E10,
          "operatingProfit" : 4.5015E10,
          "exchangeGains" : 0.0,
          "attributableToOwnersOfParentCompany" : 2.645E10,
          "financingExpenses" : 1.289E9,
          "investmentIncomeFromRelatedEnterpriseAndJointlyOperating" : "0",
          "reportDate" : "20170630",
          "disposalLossOnNonCurrentLiability" : 0.0,
          "incomeTaxExpense" : 8.915E9,
          "EPS" : 0.224,
          "nonOperatingExpenditure" : 8.16E8,
          "incomeFromChangesInFairValue" : 3.69E8,
          "attributableToMinorityShareholders" : 8.557E9,
          "reportEventDate" : "2017-08-28",
          "netProfitAttributedToParentCompanyOwner" : 2.7092E10,
          "accumulatedOtherComprehensiveIncome" : 3.5007E10,
          "businessTaxesAndSurcharges" : 1.16297E11,
          "investmentIncome" : 8.152E9,
          "operatingRevenue" : 1.165837E12,
          "OperatingCosts" : 9.42602E11,
          "securityId" : "stock_sh_600028",
          "minorityInterestIncome" : 9.025E9,
          "ManagingCosts" : 3.5903E10,
          "assetsDevaluation" : 4.076E9,
          "dilutedEPS" : 0.224,
          "netProfit" : 3.6117E10,
          "otherComprehensiveIncome" : -1.11E9
        }
      },

the result is ok

So es don't handle timeformat "YYYYmmdd" corrcectly?
I think "YYYYmmdd" and "YYYY-mm-dd" always treated as valid date...


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.