Adding @timestamp to ES index turns in no results on kibana

Elastic Stack Kibana
1

Hi all,

I wrote a pretty simple python script:

import json
import datetime
from elasticsearch5 import Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

now = datetime.datetime.now()
INDEX_NAME = 'aggregated-%s' % now.strftime("%Y.%m.%d")

settings = {
    "mappings": {
		"logs": {
            "properties": {
                "Tempo medio chiamata (ms)": {
                    "type": "float"
                },
				"Numero di chiamate" :{
					"type": "integer"
				},
				"@timestamp" :{
					"type" : "date"
					}
            }
        }
    }
}

if not es.indices.exists(INDEX_NAME):
	es.indices.create(index=INDEX_NAME, ignore=400, body=settings)
	
res = es.search(body={
	"size": 0,
	"aggs": {
		"services": {
			"terms": {
				"field": "resource_templatePath.keyword"
			},
			"aggs": {
				"avg_duration": {
					"avg": {
						"field": "http_durationMs"
					}
				}
			}
		}
	}
}
)

for f in res['aggregations']['services']['buckets']:
	req_body= {
		'Function': '%s' % f['key'],
		'Numero di chiamate': '%s' % f['doc_count'],
		'Tempo medio chiamata (ms)': '%s' % f['avg_duration']['value'],
		'@timestamp': now.isoformat()
		}
	es.index(index=INDEX_NAME, doc_type='logs', body=json.dumps(req_body))

It works fine, and if I query Elastic, result is:

{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 4,
    "max_score": 1,
    "hits": [
      {
        "_index": "aggregated-2018.04.18",
        "_type": "logs",
        "_id": "AWLZGC4XTA1Gx7az1yUk",
        "_score": 1,
        "_source": {
          "Function": "GET <ues-mocks>/v1/work-order-activity/{idWorkOrderActivity}/verify-parallel-start",
          "Numero di chiamate": "6",
          "Tempo medio chiamata (ms)": "205.33333333333334",
          "@timestamp": "2018-04-18T16:11:01.902051"
        }
      },
.
.

Problem is: kibana shows no results (of course, the time interval is not the issue).

If I take away the @timestamp field, and recreate the index pattern it all works fine on kibana, but of course I have no indication about event timing anymore.

Can anyone help me please? I'm stuck with it and I'm losing my mind..

Thank you very much!

Cheers

2

Hi,

two ideas: the mapping was really written correctly? So you could setup an index pattern in Kibana and select @timestamp as a field?

Second idea: the dates are around now. So I actually suggest it might be the time interval :slight_smile:
The dates you are writing are assumed to be UTC timezone if you are not specifying anything else. If you are not sitting in a UTC-x timezone Kibana (by default) will use that timezone to look at the documents. Meaning you won't see any documents you just created, because they are "in the future" for you.

Could you switch in the time picker to relative mode and make sure you are not just viewing a timerange till "now" but till "1 day from now" at least and check if that helps viewing the documents?

Cheers,
Tim

3

Hi Tim,

that was exactly the second idea! Kibana "is running" 2 hours ahead of my local date time.
Any suggestion on how to sync the clocks?

Thanks a lot!

Cheers,
Simone

4

You could either append the correct timezone when writing the data in the ISO format, so Elasticsearch will actually already calculate the offset when storing and store the appropriate UTC time. Or you could switch your Kibana in "Management > Kibana > Advanced Settings" via the dateFormat:tz setting to UTC, so it always assumes you are being in UTC, and not in your browsers timezone.

Cheers,
Tim

5

Great, thanks!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.