Create Elastic index from Python script

[Jose_Angel_Morena_Si (jmorena)]

Hello,

I am trying to create an index in ES 7.9.1 with python 3. In order to do so I am using elasticsearch python package.

Script:

      doc_id=hex(random.getrandbits(128))[2:]
      doc = {
        'response_time': measure_response_time(app_url, request_timeout),
        '@timestamp': datetime.now()
      }
      index_name='java-response-time-'+str(datetime.today().strftime('%Y-%m-%d'))
      logger.info('index: '+index_name)
      res = es.index(index=index_name,id=doc_id, body=doc)
      logger.info('Estado nuevo indice: '+str(res['result']))
      res = es.get(index=index_name, id=doc_id)
      logger.info("Tiempo de respuesta: "+ str(res['_source']['response_time']))
      es.indices.refresh(index=index_name)
      time.sleep(int(request_frequency))

When I specify @timestamp as a time field:

Documents are missing in "Discover" page:

Reminder: This only happens when I select the field @timestamp as a date time field. If I don't select it, then documents appear.

What is the correct date time format that I must use in my python script?

Thanks in advance!

[abrx]

Hi !
It's just an idea but do you have same UTC zone on your python client / elasticsearch server ? (Maybe you can find out by looking at the indexed doc with no @timestamp specified)
Not sure that Elasticsearch will index data happening in the future.
You can (should) add the UTC info in datetime.now (default default parameter tz=None)

[Jose_Angel_Morena_Si (jmorena)]

Hello,

Your comment shed some light on the matter and I managed to make the script work!

Eventually, what made it work was the following code:

      local_tz = pytz.timezone('Europe/Madrid')
      dt = local_tz.localize(datetime.today())
      doc = {
        'response_time': measure_response_time(app_url, request_timeout),
        '@timestamp': dt,
        'time': datetime.now().strftime("%Y-%m-%d %H:%M")
      }

Comment, I set the Kibana timezone in Advaced Settings to: "Europe/Madrid". By default it takes the one from the web navigator,
Thank you @abrx!