Write to TSDS index via python

Elastic Stack Elasticsearch
1

I create ILM/Template and Index
when I try to write via python it errors
but same record works via dev tool

POST sachin_test/_doc
{
  "@timestamp": "2024-06-06T19:21:15.000Z", --> this part I did manually
  "system_type": "gpu", 
  "available": 1, "maintenance_status": 1, "cpu_core_inventory": 0, 
  "sum_cpu_core_inventory": 0, "sum_gpu_card_inventory": 0, 
  "count": 1, "system_prod": "yes", "center": "hou", 
  "host_status": "maintenance"
}

Here is python record looks like

{'@timestamp': Timestamp('2024-06-06 13:45:00+0000', tz='UTC'), 'system_type': 'gpu', 'available':1, 'maintenance_status': 1, 'cpu_core_inventory': 0, 'sum_cpu_core_inventory': 0, 'sum_gpu_card_inventory': 0, 'count': 1, 'system_prod': 'yes', 'center': 'hou', 'host_status': 'maintenance', '_index': 'sachin_test', '_op_type': 'create'}

it says
error is 500 document(s) failed to index.
An exception of type BulkIndexError occurred. Arguments:
('500 document(s) failed to index.',)

2

Hmmm You should probaly share your python code.

Did you simply try

'timestamp': datetime.now(),

Like the Example here to see if it works...

3

Basically I do push this using helpers.bulk

mylist = [{'@timestamp': Timestamp('2024-06-06 13:45:00+0000', tz='UTC'), 'system_type': 'gpu', 'available':1, 'maintenance_status': 1, 'cpu_core_inventory': 0, 'sum_cpu_core_inventory': 0, 'sum_gpu_card_inventory': 0, 'count': 1, 'system_prod': 'yes', 'center': 'hou', 'host_status': 'maintenance', '_index': 'sachin_test', '_op_type': 'create'}]

helpers.bulk(elastic_output, mylist)
4

and I just try this. same error

{'@timestamp': Timestamp('2024-06-07 08:25:03.480489'),

5

Delete the index first and then try doing the python first.
Perhaps there's something wrong with the initial doc you posted

6

if I write to different index all togather ( regular index) then it works.
something with TSDS and timestamp for sure.

7

Did you walk through this carefully?

1 Like
8

ok it works. Thanks @stephenb

basically I had time different between TSDS creation and @timestamp that I was inserting.

1 Like
9

Yes that is interesting can you show / describe a little more for anyone else that runs into the same?

10

I think when you create TSDS with index.mode = time_series enable

then it has two value

index.time_series.start_time value set to now - index.look_ahead_time
index.time_series.end_time value set to now + index.look_ahead_time

and you can only write between these two time.

1 Like