Elsticsearch-py doubt create and index

Elastic Stack Kibana
1

Hi all I've a doubt about the difference between create and index. link Can someone explain to me when I can use one or the other one? I expose the problem. I read from a queue data and in loop a document at times need to be inserted into a same index. Actually with a counter variable I change the number of id and I use index. Everything works fine but when I want to visualize specific data using link kibana starts crushing because saved_object weights the server or have too documents. I 'm a beginner and I can make a mistake in the thinking but my intuition is that documents(with index routine) are inserted in a same document, because when this happens I manage to see into a lighter saved_object on debuggin vega doc that I have only 1 row instead of a situation like this.

image
image1042×898 148 KB

I want to know if I right the thinking supposed that : first I create the index and then I do create of document for pushing data. I want to clarify that the saved_object done is composed by a lot of data(in the order of 300k documents and the main goal is to present data timestamp as x axis and y axis the value.

2

Yes, that is the process for inserting data into Elasticsearch.

However it's not entirely clear to me what the problem you are having here is. Can you add a little bit more information on what is the issue?

3

I use helpers.bulk to load buck data
helpers.bulk(elastic_pointer, dictionary_list)

and each record I create "_index":"myindex_name" and "_id":"myuniqid"

and it does create index if index is not present.

I believe index api should create a index if it is not present

4

The problem is the following:
suppose that my data will be something like this: {timestamp:'v' ,value:'v1}obtained as following:

i=0
for data in consumer:
    
    timestampRow=datetime.datetime.strptime(dt_string,'%m/%d/%y %H:%M:%S')
    event_data={'VALUE': data['VALUE'], '@timestamp': timestampRow,'value2':data[value2],'value3':data[value3]  }
    print(event_data)
    res=es.index(index=indexVal,id=i, body=event_data)
    es.indices.refresh(index=indexVal)
    i=i+1

after inserting data into elasticsearch I want to monitor data using kibana. I'm interesting on visualizing the single value not the aggregation, so I use altair and through jupyter I manage to visualize link . What it's the problem? The problems are two:

  1. When I inserted all past data kibana crush and I don't why, the same happens when I inserted all the visualization created by me. It's impossible that data require too much effort in terms of memory because load data doesn't get over 500MB.
  2. A second problem is the following: why I think there is a problem when I generate the vega saved-object and I manage to see it on dashboard there is something that goes wrong in the convertion between data. The goal is to create something similar to this link but instead of radioboxes I use only combobox. Locally so on jypyter everything was ok but when I save it through the pattern suggested in the first link there is a warning "infinite-extent-for-field-@timestamp-infinity-infinity" so for solving this I investigating and debugging the vega and I've seen that the rapresentation is diffrent from what I presented in post blog picture.
5

Kibana crashes how? What's in the logs? What version are you running? What are you doing when it does this?

6

2 Point Solution : chart_json['config']['kibana']={'hideWarnings': true}

7

look about my post on kibana link I'm sure that elastic works

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.