Pushing of data into elastic search dashboard using Python

Elastic Stack Elasticsearch
1

I am working on a project which collects twitter data. So, up until now, what I did is I collected some tweets and then stored them in a CSV format and uploaded it to the elastic cloud. I am now working on automatically pushing the data into the elastic cloud. But, I am not getting any results. Can anyone help me with this? Here is the code I've written for pushing the data:

Screenshot (630)
Screenshot (630)832×236 12.1 KB

Can someone help me if there should be any changes made in the code? The error that I'm getting is : "ValueError: Can't specify the options 'http_auth' via a dictionary in 'hosts', only 'host', 'path_prefix', 'port', 'scheme', 'url_prefix', 'use_ssl' options are allowed"

2

Hi Shashank,

For next time, please try to include a formatted code text snippet rather than screenshot to show your code to make reading easier. Can you share which version of the client and which version of Elasticsearch you're using?

I'll try to help assuming you're using the latest version of the API. From the client docs of the latest API , although I'm no Python expert, I think you need something like this where the options are individual method parameters rather than being included in the array the way you have above:

es_client = Elasticsearch(“my-elasticsearch-host:port”, http_auth=(user,password))

Hope that helps!

3

It is showing this error: "ValueError: Could not parse URL". I put out my Cloud deployment details which I got from the elastic Cloud, so why is it showing this error? The version of Elastic Search is '8.4.1' and the code is : ```
es_client = Elasticsearch()
def doc_generator(df):
df_iter = df.iterrows()
for index, document in df_iter:
yield {
"_index": 'a1.index',
"_type": "_doc",
"_id" : f"{document['id']}",
"_source": self.filterKeys(document),
}
helpers.bulk(es_client, doc_generator(df))

`I hope I can get some help regarding this because the previous method is not working

4

Shashank,
I have no experience with cloud connection. but here is what you do

importing some library
from elasticsearch import Elasticsearch, helpers

open connection with elasticsearch
es = Elasticsearch('your hostname', http_auth=('username', 'password'), port=9200 ) --> this might be different on elastic version 8.x. works in 7.x
print (es.info()) --> if this does not work then you do not have proper connection to elastic.

create a data in to list of dictionary ,
output_data =[ { dictionary of fist record}, { dictionary of second record} ]

now push that to elastic using bulk method
pb = helpers.bulk(es, output_data)
print (pb)

5

OK thanks for that, so mine is elastic version 8.x. So, I think that's why I am getting the problem. Can you help me with the current Elastic version then?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.