Actually, I've tried pushing the data but to no use. I'm using Elastic Search version '8.4.1'. I want to push the CSV directly into the Elasticsearch without uploading it manually using the File Data Visualizer option in Elastic Cloud. So, can anybody help me with that?
You could use an ingest pipeline with csv processor
There are also many articles on this how to do with just python
https://www.logicalfeed.com/posts/1183/upload-bulk-csv-data-to-elasticsearch-using-python
Just a small doubt!! So, I'm using Elastic Cloud for my project and to use dashboards in Kibana. So, I've tried keeping my Cloud ID as my host but it's not working. Can you help me with this problem?
Have you looked at the documentation for the Python client?
Yes, I've looked into it and I am also getting the client.info() if we print that. Just during the pushing, it shows errors and I've been stuck on it for quite some time. So, just need a proper tutorial for it.
Elasticsearch stores data in JSON, so you will need to parse your CSV and create a JSON document for each line and this process is outlined in the guides Stephen linked to. Have you gone through these?
This is what Kibana does behind the scenes when you upload a file.
If you are getting errors you need to show the full errors and exactly what you are doing if anyone is going to be able to help.
1 Like
I'm getting this error: elasticsearch.helpers.BulkIndexError: 100 document(s) failed to index. Can you help me with this?
There must be more to the error describing what actually went wrong. Can you share what the a sample json document looks like?
Actually, I'm able to push sample CSV data into Elasticsearch which was included in this document: Load CSV Data Into Elasticsearch Using Python | by Sumukhi CV | Medium
I've pushed the stockerbot.csv which was used in the site as it was given. But I'm trying to push my CSV data into it, and it was showing this error: elasticsearch.helpers.BulkIndexError: 100 document(s) failed to index. If it's fine with you, can you provide me with your email, so that I can send you the CSV file that I'm trying to push?
Please post it here so others can benefit from it in the future. If the file is too large post it on some site like Pastebin and provide a link here. It would help if you could provide a sample JSON document that you are sending as well as the full error message.
Here is the link of the code, the csv file that was there in the link and the csv file that I want to push into Elasticsearch.
https://drive.google.com/drive/folders/1GlIfjoqhDry598qeaug5pIH6YQXM8gG9?usp=sharing
That does not seem to be available to everyone. Please post your code and sample data here so others can help if needed.
The code that I wrote:
from elasticsearch import Elasticsearch, helpers
import configparser
import csv
config = configparser.ConfigParser()
config.read('D:/Pushing data/push/example.ini')
es = Elasticsearch( cloud_id=config['ELASTIC']['cloud_id'], basic_auth=(config['ELASTIC']['user'], config['ELASTIC']['password']))
print(es.info())
with open('stockerbot-export.csv',encoding="UTF-8") as f:
reader = csv.DictReader(f)
helpers.bulk(es, reader, index='stock')
The sample data that I'm pushing is included in the link here:
The main data that I want to push in the Elastic Cloud is in the drive folder that I shared. I think it is visible to everyone:
https://drive.google.com/drive/folders/1GlIfjoqhDry598qeaug5pIH6YQXM8gG9?usp=sharing
Comment out the Elasticsearch parts and check what the CSV reader creates from the file you specified. Format the first few lines as JSON and print them to the screen. Then try to index them using curl or the Kibana and see if that works or see what the error is.
I am sorry Christian but I am not getting the hang of it. I'm entirely new to the software side and especially Elasticsearch and Kibana so I don't know curl and how it works. If possible, can you specify it more clearly or else can you solve it?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.