How to upload csv files with high speed to Elasticsearch ( bulk copy , block copy )

siyavusak · April 10, 2021, 9:50am

How to upload csv files with high speed / best performance to Elasticsearch ( bulk copy , block copy ) ?
we have got csv format files every with 1TB size .

regards

dadoonet · April 10, 2021, 11:24am

Welcome!

In this talk, I'm using filebeat for that with some ingest processors to extract from each line the meaningful information.

Filebeat config

github.com

dadoonet/bano-elastic/blob/master/filebeat-config/filebeat-all.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /bano-data/bano-*.csv

output.elasticsearch:
  indices:
    - index: "bano-foo"
  pipeline: bano

Ingest pipeline

github.com

dadoonet/bano-elastic/blob/master/elasticsearch-config/ingest-bano.json

{
  "processors": [
    {
      "csv": {
        "field": "message",
        "target_fields": [
          "_id",
          "address.number",
          "address.street_name",
          "address.zipcode",
          "address.city",
          "source",
          "location.lat",
          "location.lon"
        ]
      }
    },
    {
      "gsub": {
        "field": "log.file.path",

This file has been truncated. show original

HTH

Topic		Replies	Views
Bulk upload from csv to elasticsearch Elasticsearch	2	854	July 5, 2017
Import (21gb) csv to elasticsearch Elasticsearch	9	658	February 1, 2019
Can we ingest csv directly into elastric search from python? Elasticsearch	15	3844	January 3, 2023
Bulk upload in Elasticsearch 6.8.19 using python Elasticsearch	3	237	May 23, 2023
Envío de archivo CSV a elasticsearch Logstash	2	448	June 7, 2019

How to upload csv files with high speed to Elasticsearch ( bulk copy , block copy )

Filebeat config

Ingest pipeline

Related topics