How to upload csv files with high speed to Elasticsearch ( bulk copy , block copy )

siyavusak · April 10, 2021, 9:50am

How to upload csv files with high speed / best performance to Elasticsearch ( bulk copy , block copy ) ?
we have got csv format files every with 1TB size .

regards

dadoonet · April 10, 2021, 11:24am

Welcome!

In this talk, I'm using filebeat for that with some ingest processors to extract from each line the meaningful information.

Filebeat config

github.com

dadoonet/bano-elastic/blob/master/filebeat-config/filebeat-all.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /bano-data/bano-*.csv

output.elasticsearch:
  indices:
    - index: "bano-foo"
  pipeline: bano

Ingest pipeline

github.com

dadoonet/bano-elastic/blob/master/elasticsearch-config/ingest-bano.json

{
  "processors": [
    {
      "csv": {
        "field": "message",
        "target_fields": [
          "_id",
          "address.number",
          "address.street_name",
          "address.zipcode",
          "address.city",
          "source",
          "location.lat",
          "location.lon"
        ]
      }
    },
    {
      "gsub": {
        "field": "log.file.path",

This file has been truncated. show original

HTH

system · May 8, 2021, 11:25am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.