Csv files from FIlebeat to Elasticsearch

dadoonet · April 23, 2020, 3:23pm

Welcome!

Logstash is an ETL (Extract Transform and Load) tool. Used for complex use cases. You don't need it here. You can directly stream the CSV to Elasticsearch with filebeat and use https://www.elastic.co/guide/en/elasticsearch/reference/7.6/csv-processor.html to transform the data.
Here is one:

dadoonet/bano-elastic/blob/master/filebeat-config/filebeat-cloud.yml

filebeat.inputs:
- type: log
  paths:
    - /bano-data/bano-*.csv

output.elasticsearch:
  indices:
    - index: "bano"
  pipeline: bano

And the associated ingest pipeline:

github.com

dadoonet/bano-elastic/blob/master/cloud/ingest-bano.json

{
  "processors": [
    {
      "csv": {
        "field": "message",
        "target_fields": [
          "_id",
          "address.number",
          "address.street_name",
          "address.zipcode",
          "address.city",
          "source",
          "location.lat",
          "location.lon"
        ]
      }
    },
    {
      "convert": {
        "field": "location.lat",

This file has been truncated. show original

Topic		Replies	Views
Configuring filebeat to transport csv Logstash	3	903	December 12, 2019
How to process csv data from multiple folders using file beat and create separate index and move it to elastic search Elasticsearch	2	464	January 19, 2022
Filebeat+CSV Beats filebeat	4	1063	January 16, 2019
Ingesting CSV reports through Filebeat to Logstash Logstash	21	7559	July 31, 2018
Shipping CSV file using Filebeat to Logstash and then from logstash to Elasticsearch Logstash	3	8182	July 6, 2017

Csv files from FIlebeat to Elasticsearch

Related topics