Csv files from FIlebeat to Elasticsearch

dadoonet · April 23, 2020, 3:23pm

Welcome!

Logstash is an ETL (Extract Transform and Load) tool. Used for complex use cases. You don't need it here. You can directly stream the CSV to Elasticsearch with filebeat and use https://www.elastic.co/guide/en/elasticsearch/reference/7.6/csv-processor.html to transform the data.
Here is one:

dadoonet/bano-elastic/blob/master/filebeat-config/filebeat-cloud.yml

filebeat.inputs:
- type: log
  paths:
    - /bano-data/bano-*.csv

output.elasticsearch:
  indices:
    - index: "bano"
  pipeline: bano

And the associated ingest pipeline:

github.com

dadoonet/bano-elastic/blob/master/cloud/ingest-bano.json

{
  "processors": [
    {
      "csv": {
        "field": "message",
        "target_fields": [
          "_id",
          "address.number",
          "address.street_name",
          "address.zipcode",
          "address.city",
          "source",
          "location.lat",
          "location.lon"
        ]
      }
    },
    {
      "convert": {
        "field": "location.lat",

This file has been truncated. show original

Topic		Replies	Views
Upload csv fle in elasticsearch using filebeat Kibana	2	1100	July 25, 2022
Importing multiple large csv and json files into a single index Elasticsearch	11	1722	April 4, 2023
Is filebeat parse the timestamp from the log line and use it as the timestamp of the event? Beats filebeat	5	889	January 23, 2019
Kibana: What's Next? Kibana	5	5541	May 12, 2017
How to configure different indexes in logstash Logstash	4	965	October 14, 2020

Csv files from FIlebeat to Elasticsearch

Related Topics