# How to parse csv via Elastic Agent?

**URL:** <https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121>\
**Category:** Elastic Agent\
**Created:** [January 27, 2023, 3:43pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121 "2023-01-27T15:43:58Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![test\_qweqwe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_qweqwe/32/94750_2.png) [@test\_qweqwe](https://discuss.elastic.co/u/test_qweqwe)\
**Post date:** [January 27, 2023, 3:43pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/1 "2023-01-27T15:43:58Z")

</div>

Hi!  
I want to collects csv logs and if I understand correct, I need to add new integration based on "Custom Logs"?  
I'm not sure how to do it.

I have this config for logstash under conf.d folder and everything work fine:

```auto
input {
        file {
                path => "/opt/nsm/*"
                start_position => "beginning"
                sincedb_path => "/dev/null"
        }
}

filter {
        csv {
                separator => ","
                columns => ["Score, Source IP, Destination IP, Connections, Avg. Bytes, Total Bytes, TS Score, DS Score, Dur Score, Hist Score, Top Intvl"]
        }
}

output {
        elasticsearch {
                hosts => ["http://192.168.0.108:9200"]
        {
}

```

But now I need to put this config (or rewrite) somehow into Elastic Agent. How can I do it?

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [January 27, 2023, 4:09pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/2 "2023-01-27T16:09:25Z")

</div>

Related, but nearly a year old: [How to integrate custom logs with Elastic Agent | by Benoit Luttringer | Zenika](https://medium.zenika.com/how-to-integrate-custom-logs-with-elastic-agent-7f80aef5aec7)

---

<div class="post-metadata">

**Author:** ![test\_qweqwe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_qweqwe/32/94750_2.png) [@test\_qweqwe](https://discuss.elastic.co/u/test_qweqwe)\
**Post date:** [January 27, 2023, 4:21pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/3 "2023-01-27T16:21:14Z")

</div>

I saw this guide, but it about how to parse rows in log file, but I need a csv.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 27, 2023, 4:22pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/4 "2023-01-27T16:22:49Z")

</div>

It is the same approach, you just need to use the correct processor.

Check the documentation for the [CSV processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/csv-processor.html).

---

<div class="post-metadata">

**Author:** ![test\_qweqwe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_qweqwe/32/94750_2.png) [@test\_qweqwe](https://discuss.elastic.co/u/test_qweqwe)\
**Post date:** [January 27, 2023, 4:25pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/5 "2023-01-27T16:25:39Z")

</div>

I used it, but it not parsed anything.

---

<div class="post-metadata">

**Author:** ![test\_qweqwe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_qweqwe/32/94750_2.png) [@test\_qweqwe](https://discuss.elastic.co/u/test_qweqwe)\
**Post date:** [January 27, 2023, 4:38pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/6 "2023-01-27T16:38:49Z")

</div>

@leandrojmp I tried "Test pipeline" and I see that it parsed it, but in KIbana "Discover" tab I see non-parsed events.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 27, 2023, 7:51pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/7 "2023-01-27T19:51:47Z")

</div>

You need to provide context, what is the ingest pipeline you are using? Please share it.

What is the output you have in Elasticsearch? Copy the json part you have in discover and share it.

It is not possible to know what could be the issue without seeing your data and the output you are getting.

---

<div class="post-metadata">

**Author:** ![test\_qweqwe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_qweqwe/32/94750_2.png) [@test\_qweqwe](https://discuss.elastic.co/u/test_qweqwe)\
**Post date:** [January 30, 2023, 2:53pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/8 "2023-01-30T14:53:36Z")

</div>

JSON output in Discover

> **Example 1**
>
> ```auto
> {
> "_index": ".ds-logs-generic-default-2023.01.27-000001",
> "_id": "b7JV84UBXqgfEVa3cNbf",
> "_version": 1,
> "_score": 0,
> "_source": {
> "@timestamp": "2023-01-27T13:05:39.737Z",
> "host": {
> "mac": [
> "00-0C-29-B1-4F-7B"
> ],
> "name": "nsm",
> "hostname": "nsm",
> "architecture": "x86_64",
> "os": {
> "family": "debian",
> "name": "Ubuntu",
> "kernel": "5.4.0-137-generic",
> "codename": "focal",
> "type": "linux",
> "platform": "ubuntu",
> "version": "20.04.5 LTS (Focal Fossa)"
> },
> "id": "432ef884cea7488bb03fee5d5f1b56a9",
> "containerized": false,
> "ip": [
> "192.168.0.108",
> "fe80::20c:29ff:feb1:4f7b"
> ]
> },
> "event": {
> "dataset": "generic"
> },
> "elastic_agent": {
> "version": "8.6.0",
> "id": "7f80d869-90d0-4dd7-b991-c5c68f993a50",
> "snapshot": false
> },
> "agent": {
> "id": "7f80d869-90d0-4dd7-b991-c5c68f993a50",
> "type": "filebeat",
> "version": "8.6.0",
> "ephemeral_id": "e7d1928e-09d8-4770-8c7f-e3090eb480d2",
> "name": "nsm"
> },
> "ecs": {
> "version": "8.0.0"
> },
> "input": {
> "type": "log"
> },
> "message": "0.279,192.168.137.88,34.95.113.255,25,644,16119,0.424,0.667,0.025,0,0",
> "data_stream": {
> "dataset": "generic",
> "namespace": "default",
> "type": "logs"
> },
> "log": {
> "offset": 327,
> "file": {
> "path": "/opt/nsm/1.log"
> }
> }
> },
> "fields": {
> "elastic_agent.version": [
> "8.6.0"
> ],
> "host.hostname": [
> "nsm"
> ],
> "host.mac": [
> "00-0C-29-B1-4F-7B"
> ],
> "host.ip": [
> "192.168.0.108",
> "fe80::20c:29ff:feb1:4f7b"
> ],
> "agent.type": [
> "filebeat"
> ],
> "host.os.version": [
> "20.04.5 LTS (Focal Fossa)"
> ],
> "host.os.kernel": [
> "5.4.0-137-generic"
> ],
> "host.os.name": [
> "Ubuntu"
> ],
> "agent.name": [
> "nsm"
> ],
> "host.name": [
> "nsm"
> ],
> "elastic_agent.snapshot": [
> false
> ],
> "host.id": [
> "432ef884cea7488bb03fee5d5f1b56a9"
> ],
> "host.os.type": [
> "linux"
> ],
> "elastic_agent.id": [
> "7f80d869-90d0-4dd7-b991-c5c68f993a50"
> ],
> "data_stream.namespace": [
> "default"
> ],
> "host.os.codename": [
> "focal"
> ],
> "input.type": [
> "log"
> ],
> "log.offset": [
> 327
> ],
> "message": [
> "0.279,192.168.137.88,34.95.113.255,25,644,16119,0.424,0.667,0.025,0,0"
> ],
> "data_stream.type": [
> "logs"
> ],
> "host.architecture": [
> "x86_64"
> ],
> "@timestamp": [
> "2023-01-27T13:05:39.737Z"
> ],
> "agent.id": [
> "7f80d869-90d0-4dd7-b991-c5c68f993a50"
> ],
> "host.os.platform": [
> "ubuntu"
> ],
> "ecs.version": [
> "8.0.0"
> ],
> "host.containerized": [
> false
> ],
> "log.file.path": [
> "/opt/nsm/1.log"
> ],
> "data_stream.dataset": [
> "generic"
> ],
> "agent.ephemeral_id": [
> "e7d1928e-09d8-4770-8c7f-e3090eb480d2"
> ],
> "agent.version": [
> "8.6.0"
> ],
> "host.os.family": [
> "debian"
> ],
> "event.dataset": [
> "generic"
> ]
> }
> }
> 
> ```

Custom Logs Elastic Agent config:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/3/c324d72396d4a80295c8d19f1ae0c94f2967a94b.png)

> **Full config**
>
> ```auto
> {
> "csv": {
> "field": "message",
> "target_fields": ["Score", "Source IP", "Destination IP", "Connections", "Avg. Bytes", "Total Bytes", "TS Score",	"DS Score", "Dur Score", "Hist Score", "Top Intvl"]
> }
> }
> 
> ```

> **Result of tested pipeline based on Example 1**
>
> ```auto
> {
> "docs": [
> {
> "doc": {
> "_index": ".ds-logs-generic-default-2023.01.27-000001",
> "_id": "b7JV84UBXqgfEVa3cNbf",
> "_version": "-3",
> "_source": {
> "Connections": "25",
> "agent": {
> "name": "nsm",
> "id": "7f80d869-90d0-4dd7-b991-c5c68f993a50",
> "type": "filebeat",
> "ephemeral_id": "e7d1928e-09d8-4770-8c7f-e3090eb480d2",
> "version": "8.6.0"
> },
> "Total Bytes": "16119",
> "log": {
> "file": {
> "path": "/opt/nsm/1.log"
> },
> "offset": 327
> },
> "elastic_agent": {
> "id": "7f80d869-90d0-4dd7-b991-c5c68f993a50",
> "version": "8.6.0",
> "snapshot": false
> },
> "Destination IP": "34.95.113.255",
> "Dur Score": "0.025",
> "message": "0.279,192.168.137.88,34.95.113.255,25,644,16119,0.424,0.667,0.025,0,0",
> "input": {
> "type": "log"
> },
> "DS Score": "0.667",
> "Score": "0.279",
> "TS Score": "0.424",
> "@timestamp": "2023-01-27T13:05:39.737Z",
> "Avg": {
> " Bytes": "644"
> },
> "ecs": {
> "version": "8.0.0"
> },
> "data_stream": {
> "namespace": "default",
> "type": "logs",
> "dataset": "generic"
> },
> "host": {
> "hostname": "nsm",
> "os": {
> "kernel": "5.4.0-137-generic",
> "codename": "focal",
> "name": "Ubuntu",
> "family": "debian",
> "type": "linux",
> "version": "20.04.5 LTS (Focal Fossa)",
> "platform": "ubuntu"
> },
> "containerized": false,
> "ip": [
> "192.168.0.108",
> "fe80::20c:29ff:feb1:4f7b"
> ],
> "name": "nsm",
> "id": "432ef884cea7488bb03fee5d5f1b56a9",
> "mac": [
> "00-0C-29-B1-4F-7B"
> ],
> "architecture": "x86_64"
> },
> "Source IP": "192.168.137.88",
> "Hist Score": "0",
> "event": {
> "dataset": "generic"
> },
> "Top Intvl": "0"
> },
> "_ingest": {
> "timestamp": "2023-01-30T14:50:14.581791334Z"
> }
> }
> }
> ]
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 30, 2023, 3:13pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/9 "2023-01-30T15:13:10Z")

</div>

You did not show how you set up the Custom Logs integration...where / how did you do specify the ingest pipeline.?

What version are you using?

---

<div class="post-metadata">

**Author:** ![test\_qweqwe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_qweqwe/32/94750_2.png) [@test\_qweqwe](https://discuss.elastic.co/u/test_qweqwe)\
**Post date:** [January 30, 2023, 9:16pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/10 "2023-01-30T21:16:43Z")

</div>

Hi @stephenb!  
Thank for comment.  
My version is ELK 8.6.1.

> how you set up the Custom Logs integration

Integrations \> Custom logs \> Add Custom Logs integration \>  
**Custom log file = /opt/nsm/** \* (path where all my csv stores)  
**Processors** =

```auto
{
  "csv": {
    "field": "message",
    "target_fields": ["Score", "Source IP", "Destination IP", "Connections", "Avg. Bytes", "Total Bytes", "TS Score",	"DS Score", "Dur Score", "Hist Score", "Top Intvl"]
  }
}

```

> how did you do specify the ingest pipeline.?

![image](https://us1.discourse-cdn.com/elastic/original/3X/8/8/88c7bfbcd110ce1bdd43e5f077f8902aa3046f58.png)  
I modify **logs-log.log@custom**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/d/5d0882c80706bc194134a5e91725fb5eb7d08c2c.png)

And it linked to my elastic agent and host

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af66e19e85627afd2e62c3c6bead0e390c2ffe3d.png)

I didn't do anything more. Maybe I miss something? I'm new to ELK.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 30, 2023, 9:42pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/11 "2023-01-30T21:42:32Z")

</div>

@test_qweqwe you look like you are on the right path.

When you look in Discover do you see any of the logs from the files in `/opt/nsm/`

Perhaps the agent read all the files _before_ you added the pipeline?

---

<div class="post-metadata">

**Author:** ![test\_qweqwe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_qweqwe/32/94750_2.png) [@test\_qweqwe](https://discuss.elastic.co/u/test_qweqwe)\
**Post date:** [January 31, 2023, 10:11am UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/12 "2023-01-31T10:11:50Z")

</div>

Yes, I see logs from /opt/nsm/, but they not parsed.  
All new files still not parsed after I added pipeline and restarted ELK.

Few sec ago event:

```auto
{
  "_index": ".ds-logs-generic-default-2023.01.27-000001",
  "_id": "gO9QB4YBxWb1fMfgEZty",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2023-01-31T10:12:12.610Z",
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "generic"
    },
    "event": {
      "dataset": "generic"
    },
    "elastic_agent": {
      "version": "8.6.0",
      "id": "7f80d869-90d0-4dd7-b991-c5c68f993a50",
      "snapshot": false
    },
    "log": {
      "file": {
        "path": "/opt/nsm/13.log"
      },
      "offset": 115
    },
    "message": "0.839,192.168.0.108,185.125.190.58,30,76,2280,1,0.667,0.688,1,2048",
    "agent": {
      "type": "filebeat",
      "version": "8.6.0",
      "ephemeral_id": "40f0d334-e31a-49b5-b36b-bb795f5b4b69",
      "id": "7f80d869-90d0-4dd7-b991-c5c68f993a50",
      "name": "nsm"
    },
    "ecs": {
      "version": "8.0.0"
    },
    "input": {
      "type": "log"
    },
    "host": {
      "mac": [
        "00-0C-29-B1-4F-7B"
      ],
      "hostname": "nsm",
      "architecture": "x86_64",
      "os": {
        "codename": "focal",
        "type": "linux",
        "platform": "ubuntu",
        "version": "20.04.5 LTS (Focal Fossa)",
        "family": "debian",
        "name": "Ubuntu",
        "kernel": "5.4.0-137-generic"
      },
      "id": "432ef884cea7488bb03fee5d5f1b56a9",
      "containerized": false,
      "name": "nsm",
      "ip": [
        "192.168.0.108",
        "fe80::20c:29ff:feb1:4f7b"
      ]
    }
  },
  "fields": {
    "elastic_agent.version": [
      "8.6.0"
    ],
    "host.hostname": [
      "nsm"
    ],
    "host.mac": [
      "00-0C-29-B1-4F-7B"
    ],
    "host.ip": [
      "192.168.0.108",
      "fe80::20c:29ff:feb1:4f7b"
    ],
    "agent.type": [
      "filebeat"
    ],
    "host.os.version": [
      "20.04.5 LTS (Focal Fossa)"
    ],
    "host.os.kernel": [
      "5.4.0-137-generic"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "agent.name": [
      "nsm"
    ],
    "elastic_agent.snapshot": [
      false
    ],
    "host.name": [
      "nsm"
    ],
    "host.id": [
      "432ef884cea7488bb03fee5d5f1b56a9"
    ],
    "host.os.type": [
      "linux"
    ],
    "elastic_agent.id": [
      "7f80d869-90d0-4dd7-b991-c5c68f993a50"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "host.os.codename": [
      "focal"
    ],
    "input.type": [
      "log"
    ],
    "log.offset": [
      115
    ],
    "message": [
      "0.839,192.168.0.108,185.125.190.58,30,76,2280,1,0.667,0.688,1,2048"
    ],
    "data_stream.type": [
      "logs"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "@timestamp": [
      "2023-01-31T10:12:12.610Z"
    ],
    "agent.id": [
      "7f80d869-90d0-4dd7-b991-c5c68f993a50"
    ],
    "host.containerized": [
      false
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "data_stream.dataset": [
      "generic"
    ],
    "log.file.path": [
      "/opt/nsm/13.log"
    ],
    "agent.ephemeral_id": [
      "40f0d334-e31a-49b5-b36b-bb795f5b4b69"
    ],
    "agent.version": [
      "8.6.0"
    ],
    "host.os.family": [
      "debian"
    ],
    "event.dataset": [
      "generic"
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 31, 2023, 4:00pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/13 "2023-01-31T16:00:48Z")

</div>

My next Suggestion is to add a simple set processor into the custom pipeline with a field and string to see if your ingest pipeline is executed at all.

Also, did you set `"ignore_failure": true` for the CSV processor?

---

<div class="post-metadata">

**Author:** ![test\_qweqwe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_qweqwe/32/94750_2.png) [@test\_qweqwe](https://discuss.elastic.co/u/test_qweqwe)\
**Post date:** [February 9, 2023, 10:18pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/14 "2023-02-09T22:18:02Z")

</div>

Hi @stephenb  
Sorry for long reply.  
I added "ignore\_failure": true to CSV processor, but it doesn't helped.

> My next Suggestion is to add a simple set processor into the custom pipeline with a field and string to see if your ingest pipeline is executed at all.

Can you please share how to do it? Sorry, I'm new to Elastic.

---

<div class="post-metadata">

**Author:** ![rsaeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rsaeks/32/5068_2.png) [@rsaeks](https://discuss.elastic.co/u/rsaeks)\
**Post date:** [February 14, 2023, 7:25pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/15 "2023-02-14T19:25:00Z")

</div>

Following along ... I'm working on ingesting a custom file with data formatted in a CSV format and have been unable to ingest data as well. In the past I've done this through logstash and with the Elastic Agents I've been unsuccessful.

(The agent doesn't even seem to be ingesting the log file)

---

<div class="post-metadata">

**Author:** ![test\_qweqwe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_qweqwe/32/94750_2.png) [@test\_qweqwe](https://discuss.elastic.co/u/test_qweqwe)\
**Post date:** [February 14, 2023, 8:32pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/16 "2023-02-14T20:32:49Z")

</div>

> have been unable to ingest data as well

Do you mean that there was some ingestion, or no results at all?

I also thought about an alternative, such as converting the csv output to json format and then to Elastic Agent, but my competence in sprinting is not enough.

---

<div class="post-metadata">

**Author:** ![rsaeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rsaeks/32/5068_2.png) [@rsaeks](https://discuss.elastic.co/u/rsaeks)\
**Post date:** [February 14, 2023, 9:05pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/17 "2023-02-14T21:05:22Z")

</div>

It is looking like there is some security issue for mine:

`{"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [8cPGg4UBytBGNw1dSsxz] of user [elastic/fleet-server] on indices [logs-connectedclients-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}, dropping event!`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 14, 2023, 9:06pm UTC](https://discuss.elastic.co/t/how-to-parse-csv-via-elastic-agent/324121/18 "2023-03-14T21:06:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.