# Logstash fields from aggregation to csv

**URL:** <https://discuss.elastic.co/t/logstash-fields-from-aggregation-to-csv/306376>\
**Category:** Logstash\
**Created:** [June 4, 2022, 7:39am UTC](https://discuss.elastic.co/t/logstash-fields-from-aggregation-to-csv/306376 "2022-06-04T07:39:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ddcro](https://avatars.discourse-cdn.com/v4/letter/d/bbce88/32.png) [@ddcro](https://discuss.elastic.co/u/ddcro)\
**Post date:** [June 4, 2022, 7:39am UTC](https://discuss.elastic.co/t/logstash-fields-from-aggregation-to-csv/306376/1 "2022-06-04T07:39:45Z")

</div>

Hi guys,

I'm trying to export data from elastic to csv. The conf from logstash is something like:

```auto
input {
   elasticsearch {
           hosts => "127.0.0.1:9200"
           index => "log"
           ...
          query => '
            {
             "size":0,
              "query": {
                     "bool": {
                         "filter": [
                           { "term": {"country.keyword": "NZ"}}
                           ]
                 }
            },
           "aggs": {
               "byhostname": {
                     "terms": { "field": "hostname.keyword" }
              }
              }
  }'
}
}

output {
  csv {
     fields => ["key"] <==????????
     path => "/blabla/some.csv"
  }
}

```

The result from the query is something like:

```auto
.....
"aggregations" : {
    "byhostname" : {
      "doc_count_error_upper_bound" : 111,
      "sum_other_doc_count" : 301002,
      "buckets" : [
        {
          "key" : "site1",
          "doc_count" : 1335
        },
        {
          "key" : "site2",
          "doc_count" : 852
        }
       ................

```

I want to output to csv the aggregated values from buckets-\>"key" (site1, site2) but it doesn't seem to work. I can access the fields from the documents just fine. How shall I specify the field in the output csv section of the logstash?

Thank you

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [June 4, 2022, 3:22pm UTC](https://discuss.elastic.co/t/logstash-fields-from-aggregation-to-csv/306376/2 "2022-06-04T15:22:00Z")

</div>

[Elasticsearch input plugin](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html) is working just for the query and the aggs part may have no meaning.

It is an open issue.

> <https://github.com/logstash-plugins/logstash-input-elasticsearch/issues/58>
>
> Not sure if this would be better supported by a totally different plugin, but I …think it would make sense to support aggregation results.  
> 
> As an example, say we wanted to use a sum aggregation, by terms.
> 
> \`\`\`
> GET myserverlogs\*/\_search
> {
> 
> "size": 0,
> "aggs": {
> "types": {
> "terms": {
> "field": "hostname"
> },
> "aggs": {
> "connections": {
> "sum": {
> "field": "connections"
> }
> }
> }
> }
> }
> }
> \`\`\`
> 
> Results:
> \`\`\`
> "aggregations": {
> "types": {
> "doc\_count\_error\_upper\_bound": 0,
> "sum\_other\_doc\_count": 0,
> "buckets": \[
> {
> "key": "hostA",
> "doc\_count": 90310,
> "total\_connections": {
> "value": 2344
> }
> },
> {
> "key": "hostB",
> "doc\_count": 485,
> "total\_connections": {
> "value": 233
> }
> },
> {
> "key": "hostC",
> "doc\_count": 485,
> "total\_connections": {
> "value": 123
> }
> }
> \]
> }
> }
> \`\`\`
> 
> Then in Logstash we could say something like:
> 
> \`\`\`
> input {
> elasticsearch {
> ...
> results\_mode =\> "aggregation"
> source\_array =\> "types.buckets"
> }
> }
> \`\`\`
> 
> This could even be used to create aggregation based summary indices on a regular interval. I remember @polyfractal was working on an ES plugin that did something similar last year.

One possible option is using Transform in Elasticsearch to store the aggregation data into another index and use Elasticsearch input plugin on that index to retrieve the result of aggregation.

---

<div class="post-metadata">

**Author:** ![ddcro](https://avatars.discourse-cdn.com/v4/letter/d/bbce88/32.png) [@ddcro](https://discuss.elastic.co/u/ddcro)\
**Post date:** [June 6, 2022, 7:02am UTC](https://discuss.elastic.co/t/logstash-fields-from-aggregation-to-csv/306376/3 "2022-06-06T07:02:40Z")

</div>

Thank you for your response. I will try to see if I can make a transform or maybe export the data with a python script.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 4, 2022, 7:03am UTC](https://discuss.elastic.co/t/logstash-fields-from-aggregation-to-csv/306376/4 "2022-07-04T07:03:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.