Logstash. Export data from Elasticsearch's nested field

[Alex_D]

Hi all !

I'am trying to export data from ElasticSearch into csv-file.
I installed plugins "logstash-input-elasticsearch", "logstash-output-csv" before.

Part of my ES data scheme:

mappings: {
      properties: {
        books: {
          type: 'nested',
          properties: {
            bookId: { type: 'short' },
            bookTitle: { type: 'text'},
            bookAuthor: {type: 'text'}
          }
        }

I want to receive "bookId","bookTitle","bookAuthor" separately.

Now I can access only top-level field "book".
Part of my current logstash configuration file:

output {
  csv {
    # elastic field name
    fields => ["books"]
    # This is path where we store output.   
    path => "/Users/user/Desktop/csv-export.csv"
  }
}

I tried many different options, for example "[books][bookId]", ["books"]["bookId"] or ["books"."bookId"] , but this does not work.
Would be grateful for any advice.

// Using Logstash ver. 7.8.0 and ElasticSearch ver. 7.7.0

[grumo35]

Hi,

Correct syntax for nested fields :

"[books][bookid]"

Tell me if it is working !

[Alex_D]

@grumo35, checked your suggestion.
Unfortunately it doesn't work.
Thanks for the idea

Although the documentation for the plugin ("logstash-output-csv") really does specify this syntax:
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-csv.html#plugins-outputs-csv-fields

[grumo35]

Oh sorry i thought the top field was properties,

"[books][bookid]" should work ??

[Alex_D]

@grumo35, unfortunately no )
I guess that the problem is that I use a non-standard data source - ElasticSearch (through the corresponding plugin).
Typically, an ES is a data receiver.
Probably the plugin "logstash-output-csv" can only parse simple json objects, not ES documents.

Will try to find a workaround

[grumo35]

Ok final try

what's the output like with ?

filter{
  json{
    source => "books"
  }

}

Tough day today, i failed to fully understand your problem.

We can sort this out !

[Alex_D]

@grumo35, now I have some errors with json parsing.
But the error description contains information about nested attributes.
So, I guess now I have access to nested field "books"
and have to change my output block in logstash config file.

Will try to fix it ) Thanks.

[Alex_D]

JFYI.
Solved the problem using the Python client.