Logstash. Export data from Elasticsearch's nested field

Elastic Stack Logstash
1

Hi all !

I'am trying to export data from ElasticSearch into csv-file.
I installed plugins "logstash-input-elasticsearch", "logstash-output-csv" before.

Part of my ES data scheme:

mappings: {
      properties: {
        books: {
          type: 'nested',
          properties: {
            bookId: { type: 'short' },
            bookTitle: { type: 'text'},
            bookAuthor: {type: 'text'}
          }
        }

I want to receive "bookId","bookTitle","bookAuthor" separately.

Now I can access only top-level field "book".
Part of my current logstash configuration file:

output {
  csv {
    # elastic field name
    fields => ["books"]
    # This is path where we store output.   
    path => "/Users/user/Desktop/csv-export.csv"
  }
}

I tried many different options, for example "[books][bookId]", ["books"]["bookId"] or ["books"."bookId"] , but this does not work.
Would be grateful for any advice.

// Using Logstash ver. 7.8.0 and ElasticSearch ver. 7.7.0

2

Hi,

Correct syntax for nested fields :

"[books][bookid]"

Tell me if it is working !

3

@grumo35, checked your suggestion.
Unfortunately it doesn't work.
Thanks for the idea :slightly_smiling_face:

Although the documentation for the plugin ("logstash-output-csv") really does specify this syntax:
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-csv.html#plugins-outputs-csv-fields

4

Oh sorry i thought the top field was properties,

"[books][bookid]" should work ??

5

@grumo35, unfortunately no )
I guess that the problem is that I use a non-standard data source - ElasticSearch (through the corresponding plugin).
Typically, an ES is a data receiver.
Probably the plugin "logstash-output-csv" can only parse simple json objects, not ES documents.

Will try to find a workaround :slightly_smiling_face:

6

Ok final try

what's the output like with ?

filter{
  json{
    source => "books"
  }

}

Tough day today, i failed to fully understand your problem.

We can sort this out !

7

@grumo35, now I have some errors with json parsing.
But the error description contains information about nested attributes.
So, I guess now I have access to nested field "books"
and have to change my output block in logstash config file.

Will try to fix it ) Thanks.

8

JFYI.
Solved the problem using the Python client.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.