Logstash - output csv plugin - all fields are strings

Elastic Stack Logstash
1

i'm trying to move some data to mongodb from elastic using elastic input plugin and csv ouput plugin and then mongoimport.

how can i define the fields types , including nested fields?
everything is showing up as strings in the csv output -> mongo
here is an example of _source data:

{
    "id": "uf02fh-ch29ufh-hd289guf-hf92ui",
    "platform": {
      "description": "Chrome 90.0.4430.19 on OS X 10.14.6 64-bit",
      "layout": "Blink",
      "manufacturer": null,
      "os": {
        "architecture": 64,
        "family": "OS X",
        "version": "10.14.6"
      }
    },
    "videos": [
      {
        "filename": "1.mpeg"
      },
      {
        "filename": "2.mpeg"
      }
    ],
    "completed": false,
    "createdAt": "2021-03-15T14:39:56.059Z",
    "brightnessLevel": [
      106.56687102472326
    ],
    "gdprTimestamp": "2021-03-15T14:39:57.468Z",
    "quality": "hd",
    "updatedAt": "2021-03-15T14:40:53.147Z"
}

here is the conf file:

input {
 elasticsearch {
    hosts => "xxx:port"
    user => "elastic"
    password => "password"
    index => "some_index"
    query => '{
    "query": { "match_all": {} },
    "_source": ["id","platform","videos","completed","createdAt","brightnessLevel","gdprTimestamp","quality","updatedAt"]

    }'
    }
  }

output {
  csv {
    # elastic field name
    fields => ["id","platform","videos","completed","createdAt","brightnessLevel","gdprTimestamp","quality","updatedAt"]
    # This is path where we store output.   
    path => "/tmp/csv-export.csv"
  }

}
2

The csv output calls Array::to_csv, which is equivalent to CSV::generate_line, which is defined as returning a string. Anything in the csv file will be a string.

3

Is there a way to avoid that?
What would the best practice to move some index data from elastic to mongo using logstash?

4

You could try using the mongodb output.

5

so i tried with the following conf:

input {
 elasticsearch {
    hosts => "ES:port"
    user => "elastic"
    password => "password"
    index => "some_index"
    query => '{
    "query": { "match_all": {} },
    "_source": ["id","platform","videos","completed","createdAt","brightnessLevel","gdprTimestamp","quality","updatedAt"]
    }'
    }
  }

output {
mongodb {
    id => "my_id"
    collection => "some_collection"
    database => "dbname"
    uri => "mongodb+srv://user:pass@mongo_hostname/dbname?retryWrites=true&w=majority"
    codec => "json"
  }

}

but got this error:
[2021-03-27T12:33:37,286][WARN ][logstash.outputs.mongodb ][main] MONGODB | Failed to handshake with [mongodb shard address]: ArgumentError: wrong number of arguments (given 2, expected 1)

any idea what's wrong in the config ? logstash-output-mongodb (3.1.6)

6

A similar error came up recently here. I do not have a good answer for it. If it threw an exception you could capture the init of that exception in a debugger, if it provided a stack trace you could review the code where it happens. However, we have neither.

7

Thanks @Badger
i was thinking logstash and its plugins are much more stabled and working out of the box.
especially for such basic use cases...
it's a bummer.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.