Import from BigQuery to Elasticsearch

Elastic Stack Elasticsearch
1

This is a follow-up to Import from BigQuery to Elasticsearch?

In https://github.com/DataBiosphere/data-explorer-indexers, we're developing a solution that is pretty fast:

  • Copy BigQuery table to JSON files on GCS. It only take a couple minutes to load 2G table into python.
  • Based on BigQuery schema, supply mappings. Turn off dynamic mappings.

We were able to index a 2G BigQuery table into ES in 4 hours.

If there is interested, we could extract BigQuery -> Elasticsearch into a separate repo.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.