This is a follow-up to Import from BigQuery to Elasticsearch?
In https://github.com/DataBiosphere/data-explorer-indexers, we're developing a solution that is pretty fast:
- Copy BigQuery table to JSON files on GCS. It only take a couple minutes to load 2G table into python.
- Based on BigQuery schema, supply mappings. Turn off dynamic mappings.
We were able to index a 2G BigQuery table into ES in 4 hours.
If there is interested, we could extract BigQuery -> Elasticsearch into a separate repo.