Bulk Load Large Spatial Datasets


(brian) #1

Hello.

I'm trying to bulk load about 550k records with spatial data into
ElasticSearch. After about 20 mins, an error occurs "No Handlers Can Be
Found For Logger elasticsearch', then the connection times out and the
Python scripts stops.

The Python loading script was working fine before adding the spatial data.

Anyone have some ideas on how to load large spatial datasets?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c2866bc2-f511-45ef-b780-3c7e275da3b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Honza Král) #2

Hi Brian,

that message you are seeing is not an error - it's a warning from the
python logging system that you don't have any logging configured. So
when elasticsearch tries to log something it cannot.

I'd suggest to set up your logging and try again. To set up logging
just include:

import logging
logging.basicConfig(level=logging.INFO)

at the top of your script.

On Mon, May 12, 2014 at 6:21 PM, Brian Behling brian.behling@gmail.com wrote:

Hello.

I'm trying to bulk load about 550k records with spatial data into
ElasticSearch. After about 20 mins, an error occurs "No Handlers Can Be
Found For Logger elasticsearch', then the connection times out and the
Python scripts stops.

The Python loading script was working fine before adding the spatial data.

Anyone have some ideas on how to load large spatial datasets?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c2866bc2-f511-45ef-b780-3c7e275da3b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABfdDipHG3g9k%3D0i93iO1g5%3D1Pgi7GcLF3JMLgfY9dX8%3DSZ-bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(brian) #3

Thank you.

I did find the error causing this script to crash. Looks like there many
invalid polygons (self intersecting) that is causing this problem. But this
should be for another thread if the business rules dictate we can't
simplify the geometries.

On Monday, May 12, 2014 10:24:24 AM UTC-6, Honza Král wrote:

Hi Brian,

that message you are seeing is not an error - it's a warning from the
python logging system that you don't have any logging configured. So
when elasticsearch tries to log something it cannot.

I'd suggest to set up your logging and try again. To set up logging
just include:

import logging
logging.basicConfig(level=logging.INFO)

at the top of your script.

On Mon, May 12, 2014 at 6:21 PM, Brian Behling <brian....@gmail.com<javascript:>>
wrote:

Hello.

I'm trying to bulk load about 550k records with spatial data into
ElasticSearch. After about 20 mins, an error occurs "No Handlers Can Be
Found For Logger elasticsearch', then the connection times out and the
Python scripts stops.

The Python loading script was working fine before adding the spatial
data.

Anyone have some ideas on how to load large spatial datasets?

--
You received this message because you are subscribed to the Google
Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/c2866bc2-f511-45ef-b780-3c7e275da3b6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/56a80e65-cc36-4388-ac07-0f4a228922f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4