How to import a GeoJSON to elastic.. using Logstash or any other possible way

Elastic Stack Elasticsearch
1

My sample GeoJSON file..

"type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "FIDindex": 6035
      },
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -79.51092606323992,
              48.344613038602226
            ],
            [
              -79.5331359417738,
              48.330644012951296
            ],
            [
              -79.54767721776007,
              48.34385594693975
            ],
            [
              -79.54098581498502,
              48.34806594628507
            ],
            [
              -79.51092606323992,
              48.344613038602226
            ]
          ]
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "FIDindex": 6036
      },
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -79.49765416107279,
              48.50665405729244
            ],
            [
              -79.50452010147922,
              48.497367987214574
            ],
            [
              -79.52998110253995,
              48.49616195107807
            ],
            [
              -79.51270010258166,
              48.52528298266071
            ],
            [
              -79.49765416107279,
              48.50665405729244
            ]
          ]
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "FIDindex": 6037
      },
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -79.4937419793212,
              49.08821604910215
            ],
            [
              -79.52318087215492,
              49.02943208366772
            ],
            [
              -79.55202111684834,
              49.03141796850029
            ],
            [
              -79.56778415808171,
              49.056759963348156
            ],
            [
              -79.4937419793212,
              49.08821604910215
            ]
          ],
          [
            [
              -79.5365910174132,
              49.04234909537448
            ],
            [
              -79.53981414365047,
              49.04198099108133
            ],
            [
              -79.5416999721036,
              49.04055804587344
            ],
            [
              -79.538320178897,
              49.04091392507655
            ],
            [
              -79.5365910174132,
              49.04234909537448
            ]
          ],
          [
            [
              -79.53314181028657,
              49.047972036859456
            ],
            [
              -79.54218399465174,
              49.04691700341087
            ],
            [
              -79.54466192708877,
              49.04621599583252
            ],
            [
              -79.53695381365907,
              49.044765936954086
            ],
            [
              -79.53314181028657,
              49.047972036859456
            ]
          ],
          [
            [
              -79.52812883212242,
              49.050876877000356
            ],
            [
              -79.53206385269057,
              49.051771037330965
            ],
            [
              -79.5346651964943,
              49.05115596741972
            ],
            [
              -79.5336059608879,
              49.05080696502161
            ],
            [
              -79.52812883212242,
              49.050876877000356
            ]
          ],
          [
            [
              -79.5202589809016,
              49.05320498871566
            ],
            [
              -79.52311796828306,
              49.0535590070871
            ],
            [
              -79.5249519759151,
              49.0516320071246
            ],
            [
              -79.52163902727364,
              49.05194802154083
            ],
            [
              -79.5202589809016,
              49.05320498871566
            ]
          ]
        ]
      }
    },

..... goes on like this ..

am trying to index this in Kibana..

It is a Shapefile .. which I converted to GeoJSON .. I dint upload it directly on Kibana because of the size limitation and also I have this GeoJSON size(7 MB) still takes forever to index it when uploaded as GeoJSON in Kibana maps directly.. Looking for a best way to load this GeoJSON on to kibana/elastic

How can this be done using Logstash / elastic ?

2

If you are using 7.3 or 7.4 of the default distribution you could try using the GeoJSON Upload feature in the Maps application. https://www.elastic.co/guide/en/kibana/current/geojson-upload.html

3

@nickpeihl I am using 7.3 version and my GeoJSON file size is 7MB it shows on the map but it takes forever to index. And also I have a GeoJSON of file size 30MB it doesn't show on the Map either and the progress bar shows loading... So I was wondering what could be done. In the first case while loading the 7MB file .. it doesn't throws this error

{
  "success": false,
  "failures": [],
  "docCount": 0,
  "error": {
    "displayName": "RequestTimeout",
    "message": "Request Timeout after 30000ms"
  }
}

And also what if the GeoJSON size it greater than 50MB, how to ingest it ?

4

@nickpeihl Can you help me to figure this out.

Thanks in advance!

5

Strange the GeoJSON Upload isn't working for you. Perhaps the 7.4 release of Elasticsearch and Kibana might work better.

For files larger than 50MB, you might use GDAL. GDAL has an output driver for Elasticsearch. However, the current release (v3.0.2) does not work with Elasticsearch 7. Support for Elasticsearch 7+ was added in the master branch. So you can build GDAL from source or use one of the nightly docker images if you have Docker installed.

I have some examples here for using GDAL with Elasticsearch: https://gist.github.com/nickpeihl/1a8f9cbecc78e9e04a73a953b30da84d

6

Hi @nickpeihl, I have tried using GDAL and I got this error

image
image2806×224 160 KB

ERROR 1: HTTP error code : 400
ERROR 1: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."}],"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."},"status":400}
ERROR 1: Unable to write feature 0 from layer quebec.
ERROR 1: Terminating translation prematurely after failed
translation of layer quebec (use -skipfailures to skip errors)

7

Hi @Jayasri_Cimba, I believe this error is caused by an older version of GDAL that does not support Elasticsearch 7. Can you paste the output of ogr2ogr --version?

GDAL 3.1.0 is required to ingest into Elasticsearch 7. But GDAL 3.1.0 is not released yet. So the best way to get GDAL to support Elasticsearch 7 is by building from source. This may look daunting, but should be relatively easy on most operating systems.

If you have Docker installed on your local machine, you can also try using the nightly Docker images of GDAL (examples here).

I have also written a detailed blog post that contains most of the information I have shared with you.

8

Hi @nickpeihl
I have gdal installed on my MACOS its version 2.4.2

I have tried using this command

ogr2ogr -lco INDEX_NAME=gdal-data -lco NOT_ANALYZED_FIELDS={ALL} -lco WRITE_MAPPING=./mapping.json ES:http://localhost:9200 my_shapefile.shp

to create a mapping file and I get this in the output file

{ "FeatureCollection": { "properties": { "type": { "type": "text" }, "properties": { "properties": { "Lower": { "type": "double" }, "Upper": { "type": "double" } } }, "geometry": { "type": "geo_shape" } }, "_meta": { "fid": "ogc_fid", "geomfields": { "geometry": "POLYGON" } } } }

So, Once after I get the map.json what is the next step? How can I see the index on Kibana?

10

Hi @Jayasri_Cimba,

Unfortunately, GDAL version 2.4.2 will not work with your Elasticsearch 7.3 version. You will have to upgrade to GDAL 3.1.0 by building from the source code.

I also have a Mac and this is how I build GDAL from source.

  1. Clone the gdal repository to my computer
  2. cd into the gdal subdirectory of the repository
  3. Run ./configure
  4. Run make (this will take a long time)
  5. Run make install
  6. Run ogr2ogr --version to confirm I am running 3.1.0-dev-*.
11

Hi @nickpeihl , I have installed gdal-3.0.2 and tried to do this

jayasri-cimba:Users jayasri$ ogr2ogr ES:http://localhost:9200 /Users/jayasri/quebec.shp
ERROR 1: HTTP error code : 400
ERROR 1: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."}],"type":"illegal_argument_exception","reason":"Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."},"status":400}
ERROR 1: Unable to write feature 0 from layer quebec.
ERROR 1: Terminating translation prematurely after failed
translation of layer quebec (use -skipfailures to skip errors)

jayasri-cimba:Users jayasri$ ogr2ogr --version
GDAL 3.0.2, released 2019/10/28

Still it throws the same error. Am not sure what to be done.

12

Hi @Jayasri_Cimba,

GDAL 3.0.2 is not compatible with Elasticsearch 7. Please follow the steps above to build GDAL 3.1.0 from the source code.

13

Hi @nickpeihl..

Thank you so much. It fixed all my issues. Appreciating your patience :smiley:
Thanks once again

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.