Painful! postgres-->logstash-->elasticsearch mapper_parsing_exception!


#1

hi, I am so sorry to bother you, but now I am painful and rage about
a problem. the problem has spent more that one weeks. and so sad I
cann't find any useful information, anyone any comments and suggestions, I really appreciate that.

logstash 2.1.1 , elasticsearch 2.1.1 , and postgres 9.3 is the versions I used.

I think the problem is elasticsearch can't parse the geojson string data I exported from postgres. the elasticsearch want to parse an object. :frowning:

The exception information is(when I start the logstash) :

Firstly: logstash is used to export data from postgres. and my cocnfiguration file is :

input {

jdbc {

    # Postgres jdbc connection string to our database, mydb

    jdbc_connection_string => "jdbc:postgresql://...:5432/handle_service"

    # The user we wish to execute our statement as

    jdbc_user => "user"

    jdbc_password => "*****"

    # The path to our downloaded jdbc driver

    jdbc_driver_library => "/home/admin/dataservice-search/elasticsearch-2.1.1/lib/postgresql-9.3-1102-jdbc41.jar"

    # The name of the driver class for Postgresql

    jdbc_driver_class => "org.postgresql.Driver"

    parameters => { "favorite_artist" => "Beethoven" }

    schedule => " * * * "

    # our query

    statement => " select st_asgeojson(st_makevalid(geom)) As geometry_offset_geo , from t_delete_road limit 20"

    type => "datasearch"

}

}

output {

stdout {codec => rubydebug }

    elasticsearch {

    index => "datasearch"

    document_type => "datasearch"

    document_id => "%{id}"

    hosts => [ "...:9200" ]

}

}

second, I created the index in the elasticsearch:

curl -XPOST http://...:9200/datasearch/datasearch/_mapping  -d '{
"datasearch":{
               "properties" : {
                       "geometry_offset_geo":{
                                             "type":"geo_shape",
                                             "tree":"quadtree",
                                            "precision":"1m"
                                                  }
                                        }
                        }

} '

third, I start the logstash. and then something was wrong.

But it's works when I use

curl -XPOST 'http://...:9200/datasearch/datasearch/1' -d '{"geometry":{

"type":"Polygon",

"coordinates":[

[[-85.0018514,37.1311314],

[-85.0016645,37.1315293],

[-85.0016246,37.1317069],

[-85.0016526,37.1318183],

[-85.0017119,37.1319196],

[-85.0019371,37.1321182],

[-85.0019972,37.1322115],

[-85.0000002,37.1317672]]]

}

}'

I really wonder what happen among those steps. so I come here and ask for some help, please.

It's so important to me to solve this problem.
thanks in advance.


(Mark Walkom) #2

Is that the actual data or have you changed the error?


(Jörg Prante) #3

FYI my JDBC importer https://github.com/jprante/elasticsearch-jdbc can parse GeoJSON points and shapes from PostgreSQL.

If there are problems, you can ask for help at https://github.com/jprante/elasticsearch-jdbc/issues


#4

Actually I did. But I got the same error. so I want to know if my conf file has something wrong. I really need some help. The geometry data is sotred in the db like this :

"0101000020E6100000CDCCCCCC2D835E408C25BF58A6054A40"

and I read it in the logstash using the follow sentences:

select st_asgeojson(st_makevalid(geom)) As geometry_offset_geo , from t_delete_road limit 20

and elasticsearch-jdbc-2.1.1.2 is my choice.

thanks in advance, any comments I will appreciate that.


#5

sorry, I corrected that. and thank you!


(system) #6