Logstash plugin unrecoverable error when using a mapping

Iain_Woolf · November 4, 2015, 10:18pm

I'm trying to pull in UK Crime data to Elasticsearch so that I can experiment with the map visualizations in Kibana 4. When I create a mapping to show that the location field is a geo_point, I then get errors when trying to import data.

Here are the steps to reproduce:

Create an empty index to receive crime data:

curl -XPUT 'http://localhost:9200/crime' -d '
index :
  number_of_shards : 1
  number_of_replicas : 1
'

Create a mapping to show that the json data already has a location field which gives latitude and longitude:

curl -XPUT 'http://localhost:9200/crime/_mapping/crimestats' -d '
{
  "crimestats" : {
     "properties" : {
        "location" : {"type" : "geo_point"}
     }
  }
}
'

Using data from https://data.gov.uk/dataset/england-national-crime-mapping, paste a single line of JSON using the stdin input plugin. Get an error from Elasticsearch.

/opt/logstash/bin/logstash -e '
input {
  stdin {

    type => "crimestats"
    codec => "json_lines"

  }
}
output {
  elasticsearch {
    index => "crime"
    host     => "127.0.0.1"
    protocol => "http"
    manage_template => "false"
  }
  stdout {
  codec => "rubydebug"
  }
}
'
[2015-11-04 16:39:30.927]  WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
Java 7 support is only best effort, it may not work. It will be removed in next release (1.0).
Logstash startup completed

Paste the following:

{"category":"anti-social-behaviour","location_type":"Force","location":{"latitude":"52.634845","street":

Here's the error returned from Elasticsearch:

{"id":332249,"name":"On or near Queen Street"},"longitude":"-1.124838"},"context":"","outcome_status":null,"persistent_id":"","id":3350469,"location_subtype":"","month":"2011-08"}
failed action with response of 400, dropping action: ["index", {:_id=>nil, :_index=>"crime", :_type=>"crimestats", :_routing=>nil}, #<LogStash::Event:0x7ac5cba7 @metadata={"retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x2218c2e5 @store={"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "street"=>{"id"=>332249, "name"=>"On or near Queen Street"}, "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-04T22:21:56.913Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, @lut={"type"=>[{"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "street"=>{"id"=>332249, "name"=>"On or near Queen Street"}, "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-04T22:21:56.913Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, "type"], "host"=>[{"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "street"=>{"id"=>332249, "name"=>"On or near Queen Street"}, "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-04T22:21:56.913Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, "host"]}>, @data={"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "street"=>{"id"=>332249, "name"=>"On or near Queen Street"}, "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-04T22:21:56.913Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, @metadata_accessors=#<LogStash::Util::Accessors:0x7ccb7779 @store={"retry_count"=>0}, @lut={}>, @cancelled=false>] {:level=>:warn}

And rubydebug output

{
            "category" => "anti-social-behaviour",
       "location_type" => "Force",
            "location" => {
         "latitude" => "52.634845",
           "street" => {
              "id" => 332249,
            "name" => "On or near Queen Street"
        },
        "longitude" => "-1.124838"
    },
             "context" => "",
      "outcome_status" => nil,
       "persistent_id" => "",
                  "id" => 3350469,
    "location_subtype" => "",
               "month" => "2011-08",
            "@version" => "1",
          "@timestamp" => "2015-11-04T22:21:56.913Z",
                "type" => "crimestats",
                "host" => "uottvr17ctrl01"
}

Note: no error if the index was created without a mapping.

Versions I'm using:

rpm -q elasticsearch kibana logstash
 elasticsearch-1.7.1-1.noarch
 kibana-4.1.1-1.x86_64
 logstash-1.5.4-1.noarch

magnusbaeck · November 5, 2015, 7:50am

You're telling ES that the location field is a geo_point when the data you're passing looks like this:

"location": {
  "latitude": "52.634845",
  "street": {
    "id": 332249,
    "name": "On or near Queen Street"
  },
  "longitude": "-1.124838"
}

I don't recall exactly what a field must look like to fit in a geo_point, but you definitely have to remove the street subfield from there.

Iain_Woolf · November 5, 2015, 1:40pm

I thought it would be OK with an extra field in there, but I've taken it out on your recommendation. The result is the same.

(BTW, the input type is documented here https://www.elastic.co/guide/en/elasticsearch/reference/1.7/mapping-geo-point-type.html)

Input:

{"category":"anti-social-behaviour","location_type":"Force","location":{"latitude":"52.634845","longitude":"-1.124838"},"context":"","outcome_status":null,"persistent_id":"","id":3350469,"location_subtype":"","month":"2011-08"}

Elasticsearch plugin output:

failed action with response of 400, dropping action: ["index", {:_id=>nil, :_index=>"crime", :_type=>"crimestats", :_routing=>nil}, #<LogStash::Event:0x79f7411 @metadata={"retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x649ece9b @store={"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-05T13:36:59.098Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, @lut={"type"=>[{"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-05T13:36:59.098Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, "type"], "host"=>[{"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-05T13:36:59.098Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, "host"]}>, @data={"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-05T13:36:59.098Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, @metadata_accessors=#<LogStash::Util::Accessors:0x2c5b1cf5 @store={"retry_count"=>0}, @lut={}>, @cancelled=false>] {:level=>:warn}

Stdout rubydebug output:

{
            "category" => "anti-social-behaviour",
       "location_type" => "Force",
            "location" => {
         "latitude" => "52.634845",
        "longitude" => "-1.124838"
    },
             "context" => "",
      "outcome_status" => nil,
       "persistent_id" => "",
                  "id" => 3350469,
    "location_subtype" => "",
               "month" => "2011-08",
            "@version" => "1",
          "@timestamp" => "2015-11-05T13:36:59.098Z",
                "type" => "crimestats",
                "host" => "uottvr17ctrl01"
}

magnusbaeck · November 5, 2015, 2:24pm

See the Input Structure section in the documentation page you linked to. The location value you're passing doesn't match any of the supported formats, but if you convert the string values to floats (using the mutate filter's convert option) things should be fine.

Iain_Woolf · November 5, 2015, 2:39pm

Doh! Thanks for pointing that out. My brain was equating "lat" with "latitude" and I'd overlooked it and the string/float.

This works:

Input:

{"category":"anti-social-behaviour","location_type":"Force","location":{"lat":52.634845,"lon":-1.124838},"context":"","outcome_status":null,"tent_id":"","id":3350469,"location_subtype":"","month":"2011-08"}

Stdout/rubydebug:

{
            "category" => "anti-social-behaviour",
       "location_type" => "Force",
            "location" => {
        "lat" => 52.634845,
        "lon" => -1.124838
    },
             "context" => "",
      "outcome_status" => nil,
       "persistent_id" => "",
                  "id" => 3350469,
    "location_subtype" => "",
               "month" => "2011-08",
            "@version" => "1",
          "@timestamp" => "2015-11-05T14:35:31.505Z",
                "type" => "crimestats",
                "host" => "uottvr17ctrl01"
}

More importantly I can also create a tilemap visualization from this data!

Topic		Replies	Views
Help with Geo_point (longitude and latitude) Elasticsearch	7	6509	March 23, 2018
Logstash parse_exception geo_point expected Logstash	14	4203	January 10, 2017
Location Mapping Logstash	40	2381	September 2, 2019
Illegal latitude value exception in elasticsearch Elasticsearch	27	8611	September 11, 2017
JSON latitude/longitude to geo_point Elasticsearch	5	12580	July 5, 2017

Logstash plugin unrecoverable error when using a mapping

Related topics