Logstash plugin unrecoverable error when using a mapping

Elastic Stack Logstash
1

I'm trying to pull in UK Crime data to Elasticsearch so that I can experiment with the map visualizations in Kibana 4. When I create a mapping to show that the location field is a geo_point, I then get errors when trying to import data.

Here are the steps to reproduce:

Create an empty index to receive crime data:

curl -XPUT 'http://localhost:9200/crime' -d '
index :
  number_of_shards : 1
  number_of_replicas : 1
'

Create a mapping to show that the json data already has a location field which gives latitude and longitude:

curl -XPUT 'http://localhost:9200/crime/_mapping/crimestats' -d '
{
  "crimestats" : {
     "properties" : {
        "location" : {"type" : "geo_point"}
     }
  }
}
'

Using data from https://data.gov.uk/dataset/england-national-crime-mapping, paste a single line of JSON using the stdin input plugin. Get an error from Elasticsearch.

/opt/logstash/bin/logstash -e '
input {
  stdin {

    type => "crimestats"
    codec => "json_lines"

  }
}
output {
  elasticsearch {
    index => "crime"
    host     => "127.0.0.1"
    protocol => "http"
    manage_template => "false"
  }
  stdout {
  codec => "rubydebug"
  }
}
'
[2015-11-04 16:39:30.927]  WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
Java 7 support is only best effort, it may not work. It will be removed in next release (1.0).
Logstash startup completed

Paste the following:

{"category":"anti-social-behaviour","location_type":"Force","location":{"latitude":"52.634845","street":

Here's the error returned from Elasticsearch:

{"id":332249,"name":"On or near Queen Street"},"longitude":"-1.124838"},"context":"","outcome_status":null,"persistent_id":"","id":3350469,"location_subtype":"","month":"2011-08"}
failed action with response of 400, dropping action: ["index", {:_id=>nil, :_index=>"crime", :_type=>"crimestats", :_routing=>nil}, #<LogStash::Event:0x7ac5cba7 @metadata={"retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x2218c2e5 @store={"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "street"=>{"id"=>332249, "name"=>"On or near Queen Street"}, "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-04T22:21:56.913Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, @lut={"type"=>[{"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "street"=>{"id"=>332249, "name"=>"On or near Queen Street"}, "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-04T22:21:56.913Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, "type"], "host"=>[{"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "street"=>{"id"=>332249, "name"=>"On or near Queen Street"}, "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-04T22:21:56.913Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, "host"]}>, @data={"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "street"=>{"id"=>332249, "name"=>"On or near Queen Street"}, "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-04T22:21:56.913Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, @metadata_accessors=#<LogStash::Util::Accessors:0x7ccb7779 @store={"retry_count"=>0}, @lut={}>, @cancelled=false>] {:level=>:warn}

And rubydebug output

{
            "category" => "anti-social-behaviour",
       "location_type" => "Force",
            "location" => {
         "latitude" => "52.634845",
           "street" => {
              "id" => 332249,
            "name" => "On or near Queen Street"
        },
        "longitude" => "-1.124838"
    },
             "context" => "",
      "outcome_status" => nil,
       "persistent_id" => "",
                  "id" => 3350469,
    "location_subtype" => "",
               "month" => "2011-08",
            "@version" => "1",
          "@timestamp" => "2015-11-04T22:21:56.913Z",
                "type" => "crimestats",
                "host" => "uottvr17ctrl01"
}

Note: no error if the index was created without a mapping.

Versions I'm using:

rpm -q elasticsearch kibana logstash
 elasticsearch-1.7.1-1.noarch
 kibana-4.1.1-1.x86_64
 logstash-1.5.4-1.noarch
2

You're telling ES that the location field is a geo_point when the data you're passing looks like this:

"location": {
  "latitude": "52.634845",
  "street": {
    "id": 332249,
    "name": "On or near Queen Street"
  },
  "longitude": "-1.124838"
}

I don't recall exactly what a field must look like to fit in a geo_point, but you definitely have to remove the street subfield from there.

3

I thought it would be OK with an extra field in there, but I've taken it out on your recommendation. The result is the same.

(BTW, the input type is documented here https://www.elastic.co/guide/en/elasticsearch/reference/1.7/mapping-geo-point-type.html)

Input:

{"category":"anti-social-behaviour","location_type":"Force","location":{"latitude":"52.634845","longitude":"-1.124838"},"context":"","outcome_status":null,"persistent_id":"","id":3350469,"location_subtype":"","month":"2011-08"}

Elasticsearch plugin output:

failed action with response of 400, dropping action: ["index", {:_id=>nil, :_index=>"crime", :_type=>"crimestats", :_routing=>nil}, #<LogStash::Event:0x79f7411 @metadata={"retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x649ece9b @store={"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-05T13:36:59.098Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, @lut={"type"=>[{"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-05T13:36:59.098Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, "type"], "host"=>[{"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-05T13:36:59.098Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, "host"]}>, @data={"category"=>"anti-social-behaviour", "location_type"=>"Force", "location"=>{"latitude"=>"52.634845", "longitude"=>"-1.124838"}, "context"=>"", "outcome_status"=>nil, "persistent_id"=>"", "id"=>3350469, "location_subtype"=>"", "month"=>"2011-08", "@version"=>"1", "@timestamp"=>"2015-11-05T13:36:59.098Z", "type"=>"crimestats", "host"=>"uottvr17ctrl01"}, @metadata_accessors=#<LogStash::Util::Accessors:0x2c5b1cf5 @store={"retry_count"=>0}, @lut={}>, @cancelled=false>] {:level=>:warn}

Stdout rubydebug output:

{
            "category" => "anti-social-behaviour",
       "location_type" => "Force",
            "location" => {
         "latitude" => "52.634845",
        "longitude" => "-1.124838"
    },
             "context" => "",
      "outcome_status" => nil,
       "persistent_id" => "",
                  "id" => 3350469,
    "location_subtype" => "",
               "month" => "2011-08",
            "@version" => "1",
          "@timestamp" => "2015-11-05T13:36:59.098Z",
                "type" => "crimestats",
                "host" => "uottvr17ctrl01"
}
4

See the Input Structure section in the documentation page you linked to. The location value you're passing doesn't match any of the supported formats, but if you convert the string values to floats (using the mutate filter's convert option) things should be fine.

1 Like
5

Doh! Thanks for pointing that out. My brain was equating "lat" with "latitude" and I'd overlooked it and the string/float.

This works:

Input:

{"category":"anti-social-behaviour","location_type":"Force","location":{"lat":52.634845,"lon":-1.124838},"context":"","outcome_status":null,"tent_id":"","id":3350469,"location_subtype":"","month":"2011-08"}

Stdout/rubydebug:

{
            "category" => "anti-social-behaviour",
       "location_type" => "Force",
            "location" => {
        "lat" => 52.634845,
        "lon" => -1.124838
    },
             "context" => "",
      "outcome_status" => nil,
       "persistent_id" => "",
                  "id" => 3350469,
    "location_subtype" => "",
               "month" => "2011-08",
            "@version" => "1",
          "@timestamp" => "2015-11-05T14:35:31.505Z",
                "type" => "crimestats",
                "host" => "uottvr17ctrl01"
}

More importantly I can also create a tilemap visualization from this data!