Geo_point not indexed in Array format

Elastic Stack Logstash
1

Hi all,

I'm using logstash to index mongodb documents into ES. every thing goes well but the location object is not indexed. I defined a template mapping as geo_point as follow :

      "arrival_in": {
        "ignore_malformed": true,
        "type": "geo_point",
        "ignore_z_value": true
      },

Here is the array object I want to index :

{...
    "arrival_in" : [ 
        10.1039266666667, 
        33.889845
    ],
...
}

nothing indexed in ES, but when I try to insert manually with dev console, every thing OK.
Could you help please.
Jalil

2

Any help please ?

3

Do you see any ERROR OR WARN in your Logstash log when you try to index the document? Is your time filter in Discover tab correctly set to show the time the document was set to indexed at?

4

Time filter in discover is ok and I see my documents but without arrival_in.

there is 4 WARN lines in log :

[2020-04-27T11:16:22,808][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
...
[2020-04-27T11:16:43,563][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-04-27T11:16:43,680][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-04-27T11:16:43,688][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-27T11:16:43,762][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2020-04-27T11:16:43,776][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2020-04-27T11:16:43,805][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2020-04-27T11:16:43,968][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.

No ERROR line.
thank you

5

Ok, so doc is getting indexed but you don't see the field in the Discover tab?. Have you tried refreshing your index pattern in Kibana?

Screen Shot 2020-04-27 at 6.10.34 AM
Screen Shot 2020-04-27 at 6.10.34 AM2464×412 50.7 KB

6

Yes I did it every time I reindex or insert news documents.

7

Ok, please paste the following:

  1. Your logstash pipeline including the output Elasticsearch plugin that you are using for indexing
  2. What is the index to which your mapping template is applied to?
8

here is my logstash config :

# config pour intégrer des données mongodb dans ES

input{
  mongodb {
    uri => 'mongodb://localhost:27017/trips'
    placeholder_db_dir => 'C:\datadb\logstash-mongodb'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'trip'
    batch_size => 5000
  }
}

filter{
	mutate {
		copy => { "_id" => "[@metadata][_id]"}
		remove_field => ["_id"]
	}
	
}

output{
elasticsearch {
	manage_template => "false"
    #document_type => "stop"
    #template_overwrite => "true"
    template_name=>"trip-template"
	hosts => ["localhost:9200"]
	index => "trip-3-%{+YYYY.MM.dd}"
	}
	stdout { codec => rubydebug }
}

and here is some examples of data :

[
	{
		"_id": {
			"$oid": "5e89cca6bf8ce1de0e13fa19"
		},
		"departure_at": {
			"$date": "2019-03-18T14:27:46Z"
		},
		"departure_place": null,
		"departure_city": "Tina",
		"distance": 119,
		"duration": 13231,
		"fuel_consumption": null,
		"arrival_at": {
			"$date": "2019-03-18T18:08:17Z"
		},
		"arrival_in":[10.103926666666667,33.889845],
		"arrival_place": null,
		"arrival_city": "Gabes Medina"
	},
....
]

My template :

{
  "_doc": {
    "_meta": {},
    "_source": {},
    "properties": {
      "duration": {
        "type": "long"
      },
      "@timestamp": {
        "type": "date"
      },
      "distance": {
        "type": "long"
      },
      "arrival_city": {
        "type": "text",
        "fields": {
          "keyword": {
            "ignore_above": 256,
            "type": "keyword"
          }
        }
      },
      "departure_at": {
        "type": "date"
      },
      "departure_city": {
        "type": "text",
        "fields": {
          "keyword": {
            "ignore_above": 256,
            "type": "keyword"
          }
        }
      },
      "arrival_in": {
        "ignore_malformed": false,
        "type": "geo_point",
        "ignore_z_value": false
      },
      "arrival_at": {
        "type": "date"
      }
    }
  }
}

thank you in advance for your help :slight_smile:

9

I ran your log over my Logstash docker and here are a few problems that I see:

  1. Your response from your mongodb query is an array of JSON docs (from what I can see in your response). Your filter plugin does not address this. Assuming you want each of those JSON docs indexed as a separate doc in Elasticsearch, you need to loop over that array.

  2. After you do 1, you need to use json { source => "message" }, this will make sure each JSON entry gets parsed as a JSON doc. This should let your arrival_in field get indexed as a geo_point.

  3. Your arrival_at and departure_at are both nested json fields while your mappings expects them to be of type date so there is clear conflict there. You need to pull those fields out and bring them to the root. Here is the error I got when I tried to index it.

logstash | [2020-04-27T12:14:32,277][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"trip-3-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x512dda17>], :response=>{"index"=>{"_index"=>"trip-3-2020.04.27", "_type"=>"_doc", "_id"=>"JLyPu3EBaCCXfMYpcRQ4", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [arrival_at] of type [date] in document with id 'JLyPu3EBaCCXfMYpcRQ4'. Preview of field's value: '{$date=2019-03-18T18:08:17Z}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:45"}}}}}

10

Thank you again, I can do that and loop over my array, but how can you explain why all other field are well mapped (text, string and number fields) but not the GPS coordinates one.

I think that eLogstash and ES consider them as separate doc because I get the right number of doc in ES after logstash finishing his processing (647 doc indexed), the right doc count.

I tested it manuelly, I mean I inserted one of those doc using dev console of kibana and it works well and arrival_in is recognized as geo_point.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.