Geo_point not indexed in Array format

Jalil · April 22, 2020, 6:18pm

Hi all,

I'm using logstash to index mongodb documents into ES. every thing goes well but the location object is not indexed. I defined a template mapping as geo_point as follow :

      "arrival_in": {
        "ignore_malformed": true,
        "type": "geo_point",
        "ignore_z_value": true
      },

Here is the array object I want to index :

{...
    "arrival_in" : [ 
        10.1039266666667, 
        33.889845
    ],
...
}

nothing indexed in ES, but when I try to insert manually with dev console, every thing OK.
Could you help please.
Jalil

Jalil · April 25, 2020, 12:10pm

Any help please ?

Rahul_Kumar4 · April 26, 2020, 10:01pm

Do you see any ERROR OR WARN in your Logstash log when you try to index the document? Is your time filter in Discover tab correctly set to show the time the document was set to indexed at?

Jalil · April 27, 2020, 9:32am

Time filter in discover is ok and I see my documents but without arrival_in.

there is 4 WARN lines in log :

[2020-04-27T11:16:22,808][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
...
[2020-04-27T11:16:43,563][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-04-27T11:16:43,680][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-04-27T11:16:43,688][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-27T11:16:43,762][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2020-04-27T11:16:43,776][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2020-04-27T11:16:43,805][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2020-04-27T11:16:43,968][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.

No ERROR line.
thank you

Rahul_Kumar4 · April 27, 2020, 10:14am

Ok, so doc is getting indexed but you don't see the field in the Discover tab?. Have you tried refreshing your index pattern in Kibana?

Jalil · April 27, 2020, 10:25am

Yes I did it every time I reindex or insert news documents.

Rahul_Kumar4 · April 27, 2020, 10:43am

Ok, please paste the following:

Your logstash pipeline including the output Elasticsearch plugin that you are using for indexing
What is the index to which your mapping template is applied to?

Jalil · April 27, 2020, 10:48am

here is my logstash config :

# config pour intégrer des données mongodb dans ES

input{
  mongodb {
    uri => 'mongodb://localhost:27017/trips'
    placeholder_db_dir => 'C:\datadb\logstash-mongodb'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'trip'
    batch_size => 5000
  }
}

filter{
	mutate {
		copy => { "_id" => "[@metadata][_id]"}
		remove_field => ["_id"]
	}
	
}

output{
elasticsearch {
	manage_template => "false"
    #document_type => "stop"
    #template_overwrite => "true"
    template_name=>"trip-template"
	hosts => ["localhost:9200"]
	index => "trip-3-%{+YYYY.MM.dd}"
	}
	stdout { codec => rubydebug }
}

and here is some examples of data :

[
	{
		"_id": {
			"$oid": "5e89cca6bf8ce1de0e13fa19"
		},
		"departure_at": {
			"$date": "2019-03-18T14:27:46Z"
		},
		"departure_place": null,
		"departure_city": "Tina",
		"distance": 119,
		"duration": 13231,
		"fuel_consumption": null,
		"arrival_at": {
			"$date": "2019-03-18T18:08:17Z"
		},
		"arrival_in":[10.103926666666667,33.889845],
		"arrival_place": null,
		"arrival_city": "Gabes Medina"
	},
....
]

My template :

{
  "_doc": {
    "_meta": {},
    "_source": {},
    "properties": {
      "duration": {
        "type": "long"
      },
      "@timestamp": {
        "type": "date"
      },
      "distance": {
        "type": "long"
      },
      "arrival_city": {
        "type": "text",
        "fields": {
          "keyword": {
            "ignore_above": 256,
            "type": "keyword"
          }
        }
      },
      "departure_at": {
        "type": "date"
      },
      "departure_city": {
        "type": "text",
        "fields": {
          "keyword": {
            "ignore_above": 256,
            "type": "keyword"
          }
        }
      },
      "arrival_in": {
        "ignore_malformed": false,
        "type": "geo_point",
        "ignore_z_value": false
      },
      "arrival_at": {
        "type": "date"
      }
    }
  }
}

thank you in advance for your help

Rahul_Kumar4 · April 27, 2020, 12:19pm

I ran your log over my Logstash docker and here are a few problems that I see:

Your response from your mongodb query is an array of JSON docs (from what I can see in your response). Your filter plugin does not address this. Assuming you want each of those JSON docs indexed as a separate doc in Elasticsearch, you need to loop over that array.
After you do 1, you need to use json { source => "message" }, this will make sure each JSON entry gets parsed as a JSON doc. This should let your arrival_in field get indexed as a geo_point.
Your arrival_at and departure_at are both nested json fields while your mappings expects them to be of type date so there is clear conflict there. You need to pull those fields out and bring them to the root. Here is the error I got when I tried to index it.

logstash | [2020-04-27T12:14:32,277][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"trip-3-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x512dda17>], :response=>{"index"=>{"_index"=>"trip-3-2020.04.27", "_type"=>"_doc", "_id"=>"JLyPu3EBaCCXfMYpcRQ4", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [arrival_at] of type [date] in document with id 'JLyPu3EBaCCXfMYpcRQ4'. Preview of field's value: '{$date=2019-03-18T18:08:17Z}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:45"}}}}}

Jalil · April 27, 2020, 12:33pm

Thank you again, I can do that and loop over my array, but how can you explain why all other field are well mapped (text, string and number fields) but not the GPS coordinates one.

I think that eLogstash and ES consider them as separate doc because I get the right number of doc in ES after logstash finishing his processing (647 doc indexed), the right doc count.

I tested it manuelly, I mean I inserted one of those doc using dev console of kibana and it works well and arrival_in is recognized as geo_point.

system · May 25, 2020, 12:33pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Location array object not indexed with logstash Logstash	2	271	May 23, 2020
Geo_point problems with multiple indexes and Logstash Elasticsearch	16	1306	October 16, 2019
Elasticsearch 7.x and geo_point Type: can not do it via index_template mapping? Elasticsearch	3	802	November 19, 2019
Index with geo_point entry doe not show up in Maps/Heatmap Kibana	3	440	November 26, 2020
Geoip filter in logstash with Index Template not creating geo_point object as expected Kibana	6	1216	November 19, 2021

Geo_point not indexed in Array format

Related Topics