Help Using Python to Load Data into ES

Elastic Stack Elasticsearch
1

I am having some issues getting the following python script to load data into ElasticSearch, can someone please guide me. Here is my python script. My goal is to use python to load this into ElastiSearch and also view my index in Kibana.

Excuse the long json file.

import json
from datetime import datetime
from elasticsearch import Elasticsearch
# use ES default connection
es = Elasticsearch()
directory = r'path to file'
i = 1
for filename in os.listdir(directory):
    if filename.endswith(".json"):
        print(filename)
        f = open(filename)
        docket_content = f.read()
        es.index(index='STates_Index', ignore=400, doc_type='docket', 
        id=i, body=json.loads(docket_content))
        i = i + 1

Here is the json file that I am working with.

    {
"type" : "FeatureCollection",
"name" : "Outliner",
"features" : [
	{
		"type" : "Feature",
		"geometry" : {
			"type" : "Polygon",
			"coordinates" : [
				[
					[ -77.9741261597, 35.2186618283 ],
					[ -78.3147315937, 35.7579537654 ],
					[ -79.2513965371, 35.6444186208 ],
					[ -78.541801883, 34.7645212497 ],
					[ -77.9741261597, 35.2186618283 ]
				]
			]
		},
		"properties" : {
			"Name" : "North Carolina",
			"abbrev" : "NC"
		}
	},
	{
		"type" : "Feature",
		"geometry" : {
			"type" : "Polygon",
			"coordinates" : [
				[
					[ -79.4784668264, 37.1771430737 ],
					[ -79.4784668264, 36.8649214259 ],
					[ -78.541801883, 36.8649214259 ],
					[ -78.541801883, 37.1771430737 ],
					[ -79.4784668264, 37.1771430737 ]
				]
			]
		},
		"properties" : {
			"Name" : "Virginia",
			"abbrev" : "VA"
		}
	},
	{
		"type" : "Feature",
		"geometry" : {
			"type" : "Polygon",
			"coordinates" : [
				[
					[ -81.2098777824, 34.5942185327 ],
					[ -81.718192491, 34.6305267262 ],
					[ -81.7465762772, 34.2331537199 ],
					[ -81.2382615686, 34.1968455264 ],
					[ -81.2098777824, 34.5942185327 ]
				]
			]
		},
		"properties" : {
			"Name" : "South Carolina",
			"abbrev" : "SC"
		}
	}
]
}

Thanks

2

Any ideas please?

3

Hi,

What is the problem you are getting?

To start, your index name should be lower case index-name-type-name-and-field-name-rules

Secondly, to make the data visible in Kibana you need to have a timestamp in the document.

Paul.

4

I have modified to lower case and also added a timestamp as requested. GeoJSON included below. Also made some adjustments to the code, I do see it in the Kibana Indices list, and not on the map. Can someone please help point me in the right direction? I am not getting any errors in my code listed below: What am I missing?

#load package modules
import sys, json
import os
import time
from datetime import datetime
from elasticsearch import Elasticsearch, helpers
from pprint import pprint

root_path = "C:/Users/Base/Desktop/states/"
raw_data_path = root_path + "outlineddate/"
json_filename = "outlinedate.json"
directory = raw_data_path + json_filename

es = Elasticsearch(['localhost'],port=9200)

with open(directory, 'r', encoding='utf-8') as f:
    data = json.loads(f.read())
pprint(data)

es.index(index='statesdates_index', doc_type='doc', body=data)

Here is the Updated GeoJSON

{
"type" : "FeatureCollection",
"name" : "Outline_date",
"features" : [
	{
		"type" : "Feature",
		"geometry" : {
			"type" : "Polygon",
			"coordinates" : [
				[
					[ -77.9741261597, 35.2186618283 ],
					[ -78.3147315937, 35.7579537654 ],
					[ -79.2513965371, 35.6444186208 ],
					[ -78.541801883, 34.7645212497 ],
					[ -77.9741261597, 35.2186618283 ]
				]
			]
		},
		"properties" : {
			"timestamp" : "20190909071754.9718596",
			"Name" : "North Carolina",
			"abbrev" : "NC"
		}
	},
	{
		"type" : "Feature",
		"geometry" : {
			"type" : "Polygon",
			"coordinates" : [
				[
					[ -79.4784668264, 37.1771430737 ],
					[ -79.4784668264, 36.8649214259 ],
					[ -78.541801883, 36.8649214259 ],
					[ -78.541801883, 37.1771430737 ],
					[ -79.4784668264, 37.1771430737 ]
				]
			]
		},
		"properties" : {
			"timestamp" : "20190909071755.0440474",
			"Name" : "Virginia",
			"abbrev" : "VA"
		}
	},
	{
		"type" : "Feature",
		"geometry" : {
			"type" : "Polygon",
			"coordinates" : [
				[
					[ -81.2098777824, 34.5942185327 ],
					[ -81.718192491, 34.6305267262 ],
					[ -81.7465762772, 34.2331537199 ],
					[ -81.2382615686, 34.1968455264 ],
					[ -81.2098777824, 34.5942185327 ]
				]
			]
		},
		"properties" : {
			"timestamp" : "20190909071755.0442809",
			"Name" : "South Carolina",
			"abbrev" : "SC"
		}
	}
]

}

5

I would drop the doc_type parameter, it can't be used in ES7 and there's no point starting with one. In ES6 it has to be _doc I believe. But you can safely just leave it out.

Now that you can see the index, you may need to make sure the locations are being stored as GeoPoints https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

6

So two things,

(1) They are actually boundaries of 3 states.
(2) I went to your link and could not understand what I am looking at, I understand it to be mappings, correct me if i am wrong. How do I apply this to any of the states I am working on, an example will be very appreciated. Thanks.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.