I'm newbie and i need index large json data to elasticsearch so i use logstash to index my json files in some directory here is the logstash config:
<
input {
file {
path => "/home/abahadir/Desktop/Proje/searchflask/elasticsearch/*.json"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline { pattern => "^}" negate => true what => next auto_flush_interval => 1 }
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "restaurants"
document_type => "docket"
document_id => "%{[path]}"
}
stdout{ codec => rubydebug }
}
/>
here is the simple macth_all query to check my data indexed
resp2 = es.search(index="restaurants", doc_type="docket",body={ "query": { "match_all": { } }})
and when i printed it this is the result i get
<
{u'hits': {u'hits': [{u'_score': 1.0, u'_type': u'docket', u'_id': u'/home/abahadir/Desktop/Proje/searchflask/elasticsearch/DCMs.json', u'_source': {u'tags': [u'multiline'], u'@timestamp': u'2020-03-08T08:12:05.185Z', u'host': u'abahadir-Lenovo-Y520-15IKBN', u'path': u'/home/abahadir/Desktop/Proje/searchflask/elasticsearch/DCMs.json', u'message': u' "DICOM_QR_SCP"\n ]\n },\n "0018A002": {\n "vr": "DT",\n "Value": [\n "20190509151148"\n ]\n },\n "0040A170": {\n "vr": "SQ",\n
this is the part that i dont understand, i know that when i index my file to elasticsearch i should reach the indexed data form hits.hits._souce am i wrong about that end even if u'message' part contains some of the data it is partial and start from middle of it and contains a very little amount of the data