Hi all,
I am testing indexing with elasticsearch 1.3.2 on my Ubuntu-PC with 8GB of
RAM and an SSD-Disk.
export ES_HEAP_SIZE=5g
in elasticsearch.yml I set
bootstrap.mlockall: true
all other attributes are commented out
First i am creating an index with
curl -XPUT 'http://localhost:9200/mih'
then i start a Java-Program that in a loop indexes quite big
JSON-Documents, using Apache HttpClient
CURL-Equivalent to the requests Java-Program sends:
curl -XPOST 'http://localhost:9200/mih/mih' -d '{
"id":"AaqOeBfCwZGBXTBP",
"parentId":"VOWyQAzBqPFpveBV",
"contentModelId":"data",
"state":"PENDING",
"version":1,
"children":[
],
"label":"benchtool-FpqSInvLvZlqkYEs",
"utcCreated":"2014-12-17T06:24:22.728Z",
"utcLastModified":"2014-12-17T06:24:22.728Z",
"tags":null,
"metadata":{
"DCgOxyg":{
"name":"DCgOxyg",
"size":316,
"mimetype":"text/xml",
"filename":"dc.xml",
"checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5",
"checksumType":"MD5",
"path":"6,0f0428220f57",
"source":{
"type":"url-source",
"uri":"/entity/AaqOeBfCwZGBXTBP/metadata/DCgOxyg/content",
"internal":true
},
"type":"DC",
"indexInline":true,
"jsonData":{
"metadata":{
"title":"Test Object",
"creator":"fasseg",
"subject":"Testing Groven",
"description":"Test Object to implement integration Tests"
}
},
"utcCreated":"2014-12-17T06:24:22.747Z",
"utcLastModified":"2014-12-17T06:24:22.747Z"
},
"DCsgYyF":{
"name":"DCsgYyF",
"size":316,
"mimetype":"text/xml",
"filename":"dc.xml",
"checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5",
"checksumType":"MD5",
"path":"1,0f054af03d1f",
"source":{
"type":"url-source",
"uri":"/entity/AaqOeBfCwZGBXTBP/metadata/DCsgYyF/content",
"internal":true
},
"type":"DC",
"indexInline":true,
"jsonData":{
"metadata":{
"title":"Test Object",
"creator":"fasseg",
"subject":"Testing Groven",
"description":"Test Object to implement integration Tests"
}
},
"utcCreated":"2014-12-17T06:24:22.754Z",
"utcLastModified":"2014-12-17T06:24:22.754Z"
},
"DCVFQgr":{
"name":"DCVFQgr",
"size":316,
"mimetype":"text/xml",
"filename":"dc.xml",
"checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5",
"checksumType":"MD5",
"path":"2,0f0651292ba2",
"source":{
"type":"url-source",
"uri":"/entity/AaqOeBfCwZGBXTBP/metadata/DCVFQgr/content",
"internal":true
},
"type":"DC",
"indexInline":true,
"jsonData":{
"metadata":{
"title":"Test Object",
"creator":"fasseg",
"subject":"Testing Groven",
"description":"Test Object to implement integration Tests"
}
},
"utcCreated":"2014-12-17T06:24:22.792Z",
"utcLastModified":"2014-12-17T06:24:22.792Z"
}
},
"binaries":{
"binary-BFpeqrSaGFekwGCX":{
"name":"binary-BFpeqrSaGFekwGCX",
"size":81920,
"mimetype":"application/octet-stream",
"metadata":{
"DCYCtYC":{
"name":"DCYCtYC",
"size":316,
"mimetype":"text/xml",
"filename":"dc.xml",
"checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5",
"checksumType":"MD5",
"path":"3,0f0b7df76af5",
"source":{
"type":"url-source",
"uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-BFpeqrSaGFekwGCX/metadata/DCYCtYC/content",
"internal":true
},
"type":"DC",
"indexInline":true,
"jsonData":{
"metadata":{
"title":"Test Object",
"creator":"fasseg",
"subject":"Testing Groven",
"description":"Test Object to implement integration
Tests"
}
},
"utcCreated":"2014-12-17T06:24:22.809Z",
"utcLastModified":"2014-12-17T06:24:22.809Z"
},
"DCHCyyf":{
"name":"DCHCyyf",
"size":316,
"mimetype":"text/xml",
"filename":"dc.xml",
"checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5",
"checksumType":"MD5",
"path":"1,0f0d1709f367",
"source":{
"type":"url-source",
"uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-BFpeqrSaGFekwGCX/metadata/DCHCyyf/content",
"internal":true
},
"type":"DC",
"indexInline":true,
"jsonData":{
"metadata":{
"title":"Test Object",
"creator":"fasseg",
"subject":"Testing Groven",
"description":"Test Object to implement integration
Tests"
}
},
"utcCreated":"2014-12-17T06:24:22.820Z",
"utcLastModified":"2014-12-17T06:24:22.820Z"
}
},
"filename":"binary-BFpeqrSaGFekwGCX.bin",
"checksum":"677ac6824c100d474255e8717523ea19",
"checksumType":"MD5",
"path":"5,0f09c92d98d1",
"source":{
"type":"url-source",
"uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-BFpeqrSaGFekwGCX/content",
"internal":true
},
"utcCreated":"2014-12-17T06:24:22.801Z",
"utcLastModified":"2014-12-17T06:24:22.801Z"
},
"binary-kOyemkEPEQkCtgTd":{
"name":"binary-kOyemkEPEQkCtgTd",
"size":81920,
"mimetype":"application/octet-stream",
"metadata":{
"DCMDzAk":{
"name":"DCMDzAk",
"size":316,
"mimetype":"text/xml",
"filename":"dc.xml",
"checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5",
"checksumType":"MD5",
"path":"2,0f138a21e637",
"source":{
"type":"url-source",
"uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-kOyemkEPEQkCtgTd/metadata/DCMDzAk/content",
"internal":true
},
"type":"DC",
"indexInline":true,
"jsonData":{
"metadata":{
"title":"Test Object",
"creator":"fasseg",
"subject":"Testing Groven",
"description":"Test Object to implement integration
Tests"
}
},
"utcCreated":"2014-12-17T06:24:22.900Z",
"utcLastModified":"2014-12-17T06:24:22.900Z"
},
"DCZEQsj":{
"name":"DCZEQsj",
"size":316,
"mimetype":"text/xml",
"filename":"dc.xml",
"checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5",
"checksumType":"MD5",
"path":"2,0f16ce4bfc03",
"source":{
"type":"url-source",
"uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-kOyemkEPEQkCtgTd/metadata/DCZEQsj/content",
"internal":true
},
"type":"DC",
"indexInline":true,
"jsonData":{
"metadata":{
"title":"Test Object",
"creator":"fasseg",
"subject":"Testing Groven",
"description":"Test Object to implement integration
Tests"
}
},
"utcCreated":"2014-12-17T06:24:22.911Z",
"utcLastModified":"2014-12-17T06:24:22.911Z"
}
},
"filename":"binary-kOyemkEPEQkCtgTd.bin",
"checksum":"8923305820da6534f63b8bf4d40ae21c",
"checksumType":"MD5",
"path":"5,0f0fefcdd8f2",
"source":{
"type":"url-source",
"uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-kOyemkEPEQkCtgTd/content",
"internal":true
},
"utcCreated":"2014-12-17T06:24:22.826Z",
"utcLastModified":"2014-12-17T06:24:22.826Z"
}
},
"alternativeIdentifiers":[],
"relations":null
}'
The more Documents i index the slower indexing gets. So for the first 100
Documents i need about 20 ms to index 1 Document.
When i have about 1000 Documents in the Index, indexing slows down and
takes me about 500 ms to index 1 Document.
Restart of Elasticsearch or of my Computer doesnt help, indexing still
needs 500 ms per Document if I already have 1000 indexed Documents in the
index.
Index-Size at that point is about 60MB, containing about 30 Segments.
Is there any Hint how i can accelerate indexing? 1000 Documents on one
Machine shouldnt be too much.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a18976f5-898a-4d18-8e8f-ddce84841437%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.