# Update Loading Wikipedia Guide

**URL:** <https://discuss.elastic.co/t/update-loading-wikipedia-guide/99911>\
**Category:** Elasticsearch\
**Created:** [September 9, 2017, 1:49am UTC](https://discuss.elastic.co/t/update-loading-wikipedia-guide/99911 "2017-09-09T01:49:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Palisand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/palisand/32/21936_2.png) [@Palisand](https://discuss.elastic.co/u/Palisand)\
**Post date:** [September 9, 2017, 1:49am UTC](https://discuss.elastic.co/t/update-loading-wikipedia-guide/99911/1 "2017-09-09T01:49:08Z")

</div>

**UPDATE: I managed to create the index; see my UPDATEs in my response below.**

[Loading Wikipedia's Search Index For Testing](https://www.elastic.co/blog/loading-wikipedia) was posted in early 2016. I have been trying (and failing) to load the Wikipedia search index while using this guide as a reference because I cannot find a more up-to-date alternative. I have tried using elasticsearch 2.4 and 5.5 but I receive error after error when attempting to create the index with the latest CirrusSearch settings and mapping.

@nik9000 It would be awesome if this guide was updated for late 2017.

Has anyone had any luck indexing Wikipedia as of late?  
Does anyone know how I can perform a **More Like This query against Wikipedia's pages** so that I may extract the categories attached to the highest scoring pages without having to download and index Wikipedia locally?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 9, 2017, 6:45am UTC](https://discuss.elastic.co/t/update-loading-wikipedia-guide/99911/2 "2017-09-09T06:45:03Z")

</div>

Can you provide the errors you are seeing?

---

<div class="post-metadata">

**Author:** ![Palisand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/palisand/32/21936_2.png) [@Palisand](https://discuss.elastic.co/u/Palisand)\
**Post date:** [September 9, 2017, 4:32pm UTC](https://discuss.elastic.co/t/update-loading-wikipedia-guide/99911/3 "2017-09-09T16:32:17Z")

</div>

On **OS X 10.12.6** (elasticsearch installed via homebrew)

Traceback snippets from _elasticsearch-py_

**2.4**

```auto
TransportError(400, 'illegal_argument_exception', 'failed to create index')

```

Logs reveal the same issues as the traceback for es 5.5 below

**5.5**

```auto
TransportError(400, 'illegal_argument_exception', 'Custom Analyzer [plain] failed to find filter under name [preserve_original_recorder]')
TransportError(400, 'illegal_argument_exception', 'Custom Analyzer [plain] failed to find filter under name [preserve_original]')

```

**UPDATE 2: Facepalming continues. [Wikimedia's search-extra](https://github.com/wikimedia/search-extra) extension plugin might do the trick. If I actually get this working I'll post an updated guide myself.**

I receive the same type of error for other analyzers (`lowercase_keyword`, `short_text`) using `preserve_original_recorder` and `preserve_original`. ~~After removing the problem filters from the settings, I received the following type of errors for the mapping:~~  
~~`TransportError(400, 'mapper_parsing_exception', 'Unknown Similarity type [arrays] for field [FIELD NAME]')`~~

**UPDATE 1: This is likely due to me not including the `"similarity"` setting. Let's see if I have committed any more mistakes meriting a facepalm.**

I stopped at this point to start this topic.

Configuration:

```auto
{
  "settings": {
    "analysis": .content.page.index.analysis, // from below *settings*
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    .content // from below *mappings*
  }
}

```

**settings**

> <https://en.wikipedia.org/w/api.php?action=cirrus-settings-dump&format=json&formatversion=2>
>
> {"content":{"page":{"index":{"refresh\_interval":"30s","indexing":{"slowlog":{"threshold":{"index":{"warn":"10s","trace":"-1","debug":"2s","info":"5s"}}}},"auto\_expand\_replicas":"0-3","provided\_name":"enwiki\_content\_1511808841","query":{"default\_field":"all"},"creation\_date":"1511808842152","analysis":{"filter":{"asciifolding\_preserve":{"type":"asciifolding","preserve\_original":"true"},"prefix\_ngram\_filter":{"type":"edgeNGram","max\_gram":"255"},"suggest\_shingle":{"max\_shingle\_size":"3","min\_shingl...

**mappings**

> <https://en.wikipedia.org/w/api.php?action=cirrus-mapping-dump&format=json&formatversion=2>
>
> {"content":{"namespace":{"dynamic":"false","\_all":{"enabled":false},"properties":{"name":{"type":"text","norms":false,"index\_options":"docs","analyzer":"near\_match\_asciifolding"},"wiki":{"type":"text","norms":false,"index\_options":"docs","analyzer":"keyword"}}},"page":{"dynamic":"false","\_all":{"enabled":false},"properties":{"all":{"type":"text","similarity":"BM25","fields":{"plain":{"type":"text","similarity":"BM25","analyzer":"plain","search\_analyzer":"plain\_search","position\_increment\_gap":10}...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2017, 4:19pm UTC](https://discuss.elastic.co/t/update-loading-wikipedia-guide/99911/5 "2017-10-09T16:19:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.