# Index, Filter and Query Strategy **URL:** \ **Category:** Elasticsearch\ **Created:** [July 14, 2015, 4:18pm UTC](https://discuss.elastic.co/t/index-filter-and-query-strategy/25536 "2015-07-14T16:18:34Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![taylorsuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taylorsuk/32/3697_2.png) [@taylorsuk](https://discuss.elastic.co/u/taylorsuk)\ **Post date:** [July 14, 2015, 4:18pm UTC](https://discuss.elastic.co/t/index-filter-and-query-strategy/25536/1 "2015-07-14T16:18:34Z")
I have made the move to a search tool (Elastic) after using many filters / REST queries etc and it has been amazing so far however I am very 'green' on the subject. I have two main problems: 1. Bulk and strategy for the future of using ElasticSearch 2. Running, Filter Only, Query Only and Filter + Query **Background** I am using Elastic on an Exercise Prescription Mobile Application that users search for exercises and add them to Programmes. I currently have 3500 exercises that are structured in JSON like below: ``` { "difficulty": "Easy", "exerciseDescription": "Lie prone on a bed with the unaffected leg off, Bend the knee of the leg that is on the bed, Lower the leg and repeat", "exerciseID": "179", "exerciseName": "Femoral Nerve Mobilisation 1", "images": [ 683, 684 ], "tags": [ "Neural", "Hip \u0026 Pelvis", "Knee", "Balance and Proprioception" ], "words": [ "femoral", "nerve", "mobilisation", "1" ] } ``` **Problem 1** The exercise database gets updated every week so I would need to import a new index weekly. For my bulk import I have been using [elasticsearch-tools](https://github.com/skratchdot/elasticsearch-tools) and providing it data in the format below: ``` { "index": { "_index": "exercises", "_type": "exercise" } } {"difficulty":"Easy","exerciseDescription":"Lie prone on a bed with the unaffected leg off, Bend the knee of the leg that is on the bed, Lower the leg and repeat","exerciseID":"179","exerciseName":"Femoral Nerve Mobilisation 1","images":[683,684],"objectId":"AcHqZjCSV6","tags":["Neural","Hip & Pelvis","Knee"],"updatedAt":"2015-06-13T21:39:00.084Z","words":["femoral","nerve","mobilisation","1"]} { "index": { "_index": "exercises", "_type": "exercise" } } {"difficulty":"Easy","exerciseDescription":"Turn head to the side and raise the hand to the mouth as if to smoke, Lower hand to to the side","exerciseID":"216","exerciseName":"Ulnar Nerve Fingers to Lips","images":[788,789],"objectId":"udJSAa5rtH","tags":["Neural","Elbow, Wrist & Hand","Shoulder","No Equipment","Self Treatment"],"updatedAt":"2015-07-02T20:28:48.909Z","words":["ulnar","nerve","fingers","to","lips"]} ``` Which I managed to create my minifying my Pretty JSON and then finding the end of each exercise using Sublime Text and then pasting in the `{ "index": { "_index": "exercises", "_type": "exercise" } }` above each line. **Question 1** How can I improve my workflow to index my exercises without having to do all this laboursome JSON manipulation before pressing GO on the bulk import. Additionally it would be ideal if I am able to index by the exercise ID however I have not figured out how to do that. **Problem 2** There are three methods of search in my App. 1. Freetext / Query 2. Tags Only 3. Freetext / Query + Tags. The user activates 'Tags' in the UI that pushes the name of the Tag to an Array. In the above case the exercise has the tags `["Neural","Hip \u0026 Pelvis","Knee", Balance and Proprioception"]`. I believe that when my index is mapped then the exercise tags are being analysed and therefore made into tokens, therefore when I send the array of tags `["Balance and Proprioception"]` it will not be found and returns an error. I have been using the below query to solve the problem above - however understand that it is far from an ideal way to use ElasticSearch (however I have to say I am getting fantastic results - but its not **correct** ). ``` var searchQuery = { "query": { "match": { "_all": { "query": term + " " + filterArray, "operator": "and" } } }, "aggs": {}, "size": 20 }; ``` In this case an example of `term` would be `squat with dumbells` and an example of `filterArray` is `["Ball","Leg","Strength"]` **Question 2** How can I change my Query to ensure that if there is a filter active then it will perform the filtering on the `tags` index prior to performing the text search on the results. Many thanks for taking the time to read this, after a week I have had to resort to a forum! Both of my questions have been asked on Stackoverflow [here](http://stackoverflow.com/questions/31371958/programatically-prepare-json-for-elasticsearch-bulk-post) and [here](http://stackoverflow.com/questions/31389726/elastic-search-query-array-of-terms-with-spaces). Kind regards Simon ---
**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\ **Post date:** [July 14, 2015, 4:54pm UTC](https://discuss.elastic.co/t/index-filter-and-query-strategy/25536/2 "2015-07-14T16:54:32Z")
**For Question 1:** You could create a [Logstash](https://www.elastic.co/guide/en/logstash/current/index.html) config that collects the data from your source (the database I presume) and outputs to Elasticsearch. This has the benefit that Logstash deals with the bulk requests for you, you just need to configure the [input](https://www.elastic.co/guide/en/logstash/current/input-plugins.html) to point to your data, and the [Elasticsearch output](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) to send the index requests to Elasticsearch. This other advantage is that you can use Logstash to enrich or massage your data if you need to, using [filters](https://www.elastic.co/guide/en/logstash/current/filter-plugins.html). There are also many language clients where you can programmatically create the bulk requests for your documents instead of manually editing the JSON. Documents for how to use those clients are [here](https://www.elastic.co/guide/index.html) (under the clients section). **For Question 2:** Consider using a [`filtered`query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-filtered-query.html) here. You can add your free-text search as a [`match` query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html) to the `query` section (similar to how you currently do it) and the filterArray can be added to the `filter` section using the [`terms` filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html). What you would end up with would look something like the following: ```javascript var searchQuery = { "query": { "filtered": { "query": { "match": { "_all": { "query": term, "operator": "and" } } }, "filter": { "terms": { "tags": filterArray } } } }, "size": 20 } ``` Hope that helps ---
**Author:** ![taylorsuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taylorsuk/32/3697_2.png) [@taylorsuk](https://discuss.elastic.co/u/taylorsuk)\ **Post date:** [July 14, 2015, 4:59pm UTC](https://discuss.elastic.co/t/index-filter-and-query-strategy/25536/3 "2015-07-14T16:59:33Z")
@colings86 - Fantastic, many thanks. Once quick question on using the `terms` filter to match my exercise `tags` on indexing it seems that they are mapped as Strings (analysed) therefore if I try to match terms using and array like this `["Balance and Proprioception"]` it does not match does to it not matching the tags index exactly ? In order for the filter (bool) to work does my tags field need to be non\_analysed? ---
**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\ **Post date:** [July 14, 2015, 5:07pm UTC](https://discuss.elastic.co/t/index-filter-and-query-strategy/25536/4 "2015-07-14T17:07:18Z")
Yes, thats exactly right. If you want to do exact matching on the tags you will need to `tags` fields to be `"index": "not_analyzed"` in your mappings. If you want to be able to search for both exact searches and for searching keywords in the `tags` field, have a look at [multifields](https://www.elastic.co/guide/en/elasticsearch/guide/current/multi-fields.html). This allows you to create two fields in your index from one field in your JSON document. So you could set one to `analyzed` to let you do keyword searching and another to `not_analyzed` to let you do exact matching. ---
**Author:** ![taylorsuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taylorsuk/32/3697_2.png) [@taylorsuk](https://discuss.elastic.co/u/taylorsuk)\ **Post date:** [July 14, 2015, 5:56pm UTC](https://discuss.elastic.co/t/index-filter-and-query-strategy/25536/5 "2015-07-14T17:56:22Z")
So I am up and running with logstash (kind of!) and have the following config ``` input { file { path => "/Users/taylorsuk/Desktop/Exercises_Single.json" type => "exercises" codec => "json" } } output { stdout { codec => rubydebug } elasticsearch { host => localhost protocol => http } } ``` I am getting `waited for 30s and no initial state was set by the discovery` do you have any ideas for where I can get info on getting setup with a simple `input (JSON) > filter / sort > ElasticSearch Index` Many thanks ---
**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\ **Post date:** [July 14, 2015, 6:06pm UTC](https://discuss.elastic.co/t/index-filter-and-query-strategy/25536/6 "2015-07-14T18:06:56Z")
hmmm, unfortunately I only have limited knowledge about troubleshooting Logstash. I have checked my logstash configs and usual I specify the protocol as `protocol => "http"` but I'm not sure the lack of double quotes is the issue here. It might be worth starting a topic in the Logstash category for this issue. The users there will almost certainly know how to solve this for you. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:01am UTC](https://discuss.elastic.co/t/index-filter-and-query-strategy/25536/7 "2017-07-06T00:01:31Z")