# How do I assign a second analyzer to my mapping

**URL:** <https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728>\
**Category:** Elasticsearch\
**Created:** [November 30, 2017, 10:20am UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728 "2017-11-30T10:20:42Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [November 30, 2017, 10:20am UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/1 "2017-11-30T10:20:42Z")

</div>

HI All,  
Here is my index so far:

```
put fullsiteindex
{
  "settings" : {
    "number_of_shards":3,
    "number_of_replicas":2,
    "analysis": {
             "filter": {
                 "my_stop": {
                     "type": "stop",
                     "stopwords": "_english_"
                 },
                 "english_stemmer" : {
           		       "type": "stemmer",
          		      "language": "english"
    	            },
    	           "vbscript": {
    	              "type" : "pattern_replace",
    	              "pattern": "<\\%*\\%>",
    	              "replacement": ""
    	            }

             },
"analyzer" : {
     		    "english": {
        		 	"tokenizer" : "standard",
        			 "filter": [
           		 		"lowercase",
            				"my_stop",
           		 		"english_stemmer"
      		 	]
    		        },
    		    "scripttoken": {
    		        "tokenizer" : "pattern",
    		        "filter": ["vbscript"]
    		    }
    		  }
        }
  },
  "mappings" : {
    "site": {
      "properties": {
"data": {
  "type": "text",
  "analyzer": "english",
  "search_analyzer": "english"
}
  }
  }
  }
}

```

How do I add my scripttoken analyzer to my mappings property since it needs both analyzers assigned.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 30, 2017, 11:12am UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/2 "2017-11-30T11:12:37Z")

</div>

You can do something like:

```auto
{
   "settings":{
      "number_of_shards":3,
      "number_of_replicas":2,
      "analysis":{
         "filter":{
            "my_stop":{
               "type":"stop",
               "stopwords":"_english_"
            },
            "english_stemmer":{
               "type":"stemmer",
               "language":"english"
            },
            "vbscript":{
               "type":"pattern_replace",
               "pattern":"<\\%*\\%>",
               "replacement":""
            }
         },
         "analyzer":{
            "english":{
               "tokenizer":"standard",
               "filter":[
                  "lowercase",
                  "my_stop",
                  "english_stemmer"
               ]
            },
            "scripttoken":{
               "tokenizer":"pattern",
               "filter":[
                  "vbscript"
               ]
            }
         }
      }
   },
   "mappings":{
      "site":{
         "properties":{
            "data":{
               "type":"text",
               "analyzer":"english",
               "search_analyzer":"english",
               "fields": {
                 "foo": {
                   "type": "text",
                   "analyzer": "scripttoken"
                 }
               }
            }
         }
      }
   }
}

```

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [November 30, 2017, 12:09pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/3 "2017-11-30T12:09:18Z")

</div>

I changed my filter to be:

```
"vbscript": {
    	              "type" : "pattern_replace",
    	              "pattern": "(?:..)[^<%]+[^%>](?:..)",
    	              "replacement": ""
    	            }

```

as I realised the Regex was wrong. I made the following change to the mapping as per your example. I called my field "content" since this is the name of the field n my \_source. I then re-indexed.

Using the following Query still returns me the script in the content, which I did not expect:

> GET fullsiteindex/\_search  
> {  
> "query": { "query\_string" : { "query" :"my query string", "fields" :["content"] }}, "\_source" :["content", "file.url", "path.virtual", "meta.title", "file.last\_modified"]  
> }

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 30, 2017, 12:24pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/4 "2017-11-30T12:24:33Z")

</div>

Elasticsearch does not change the content of the field. That’s probably why you are still seeing it.  
It should not index that content though.

If you want to remove that from the source, try ingest node feature and preprocess the document before it gets indexed.

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [November 30, 2017, 1:41pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/5 "2017-11-30T13:41:21Z")

</div>

I'm using FSCrawler to create this index. Can I still use ingest? I'm just looking for a good example now

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [November 30, 2017, 2:00pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/6 "2017-11-30T14:00:58Z")

</div>

I also just run the following query where my query string is a variable defined within my script block and it still returned 2 hits.

```
GET fullsiteindex/_search
{ 
  "query": { "query_string" : { "query" :"arListHeadOffice", "fields" :["content"] }}, "_source" :["content", "file.url", "path.virtual", "meta.title", "file.last_modified"]
}

```

So the filter hasn't worked

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [November 30, 2017, 4:45pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/7 "2017-11-30T16:45:09Z")

</div>

as a starter for 10 I think I need to use a 'split' processor, however, I don't think this is complete because I haven't said what to do with it now I've split it out. How do I expand this to discard text in the separator? I'm guessing I need to introduce a 'foreach' somewhere in here since I have no way of knowing how many script blocks or where they are in a page:

```
put _ingest/pipeline/removescript
{
  "pipeline" :{
    "description": "remove script",
    "processors": [
      { "split" :{
        "field": "content",
        "separator": "(?:..)[^<%]+[^%>](?:..)"
        }
      }]
  }
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 30, 2017, 7:44pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/8 "2017-11-30T19:44:09Z")

</div>

May be gsub would be a better fit? [https://www.elastic.co/guide/en/elasticsearch/reference/current/gsub-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/gsub-processor.html)

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [December 1, 2017, 8:32am UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/9 "2017-12-01T08:32:52Z")

</div>

HI dadoonet, Yes, the gsub looks a better fit. I've changed the pipeline to be:

```
put _ingest/pipeline/removescript
{
  "pipeline" :{
    "description": "remove script",
    "processors": [
      { "gsub" :{
        "field": "content",
        "pattern": "(?:..)[^<%]+[^%>](?:..)",
        "replacement": ""
        }
      }]
  }
}

```

but I get the following error:

```
{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "[processors] required property is missing",
        "header": {
          "property_name": "processors"
        }
      }
    ],
    "type": "parse_exception",
    "reason": "[processors] required property is missing",
    "header": {
      "property_name": "processors"
    }
  },
  "status": 400
}
```

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [December 1, 2017, 10:57am UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/10 "2017-12-01T10:57:32Z")

</div>

I have finally managed to create the pipeline with:

```
put _ingest/pipeline/removescript
{

    "description" : "remove script",
    "processors": [
      { "gsub" :{
        "field": "content",
        "pattern": "(?:..)[^<%]+[^%>](?:..)",
        "replacement": ""
        }
      }]
}

```

My issue now is that when I try to create the index with :

`put fullsiteindex?pipeline=removescript {......`

it errors with illegal\_request. unrecognized parameter [pipeline]

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 1, 2017, 11:23am UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/11 "2017-12-01T11:23:34Z")

</div>

The pipeline must be applied per document

```
PUT index/doc/1?pipeline=foo
{
}
```

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [December 1, 2017, 11:44am UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/12 "2017-12-01T11:44:03Z")

</div>

I'm using FSCrawler since we are talking over 1000 documents. How can I apply it using this?

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [December 1, 2017, 12:18pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/13 "2017-12-01T12:18:10Z")

</div>

In my \_settings file within elasticsearch I added a reference to the pipeline:

```
"elasticsearch" : {
    "nodes" : [ {
      "pipeline" : "removescript",
      "host" : "localhost",
      "port" : 9200,
      "scheme" : "HTTP"
    } ],

```

I deleted the \_status file and re-ran the index. The txt still appears in my document. I know my pattern is ok as I've tried it via a number of online regex test tools.

basically in the content of some documents there may be text in the form **Hello world \<% globalheadoffice = true ..... %\> from here** so I don't want anything to appear that is between the \<% %\> values or those values themselves. But I do want to see in the document **Hello world from here**

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 1, 2017, 12:29pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/14 "2017-12-01T12:29:51Z")

</div>

Any chance you are using FSCrawler REST endpoint?

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [December 1, 2017, 1:35pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/15 "2017-12-01T13:35:04Z")

</div>

Hi dadoonet,

I think the pipeline is working for a degree. That is to say that some of the text has been removed but not all. If I use [www.regextester.com](http://www.regextester.com) this replaces everything that I expect. If I use [freeformatter.com](http://freeformatter.com) it doesn't so I guess my next question is what regex parser does elasticsearch use/conform too?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 1, 2017, 2:11pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/16 "2017-12-01T14:11:37Z")

</div>

The Java one according to [https://github.com/elastic/elasticsearch/blob/master/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/GsubProcessor.java](https://github.com/elastic/elasticsearch/blob/master/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/GsubProcessor.java)

[https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html](https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html)

---

<div class="post-metadata">

**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\
**Post date:** [December 1, 2017, 3:41pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/17 "2017-12-01T15:41:25Z")

</div>

it is strange. changing to a Java regex parser, if I change my pattern to be `(?:..)[^<%]+[^%>](?:..)+g`

with my test script, it get's me close but not close enough, but going with this for now, I recreated my pipeline and index, repopulated with FsCrawler, but the text remained the same (i.e. not as per Java regex test harness).

Not sure where to go from here.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 1, 2017, 5:58pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/18 "2017-12-01T17:58:31Z")

</div>

You need a regex expert maybe, which I'm not! 😔

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 29, 2017, 5:58pm UTC](https://discuss.elastic.co/t/how-do-i-assign-a-second-analyzer-to-my-mapping/109728/19 "2017-12-29T17:58:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.