# Changing Elasticsearch + Fscrawler tokenizing methods to recognize Social Security Numbers/Passport Machine Readable Zones

**URL:** <https://discuss.elastic.co/t/changing-elasticsearch-fscrawler-tokenizing-methods-to-recognize-social-security-numbers-passport-machine-readable-zones/251502>\
**Category:** Elasticsearch\
**Created:** [October 8, 2020, 7:30pm UTC](https://discuss.elastic.co/t/changing-elasticsearch-fscrawler-tokenizing-methods-to-recognize-social-security-numbers-passport-machine-readable-zones/251502 "2020-10-08T19:30:01Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![morningkaren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/morningkaren/32/68924_2.png) [@morningkaren](https://discuss.elastic.co/u/morningkaren)\
**Post date:** [October 8, 2020, 7:30pm UTC](https://discuss.elastic.co/t/changing-elasticsearch-fscrawler-tokenizing-methods-to-recognize-social-security-numbers-passport-machine-readable-zones/251502/1 "2020-10-08T19:30:02Z")

</div>

Hello,

I have used fscrawler to index some files that have social security numbers in the format of `XXX-XX-XXXX` and also the machine readable zones of passports such as `<<<<<<Karen<Ouyang<<<<<`. They appear in the content when I do a general search of what is in my index.

(I used the default fscrawler 2.7 settings to tokenize the index.)

But, when I use Kibana and query\_string search to find social security numbers and the passport MRZs, I don't get any results back. Here are the queries I used:

```auto
GET /test_10082020_v3/_search
{"query": {"query_string" : {"query" : "*<<<*", "default_field" : "content"}}, "_source":{"excludes":["content"]},"highlight": {"fields": {"*": {}}}, "size": 10000}

```

```auto
GET /test_10082020_v3/_search
{"query": {"query_string" : {"query" : "/[0-9]{3}-[0-9]{2}-[0-9]{4}/", "default_field" : "content"}}, "_source":{"excludes":["content"]},"highlight": {"fields": {"*": {}}}, "size": 10000}

```

I want to asK:

1. Is there a better search query to query with to find the social security numbers and the passport machine readable zones?

2. Do I need to change the way the source is tokenized in fscrawler settings, so that I can search for SSNs and passport MRZs easier? If so, what do I change it to? The tokenizer is currently `fscrawler_path` and type `path_hierarchy`.

Thank you.

Karen

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 12, 2020, 3:01pm UTC](https://discuss.elastic.co/t/changing-elasticsearch-fscrawler-tokenizing-methods-to-recognize-social-security-numbers-passport-machine-readable-zones/251502/2 "2020-10-12T15:01:26Z")

</div>

Hi

Some thoughts about this.

First, the fact you are using FSCrawler to index documents is not really the problem so I'd remove FSCrawler from the picture for now.

Your main concern is that you would like to find which documents contains a SSN or a passport MRZ, right?  
Or do you want to be able to find a document which contains `Karen` as a passport MRZ?

If this is one of the use case of your project, I'd try to get that information at index time instead of doing a very slow search...  
It could be by generating fields like `hasSSN`and `hasMRZ` boolean values.

```
{
  "content": "WHATEVER HERE <<<<<<Karen<Ouyang<<<<< AND XXX-XX-XXXX",
  "hasSSN": true,
  "hasMRZ": true
}

```

Or generating fields like `ssn` and `mrz`...

```
{
  "content": "WHATEVER HERE <<<<<<Karen<Ouyang<<<<< AND XXX-XX-XXXX",
  "mrz": "Karen Ouyang",
  "ssn": "XXX-XX-XXXX"
}

```

To do that, I'd try to use an ingest pipeline that will generate those fields at index time.  
FSCrawler supports this `pipeline` option.

**Side note:**

> [@morningkaren](#):
>
> The tokenizer is currently `fscrawler_path` and type `path_hierarchy` .

The tokenizer you mentioned is defined by FSCrawler only for some fields but not for the `content` field.

---

<div class="post-metadata">

**Author:** ![morningkaren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/morningkaren/32/68924_2.png) [@morningkaren](https://discuss.elastic.co/u/morningkaren)\
**Post date:** [October 12, 2020, 6:51pm UTC](https://discuss.elastic.co/t/changing-elasticsearch-fscrawler-tokenizing-methods-to-recognize-social-security-numbers-passport-machine-readable-zones/251502/3 "2020-10-12T18:51:25Z")

</div>

Hi David,

Thanks for letting me know about the pipeline option. I think it will become useful in one of our other use cases.

However, for the SSN/Passport use case, I would like to use the slow search method.

The tokenizer that was successful in picking up whole SSNs and the \<\<\< signs in the passport MRZs was the whitespace tokenizer. This was the code we used in our \_settings.json :

```auto
{
  "settings": {
    "number_of_shards": 4,
    "index.mapping.total_fields.limit": 2000,
     "analysis": {
      "analyzer": {
        "fulltext_analyzer": {
          "type": "custom",
          "tokenizer": "whitespace",
          "filter": [
            "lowercase",
            "type_as_payload"
          ]
        }
      }
    }
  },
"mappings": {
    "properties": {
      "content": {
        "type": "text",
        "term_vector": "with_positions_offsets_payloads",
        "store" : true,
        "analyzer" : "fulltext_analyzer"
       }
    }
  }
}

```

However, even though we were able to search for the \<\< signs and SSNs, we had files that were not indexed, so we missed some files this way.

When we use the default fscrawler settings, we were able to index all the files, but were not able to search for the \<\< signs and the SSNs using query\_string.

So, my question is, is there a way to uses the whitespace tokenizer and also index all the files? I see that the difference between our modified default settings and the old default settings is that there are less defined mappings in the modified version. So, is our modified settings incomplete in that way which caused some files to not be indexed?

Thanks!

Best,

Karen

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 13, 2020, 9:36am UTC](https://discuss.elastic.co/t/changing-elasticsearch-fscrawler-tokenizing-methods-to-recognize-social-security-numbers-passport-machine-readable-zones/251502/4 "2020-10-13T09:36:38Z")

</div>

> [@morningkaren](#):
>
> So, my question is, is there a way to uses the whitespace tokenizer and also index all the files?

Yes.

> [@morningkaren](#):
>
> So, is our modified settings incomplete in that way which caused some files to not be indexed?

Yes.

If you need to apply two different analysis strategies, you need to create a sub field in the mapping.  
Something like:

```auto
{
  "settings": {
    "number_of_shards": 4,
    "index.mapping.total_fields.limit": 2000,
     "analysis": {
      "analyzer": {
        "fulltext_analyzer": {
          "type": "custom",
          "tokenizer": "whitespace",
          "filter": [
            "lowercase",
            "type_as_payload"
          ]
        }
      }
    }
  },
"mappings": {
    "properties": {
      "content": {
        "type": "text",
        "term_vector": "with_positions_offsets_payloads",
        "store" : true,
        "analyzer" : "fulltext_analyzer",
        "fields": {
           "foo": {
             "type": "text"
           }
        }
       }
    }
  }
}

```

That way, searching on `text.foo` will allow you to use the standard default analyzer.

---

<div class="post-metadata">

**Author:** ![morningkaren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/morningkaren/32/68924_2.png) [@morningkaren](https://discuss.elastic.co/u/morningkaren)\
**Post date:** [October 15, 2020, 8:34pm UTC](https://discuss.elastic.co/t/changing-elasticsearch-fscrawler-tokenizing-methods-to-recognize-social-security-numbers-passport-machine-readable-zones/251502/5 "2020-10-15T20:34:09Z")

</div>

Hi David,

That was really helpful! We were able to change the settings and crawl+ search through everything.

Thank you.

Karen

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 12, 2020, 8:34pm UTC](https://discuss.elastic.co/t/changing-elasticsearch-fscrawler-tokenizing-methods-to-recognize-social-security-numbers-passport-machine-readable-zones/251502/6 "2020-11-12T20:34:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.