# Regexps with lots of punctuation **URL:** \ **Category:** Elasticsearch\ **Created:** [March 18, 2019, 4:13pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819 "2019-03-18T16:13:39Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![goochjs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/goochjs/32/32587_2.png) [@goochjs](https://discuss.elastic.co/u/goochjs)\ **Post date:** [March 18, 2019, 4:13pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/1 "2019-03-18T16:13:39Z")
Hi, We've got a text field in Elasticsearch into which is being written a JSON document, which has been escaped. We're trying to find instances with a particular pattern of fields in, but cannot formulate the regexp required. The ES field contains this precise string:- `,\"jobs\":{\"data\":[{\"type\":\"job\",\"id\":\"19029355471\"},` The ID number can change; we want to find documents with similar JSON structures. Using the Dev Tools console in Kibana, no matter how we create the regexp, we either get a `Bad string Syntax error` in the browser or an error from the API such as `json_parse_exception` / `Unrecognized character escape ':' (code 58)\n`. Example of a failing regexp:- `,\\\"jobs\\\"\:\{\\\"data\\\"\:\[\{\\\"type\\\"\:\\\"job\\\",\\\"id\\\"\:\\\"19029355471\\\"\},` We've even tried replacing each punctuation character after the comma with a `.`. This does not return an error, but nor does it return any results. This is what the full query looks like in this case:- `GET myindex*/_search { "query": { "regexp": { "fields.payload": ".*,..jobs......data.......type.....job.....id.....19029355471...,.*" } } }` I've done some searching, and some people seem to think that the `json_parse_exception` problem is with ES's JSON parser rather than Lucene. Other people seem to say that Lucene's regexp language is a little bit specialised. How can we formulate this search? Thanks, J. ---
**Author:** ![xavierfacq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavierfacq/32/8744_2.png) [@xavierfacq](https://discuss.elastic.co/u/xavierfacq)\ **Post date:** [March 19, 2019, 7:33am UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/2 "2019-03-19T07:33:20Z")
Hi, And why don't you store your JSON as a document or a nested document ? This way, you will be able to search on any field, and it will be super fast. Bye, Xavier ---
**Author:** ![goochjs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/goochjs/32/32587_2.png) [@goochjs](https://discuss.elastic.co/u/goochjs)\ **Post date:** [March 19, 2019, 8:35am UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/3 "2019-03-19T08:35:11Z")
Because the data is already in there and we were hoping it would be quicker to search it than to write something to extract and re-import. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 19, 2019, 8:52am UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/4 "2019-03-19T08:52:37Z")
Do you mean that it's only a one time search operation? ---
**Author:** ![goochjs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/goochjs/32/32587_2.png) [@goochjs](https://discuss.elastic.co/u/goochjs)\ **Post date:** [March 19, 2019, 10:34am UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/5 "2019-03-19T10:34:46Z")
Who knows? 🙂 It's data logged by an application, which we're trying to decipher in order to investigate a problem. We've found another place where this data has been logged, this time without all of the `\`s but we're still struggling to get a working regexp. ---
**Author:** ![xavierfacq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavierfacq/32/8744_2.png) [@xavierfacq](https://discuss.elastic.co/u/xavierfacq)\ **Post date:** [March 19, 2019, 10:46am UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/6 "2019-03-19T10:46:54Z")
Ok, and if you try a simple regexp: ``` ".*jobs.*data.*type.*job.*id.*19029355471.*" ``` Does it work ? Else what is the error or result ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 19, 2019, 10:49am UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/7 "2019-03-19T10:49:19Z")
What is the mapping for this field? ---
**Author:** ![goochjs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/goochjs/32/32587_2.png) [@goochjs](https://discuss.elastic.co/u/goochjs)\ **Post date:** [March 19, 2019, 12:35pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/8 "2019-03-19T12:35:00Z")
No results... ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/b/9b6de7033f3ed1c553a7b79918ca44385b554f87.png) ---
**Author:** ![goochjs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/goochjs/32/32587_2.png) [@goochjs](https://discuss.elastic.co/u/goochjs)\ **Post date:** [March 19, 2019, 12:38pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/9 "2019-03-19T12:38:22Z")
It's "text". ``` "payload": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, ``` Looking at that - I wonder what `ignore_above` means. The data is definitely over 256 characters and is definitely being stored. ---
**Author:** ![xavierfacq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavierfacq/32/8744_2.png) [@xavierfacq](https://discuss.elastic.co/u/xavierfacq)\ **Post date:** [March 19, 2019, 1:20pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/10 "2019-03-19T13:20:04Z")
> [@goochjs](#): > > Looking at that - I wonder what `ignore_above` means. The data is definitely over 256 characters and is definitely being stored. > **[ignore\_above | Elasticsearch Guide \[6.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/ignore-above.html)** ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 19, 2019, 1:52pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/11 "2019-03-19T13:52:22Z")
`payload` is using a `text` type. It's then analyzed with the default analyzer, ie. broken into tokens. `payload.keyword` is indexed as a whole string but only for "small" texts. Doing a regex (non analyzed) on `payload` which is analyzed is not going to produce what you are expecting. I'd highly recommend using the right mapping for your use case (it depends on what you want to do) and reindex. Have a look at the `_analyze` API to see how your field has been indexed behind the scene. You'd be able to search within the tokens that have been generated but not in the whole initial string. ---
**Author:** ![goochjs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/goochjs/32/32587_2.png) [@goochjs](https://discuss.elastic.co/u/goochjs)\ **Post date:** [March 19, 2019, 2:35pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/12 "2019-03-19T14:35:32Z")
@xavierfacq Obviously I read that. It's not entirely clear, however, as the data **is** being stored, despite what the docs say. @dadoonet Thanks. I wasn't aware of the _analyze_ API. The data field we've got is huge, however, and it's hard to POST into the API. We'll have to reindex, I think. All-in-all, I guess @xavierfacq gave the right answer to begin with - we will have to get this data in as a proper JSON document so that it is fully indexed. The next issue we will hit, however, is ES's [limited support for JSON arrays](https://blog.abhi.host/blog/2016/11/24/logstashelasticsearch-best-way-to/). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 19, 2019, 3:14pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/13 "2019-03-19T15:14:06Z")
> [@goochjs](#): > > Obviously I read that. It's not entirely clear, however, as the data **is** being stored, despite what the docs say. Store does not mean indexed. The `_source` document stores the whole JSON content whatever is happening on each field then. > We'll have to reindex, I think. Yes. That's indeed what @xavierfacq proposed at the beginning. > [@goochjs](#): > > The next issue we will hit, however, is ES's [limited support for JSON arrays](https://blog.abhi.host/blog/2016/11/24/logstashelasticsearch-best-way-to/). Not sure this article is very accurate. That's what `nested` type has been made for. ---
**Author:** ![goochjs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/goochjs/32/32587_2.png) [@goochjs](https://discuss.elastic.co/u/goochjs)\ **Post date:** [March 19, 2019, 4:26pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/14 "2019-03-19T16:26:24Z")
Thanks again. Re: documentation, that page says (with my emphasis):- > Strings longer than the `ignore_above` setting will not be indexed **or stored** Re: arrays - we've had a problem storing these in ES elsewhere and have had to flatten them into hashes. I guess the `nested` data type is not being picked automatically by ES. Anyway, all help much appreciated. As recommended, we'll go back and fix this at source. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 19, 2019, 4:41pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/15 "2019-03-19T16:41:00Z")
Re: documentation, that page says (with my emphasis):- > Strings longer than the `ignore_above` setting will not be indexed **or stored** This is true. When you use the `store: true` option on a given field. Defaults to `false`. > I guess the `nested` data type is not being picked automatically by ES. True. You need to be explicit. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 16, 2019, 4:50pm UTC](https://discuss.elastic.co/t/regexps-with-lots-of-punctuation/172819/16 "2019-04-16T16:50:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.