# Need help with regular expression filtering

**URL:** <https://discuss.elastic.co/t/need-help-with-regular-expression-filtering/97801>\
**Category:** Kibana\
**Created:** [August 21, 2017, 4:46pm UTC](https://discuss.elastic.co/t/need-help-with-regular-expression-filtering/97801 "2017-08-21T16:46:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![UnitedMarsupials](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/unitedmarsupials/32/19429_2.png) [@UnitedMarsupials](https://discuss.elastic.co/u/UnitedMarsupials)\
**Post date:** [August 21, 2017, 4:46pm UTC](https://discuss.elastic.co/t/need-help-with-regular-expression-filtering/97801/1 "2017-08-21T16:46:46Z")

</div>

I'm trying to filter by HTTP "Referer" header. The values are too "hairy" for a simple matching, requiring regular expressions. I start with a simple query:

```
{
  "query": {
    "regexp": {
      "http_referer": {
        "value": "http"
      }
    }
  }
}

```

and get a bunch of hits, as expected. So far so good. However, when I try to expand the expression to read `"http:"`, I suddenly get none at all -- and there certainly are some, because not all referrers begin with `https`.

If I change the value to `"https"`, I get a couple of hits, where the string "https" occurs _in the middle_ of the referrer-field -- which is bizarre for two reasons:

- The [manual](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/query-dsl-regexp-query.html#regexp-syntax) states, regular expressions are _always_ anchored
- We have a whole bunch of referrers that _begin_ with "https" -- but those aren't shown

Something is seriously wrong -- I must not be understanding, how the regexps work in Kibana. Any advice? Our Kibana version 4.5.0 (build 9889). Thanks!

(Just double-checked -- the field `http_referer` is marked as both "Analyzed" and "Indexed" in Kibana's settings.)

---

<div class="post-metadata">

**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\
**Post date:** [August 22, 2017, 5:20pm UTC](https://discuss.elastic.co/t/need-help-with-regular-expression-filtering/97801/2 "2017-08-22T17:20:33Z")

</div>

This is a bit of a long-shot but have you tried escaping the colon character?

```auto
{
  "query": {
    "regexp": {
      "http_referer": {
        "value": "http\:"
      }
    }
  }
}

```

Also, you might want to try asking this question in the [Elasticsearch forum](https://discuss.elastic.co/c/elasticsearch), since it is a little more relevant to that part of our stack.

Thanks,  
CJ

---

<div class="post-metadata">

**Author:** ![UnitedMarsupials](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/unitedmarsupials/32/19429_2.png) [@UnitedMarsupials](https://discuss.elastic.co/u/UnitedMarsupials)\
**Post date:** [August 22, 2017, 5:40pm UTC](https://discuss.elastic.co/t/need-help-with-regular-expression-filtering/97801/3 "2017-08-22T17:40:48Z")

</div>

With a single backslash I can not even press the "Done" button in Kibana's interface, because it is invalid JSON. With two backslashes -- `"http\\:"` -- I get no results either...

---

<div class="post-metadata">

**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\
**Post date:** [August 22, 2017, 7:43pm UTC](https://discuss.elastic.co/t/need-help-with-regular-expression-filtering/97801/4 "2017-08-22T19:43:04Z")

</div>

Bah, sorry, I've never hand-rolled a filter query before like this. Dumb suggestion on my part. I took a closer look at the docs and did some experimentation. Can we start off with a very basic regexp filter like this:

```auto
{
  "query": {
    "regexp": {
      "http_referer": "https.*"
    }
  }
}

```

This will match all http\_referer values which begin with https followed by any characters. Can you also check if there's a `http_referer.raw` field? If so, you can try querying that instead:

```auto
{
  "query": {
    "regexp": {
      "http_referer.raw": "https.*"
    }
  }
}

```

Please let me know if this helps.

Thanks,  
CJ

---

<div class="post-metadata">

**Author:** ![UnitedMarsupials](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/unitedmarsupials/32/19429_2.png) [@UnitedMarsupials](https://discuss.elastic.co/u/UnitedMarsupials)\
**Post date:** [August 23, 2017, 1:21am UTC](https://discuss.elastic.co/t/need-help-with-regular-expression-filtering/97801/5 "2017-08-23T01:21:28Z")

</div>

Thanks. Both methods seem to work as quoted. However, if I try to add colon: `"https:.*"` to the simple `"http_referer"`, things break. The `"http_referer.raw"` works better -- it finds hits. But the highlighting in Kibana's interface breaks -- nothing is highlighted.

But, I can live without the highlighting -- and will continue to use the `.raw` in my searches, however surprising it is, that this is necessary... Thank you!

---

<div class="post-metadata">

**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\
**Post date:** [August 23, 2017, 1:59am UTC](https://discuss.elastic.co/t/need-help-with-regular-expression-filtering/97801/6 "2017-08-23T01:59:04Z")

</div>

That's great! If you think the missing highlighting could be a bug, then I think the team would especially like to learn more about it in a [GitHub issue](https://github.com/elastic/kibana/issues/new) if you have time.

Thanks,  
CJ

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 20, 2017, 1:59am UTC](https://discuss.elastic.co/t/need-help-with-regular-expression-filtering/97801/7 "2017-09-20T01:59:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.