# How to find only requests for /robots.txt?

**URL:** <https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343>\
**Category:** Elasticsearch\
**Created:** [June 20, 2016, 4:19pm UTC](https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343 "2016-06-20T16:19:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![fikse](https://avatars.discourse-cdn.com/v4/letter/f/c0e974/32.png) [@fikse](https://discuss.elastic.co/u/fikse)\
**Post date:** [June 20, 2016, 4:19pm UTC](https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343/1 "2016-06-20T16:19:51Z")

</div>

I've tried everything I can to find this single index:

```auto
{
  "_index": "logstash-2015.12.15",
  "_type": "logs",
  "_id": "AVVVW_wekW28kZ0eUiUh",
  "_version": 1,
  "_score": 1,
  "_source": {
    "message": "127.0.0.1 - - [15/Dec/2015:18:50:15 -0500] "GET /robots.txt HTTP/1.1" 200 172 "-" "-"",
    "@version": "1",
    "@timestamp": "2015-12-15T23:50:15.000Z",
    "type": "log",
    "host": "abc123.local",
    "remote_ip": "127.0.0.1",
    "user_name": "-",
    "time": "15/Dec/2015:18:50:15 -0500",
    "request_action": "GET",
    "request": "/robots.txt",
    "http_version": "1.1",
    "response": "200",
    "bytes": "172",
    "referrer": "-",
    "agent": "-",
    "user_agent": {
      "name": "Other",
      "os": "Other",
      "os_name": "Other",
      "device": "Other"
    }
  }
}

```

The `request` field is `not_analyzed`.  
How can I query for `/robots.txt`? I don't want other results such as `/overlay.html` or `/index.html`. I just want `/robots.txt`.

---

<div class="post-metadata">

**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\
**Post date:** [June 20, 2016, 4:21pm UTC](https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343/2 "2016-06-20T16:21:43Z")

</div>

Did you try a term query?

See here [https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-term-query.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-term-query.html) for more information.

Hope this helps,  
Isabel

---

<div class="post-metadata">

**Author:** ![fikse](https://avatars.discourse-cdn.com/v4/letter/f/c0e974/32.png) [@fikse](https://discuss.elastic.co/u/fikse)\
**Post date:** [June 20, 2016, 4:33pm UTC](https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343/3 "2016-06-20T16:33:42Z")

</div>

I did, my current query looks like this:

```auto
body = new Bodybuilder().query('match', 'request_action', 'GET')
                        .filter('term', 'request', '/robots.txt')
                        .build()
es.search(body)

```

---

<div class="post-metadata">

**Author:** ![fikse](https://avatars.discourse-cdn.com/v4/letter/f/c0e974/32.png) [@fikse](https://discuss.elastic.co/u/fikse)\
**Post date:** [June 20, 2016, 4:46pm UTC](https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343/4 "2016-06-20T16:46:33Z")

</div>

@mainec I figured out the issue:  
I needed to use`logs` instead of `log`.

---

<div class="post-metadata">

**Author:** ![fikse](https://avatars.discourse-cdn.com/v4/letter/f/c0e974/32.png) [@fikse](https://discuss.elastic.co/u/fikse)\
**Post date:** [June 20, 2016, 5:01pm UTC](https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343/5 "2016-06-20T17:01:02Z")

</div>

```auto
es.search({
    index: 'logstash-*',
    type: 'logs',
    body: { query: { match: { request: "/robots.txt" }}}
})

```

@mainec The above does actually give me results with `/robots.txt` but it doesn't give me results with _just_ requests to `/robots.txt`. Would a filter remove results which aren't requests to `/robots.txt`?

---

<div class="post-metadata">

**Author:** ![fikse](https://avatars.discourse-cdn.com/v4/letter/f/c0e974/32.png) [@fikse](https://discuss.elastic.co/u/fikse)\
**Post date:** [June 20, 2016, 5:39pm UTC](https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343/6 "2016-06-20T17:39:57Z")

</div>

Here is my latest attempt which yields 0 results:

```auto
  body: { query: { bool: {
    filter: [term: { request: '/robots.txt' }]
  }}} 

```

---

<div class="post-metadata">

**Author:** ![fikse](https://avatars.discourse-cdn.com/v4/letter/f/c0e974/32.png) [@fikse](https://discuss.elastic.co/u/fikse)\
**Post date:** [June 20, 2016, 5:40pm UTC](https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343/7 "2016-06-20T17:40:36Z")

</div>

Am I unable to filter on the `request` field because it is within the `_source` field?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:42pm UTC](https://discuss.elastic.co/t/how-to-find-only-requests-for-robots-txt/53343/8 "2017-07-05T22:42:08Z")

</div>