How to find only requests for /robots.txt?

Elastic Stack Elasticsearch
1

I've tried everything I can to find this single index:

{
  "_index": "logstash-2015.12.15",
  "_type": "logs",
  "_id": "AVVVW_wekW28kZ0eUiUh",
  "_version": 1,
  "_score": 1,
  "_source": {
    "message": "127.0.0.1 - - [15/Dec/2015:18:50:15 -0500] "GET /robots.txt HTTP/1.1" 200 172 "-" "-"",
    "@version": "1",
    "@timestamp": "2015-12-15T23:50:15.000Z",
    "type": "log",
    "host": "abc123.local",
    "remote_ip": "127.0.0.1",
    "user_name": "-",
    "time": "15/Dec/2015:18:50:15 -0500",
    "request_action": "GET",
    "request": "/robots.txt",
    "http_version": "1.1",
    "response": "200",
    "bytes": "172",
    "referrer": "-",
    "agent": "-",
    "user_agent": {
      "name": "Other",
      "os": "Other",
      "os_name": "Other",
      "device": "Other"
    }
  }
}

The request field is not_analyzed.
How can I query for /robots.txt? I don't want other results such as /overlay.html or /index.html. I just want /robots.txt.

2

Did you try a term query?

See here https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-term-query.html for more information.

Hope this helps,
Isabel

3

I did, my current query looks like this:

body = new Bodybuilder().query('match', 'request_action', 'GET')
                        .filter('term', 'request', '/robots.txt')
                        .build()
es.search(body)
4

@mainec I figured out the issue:
I needed to uselogs instead of log.

5 
es.search({
    index: 'logstash-*',
    type: 'logs',
    body: { query: { match: { request: "/robots.txt" }}}
})

@mainec The above does actually give me results with /robots.txt but it doesn't give me results with just requests to /robots.txt. Would a filter remove results which aren't requests to /robots.txt?

6

Here is my latest attempt which yields 0 results:

  body: { query: { bool: {
    filter: [ term: { request: '/robots.txt' } ]
  }}}
7

Am I unable to filter on the request field because it is within the _source field?