I am running Kibana 5x in AWS elasticsearch. I can't find a way to exclude a dash - when searching. I've tried...
-referrer:-
-referrer:"-"
-referrer:-*
And none seem to work.
I don't have data on hand to verify, but does it work if you escape the - with a backslash: \-?
"Reserved" characters need to be escaped, and dash is a reserved character: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters
Negative (get it!) didn't work...
I tried both of these...
request:jpg OR request:png -referrer:-
request:jpg OR request:png -referrer:"-"
Kibana renders a null value in the Elasticsearch data as -. You probably want to check on the data in ES (you can use Console in Dev Tools for this), to see if the values for this data is actually null.
You probably want to use _exists_:referer in your search.
How do I check the data in console? Just using "exists:referrer" in Discover search does not change the results.
Here are typical log lines from Apache and their mapping, note the dashes when fields are empty.
123.456.78.9 - - [15/Feb/2017:18:33:25 +0000] example.com "GET / HTTP/1.1" 200 42766 "http://facebook.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_2 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A456 Safari/602.1" Server=aws8 SSL=- 8868 0
123.456.78.9 - - [15/Feb/2017:18:33:25 +0000] example.com "GET /js/lib/jquery-ui/jquery-ui.js HTTP/1.1" 200 42766 "http://example.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_2 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A456 Safari/602.1" Server=aws8 SSL=- 8868 0
[ip, user, username, timestamp, vhost, request, status_code, bytes, referrer, browser, server, ssl, timems, times]
You could try a query such as
GET my_index/my_type/_search
{
"_source": [ "referer", "@timestamp" ],
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
To check on just the referer fields with their timestamps, sorted descending order. That will show the newest 10 records in raw data. If you need to see more to find what you're looking for, you can add a size parameter.
Can you also provide the mapping for the field?
Query I used:
GET cwl*/_search
{
"_source": [ "referrer", "@timestamp" ],
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
Response:
{
"took": 141,
"timed_out": false,
"_shards": {
"total": 135,
"successful": 135,
"failed": 0
},
"hits": {
"total": 1457622,
"max_score": null,
"hits": [
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584323017469410618937880535964300494310921216000000",
"_score": null,
"_source": {
"referrer": "-",
"@timestamp": "2017-03-09T17:13:01.305Z"
},
"sort": [
1489079581305
]
},
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584270276207016094014146789076066613080893805690881",
"_score": null,
"_source": {
"referrer": "-",
"@timestamp": "2017-03-09T17:12:58.940Z"
},
"sort": [
1489079578940
]
},
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584247953161072364860382111822075692071026319294464",
"_score": null,
"_source": {
"referrer": "-",
"@timestamp": "2017-03-09T17:12:57.939Z"
},
"sort": [
1489079577939
]
},
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584169744447661117965018875732689090550914860187689",
"_score": null,
"_source": {
"referrer": "http://example.com/",
"@timestamp": "2017-03-09T17:12:54.432Z"
},
"sort": [
1489079574432
]
},
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584169744447661117965018875732689090550914860187694",
"_score": null,
"_source": {
"referrer": "http://example.com/",
"@timestamp": "2017-03-09T17:12:54.432Z"
},
"sort": [
1489079574432
]
},
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584169744447661117965018875732689090550914860187697",
"_score": null,
"_source": {
"referrer": "http://example.com/",
"@timestamp": "2017-03-09T17:12:54.432Z"
},
"sort": [
1489079574432
]
},
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584169744447661117965018875732689090550914860187698",
"_score": null,
"_source": {
"referrer": "http://example.com/",
"@timestamp": "2017-03-09T17:12:54.432Z"
},
"sort": [
1489079574432
]
},
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584169744447661117965018875732689090550914860187701",
"_score": null,
"_source": {
"referrer": "http://example.com/",
"@timestamp": "2017-03-09T17:12:54.432Z"
},
"sort": [
1489079574432
]
},
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584169744447661117965018875732689090550914860187706",
"_score": null,
"_source": {
"referrer": "http://example.com/",
"@timestamp": "2017-03-09T17:12:54.432Z"
},
"sort": [
1489079574432
]
},
{
"_index": "cwl-2017.03.09",
"_type": "/var/log/httpd/access_log",
"_id": "33207584169744447661117965018875732689090550914860187708",
"_score": null,
"_source": {
"referrer": "http://example.com/",
"@timestamp": "2017-03-09T17:12:54.432Z"
},
"sort": [
1489079574432
]
}
]
}
}Field mapping:
{
"cwl-2017.03.09": {
"mappings": {
"/var/log/httpd/access_log": {
"properties": {
"@id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@log_group": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@log_stream": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@owner": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@timestamp": {
"type": "date"
},
"authuser": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"bytes": {
"type": "long"
},
"date": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ident": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"referrer": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"request": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"server": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ssl": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"status": {
"type": "long"
},
"time_ms": {
"type": "long"
},
"time_s": {
"type": "long"
},
"useragent": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"vhost": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}My theory about making an expression to match not-null was wrong -- sorry about that.
I created an index with your mapping and indexed a doc that has a - and another doc that doesn't. I wasn't able to use any of the documented tricks to create a query that excludes those - value docs.
This would be way easier for you if this data is going through Logstash, in which case you can create a filter that checks for - values before adding them to the output to Elasticsearch. Then those fields would be inherently null, and you could just use _exists_:referrer as a query to exclude them. For more help on that, or on re-indexing your existing data, I'd suggest asking in the Logstash discuss category.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.