These Elastic Discuss forums are a fantastic source of help if you get stuck, but they also a treasure trove of hints that sometimes get lost in the volume. These are a few awesome posts we've been collecting for a time just like this one, so we hope you like them as the closing topic for our 2017 Advent Series!
Kibana
Calculating deltas between time fields
The always awesome @Stacey_Gammon shows how you can use a scripted field to show deltas (ie changes in rate) between two fields in a document, and display that result in a table, which is fantastic for things like elapsed times for an event stream! Things like;
- Run times of a process
- How long a job ran
- Or the length of a session on your website
Simulating a trip on a heatmap
Currently we’re working on a way to map out a route or a trip on a map in Kibana. But if your need is pressing then @thomasneirynck is here to help!
If you index every GPS-coordinate as a single document in Elasticsearch, you can use Kibana to create a heatmap of the data. This uses the ES geohash-aggregation, which aggregates these points on a grid. Kibana does position that aggregated point in the geometric-middle of all the points it aggregated. That will give you a visual effect of more or less that route, but not give you access to the individual points of the route.
Elasticsearch
Advanced boolean filtering
Did you know you can use a bool query with should clauses in order to invert an exists query using a must_not clause? If that’s got you confused, this allows you to search on a specific field value while also looking for things like null values. Thanks to @forloop for this solution!
{
"query": {
"bool" : {
"should" : [
{ "bool" : {
"must_not" : [
{ "exists": { "field" : "field_value" } }
]
}
},
{ "term" : { "brand_name" : "some_other_field_value" } }
]
}
}
}
What happens during an alias change?
This topic, that was answered by forum regular @dadoonet, was specifically for a question around the _rollover API, but the underlying method applies to a change to any alias you may make;
What happens when you rollover:
- It creates a new index (You are still indexing to the old index through the alias)
- It switches the alias. This switch is super fast. If you have a pending operation it might go to the old index.
Once done, the new requests will go to the new index.
Logstash
Extracting hashtags with logstash-input-twitter
Here at Elastic, we use the Twitter input plugin to monitor various things on Twitter. It’s super easy to use to pull down tweets from one or more accounts or for various hashtags. If you add the following section into your Twitter pipeline you can also extract any hashtags and get even smarter analytics from your feed. We include the lowercase directive and ensure that the resultant field in Elasticsearch is set to a keyword, to make sure everything is normalised.
filter {
if [message] {
ruby {
code => "event.set('hashtags', event.get('message').scan(/\#[a-z]*/i))"
}
mutate {
lowercase => [ "hashtags" ]
}
}
}
GeoIP problems?
Ok so this is a blog post not a topic, and it was written by me, but geoip “help me!”s are a pretty consistent question on these forums and we wrote this blog post to try to provide a reference point to get through this problem.
You can find the blog post here and it covers things like;
- Starting with the filter from first principles, a single Apache log line
- Processing that via the ingest API
- Alternatively, processing via Logstash
- Mapping requirements in Elasticsearch
- Displaying the results in Kibana
- And finally, some common troubleshooting tips such as dealing with custom index names
Thank You!
As this is the last post in our inaugural Elastic Advent Calendar series, we’d like to thank you for following along! We hope you liked the format, including the ongoing discussions in each topic. If you’d like to see more like this then please let us know via @elastic or in each of the topics we’ve posted so far.