Finally a command to simulate tail -f

Announcements Community Ecosystem
1

Hi Folks,

over the last year I have been really wanting a command that simulates "tail -f " on an index. Very often my Middleware and developers just want to see the logs scrolling. The web gui's that I have found just don't seem to have anything close enough that they would adapt. (Even Splunk does not do it well)

So after searching the web I decided to create my own as the 3 or 4 others are old or just not what I am looking for

I have made a project esTail https://github.com/ElasticSearchCLITools/esTail

node:
	[--url=localhost:9200]
	[--search=<filename> default: default.search
	[--regex='([d.]+)' default: none
	[--regexflags='gm'   default: gm
	[--allfields         default: false 
	[--raw         	    default: false 
	[--fetchsize='20'  default: 100 
	[-i|--refreshInterval='1000'  default: 1000
			How often a new search is issued
	[--context='{ 'custom':'json'}'  default:{"index":"_all","from":"now-10m","fetchsize":100}
			Context is what varables pass to the search template for json markup
			context=<key>=<val> is a way to set any varable inside the context array. Make sure this is used after --contextfile or --context=<customejson>
	[--index=<index>|--context=index=<index>     default: _all
	[--from=<datestamp>|--context=from='now-5m'  default: now-10m
			from can be of any valid Elasticsearch timevalue or Caclulation

Simple a simple tail
node ./esTail.js --index=logstash*

Connected to Elasticsearch cluster.
2015-10-11T16:05:50.073Z: logstash-2015.10.11:Sun Oct 11 12:05:50 EDT 2015
2015-10-11T16:07:50.081Z: logstash-2015.10.11:Sun Oct 11 12:07:50 EDT 2015
2015-10-11T16:08:50.089Z: logstash-2015.10.11:Sun Oct 11 12:08:50 EDT 2015

with regex
node ./esTail.js --index=logstash* --regex='([\d.]+)

Connected to Elasticsearch cluster.
2015-10-11T16:05:50.073Z: std-2015.10.11:Sun Oct 11 12:05:50 EDT 2015
     regex: ["11","12","05","50","2015"]
2015-10-11T16:07:50.081Z: std-2015.10.11:Sun Oct 11 12:07:50 EDT 2015
    regex: ["11","12","07","50","2015"]
2015-10-11T16:08:50.089Z: std-2015.10.11:Sun Oct 11 12:08:50 EDT 2015

Or raw search output
node ./esTail.js --index=std* --regex='([\d.]+)' --raw

Connected to Elasticsearch cluster.
{
  "_index": "std-2015.10.11",
  "_type": "std",
  "_id": "AVBXusggsIKvqavVTCUW",
  "_score": 0,
  "_source": {
    "message": "Sun Oct 11 12:27:50 EDT 2015",
    "@version": "1",
    "@timestamp": "2015-10-11T16:27:50.169Z",
    "type": "std",
    "host": "coperdragon"
  }
}
	regex: ["11","12","27","50","2015"]
{
  "_index": "std-2015.10.11",
  "_type": "std",
  "_id": "AVBXu7KFsIKvqavVTCUX",
  "_score": 0,
  "_source": {
    "message": "Sun Oct 11 12:28:50 EDT 2015",
    "@version": "1",
    "@timestamp": "2015-10-11T16:28:50.173Z",
    "type": "std",
    "host": "coperdragon"
  }
}
	regex: ["11","12","28","50","2015"]
{
  "_index": "std-2015.10.11",
  "_type": "std",
  "_id": "AVBXwTDdsIKvqavVTCUd",
  "_score": 0,
  "_source": {
    "message": "Sun Oct 11 12:34:50 EDT 2015",
    "@version": "1",
    "@timestamp": "2015-10-11T16:34:50.201Z",
    "type": "std",
    "host": "coperdragon"
  }
}
	regex: ["11","12","34","50","2015"]

Hope you all enjoy, I have tested it over night and does not seem to have any issues running but I have not tested it against a busy index. Tell me if you like it, feel free to fork and push me updates.

4 Likes
2

Ok,

I came up with a group of idea's https://github.com/ElasticSearchCLITools/

going to create a couple different commands to help system admin and other people who like CLI tools

esTail = tail an index
esCatField = drop a raw field to a file. Like get a list of all the hosts from the host.raw field
esGrep = In addition to the the standard search , I will wrap a RegEx to help process and break out the data (might make this a subset of esCatField)
mkindex = create and index (maybe)

and create a few others even if they are simple commands or easy to issue a wget. Sometimes a command is what you need for non-elasticsearch users

Please feel free to throw suggestions my way, fork the projects or report issues.

3

Ok esTail seems to be working great against big indexes and high volume

esCatField = works great for example you want a list of hosts which are logging , or a list of files in an index

esls = Working great to pull statistics of the indexes

This are nice simple NodeJS apps so they also might be a good starting point for new people to learn the API

1 Like