Hi, I brought ELK in and scaled and it is fine -- but some of my customers want the entire logfile. Is there a way from ELK to reassemble the entire logfile as it originally came in at a prior point-in-time?
Almost, the _source field will have what was received from LS.
Not sure how to interpret this in terms of anything practical. There is no "_source" field showing in my Kibana interface. Did you mean a query to Elasticsearch or something else? Can you be more specific Mark?
This is what I mean https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html
Well, what I mean is more akin to, but not identical to as far as I can tell, this:
http://air.ghost.io/elasticsearch-extract-logstash-logs/
I seek a way to extract a given logstash-tracked-file from a given-day's logstash deposit @ Elasticsearch.
Users don't always want to do a query through kibana and are pretty advanced and like to see the entire logfile reconstructed at a given point-in-time.
That's all.
Something like:
elasticextract -logstash-date=YYYY-MM-DD --point-in-time=HH:MM:SS --name-of-logstash-collected-file=filename
If you keep the original message in the event and also record metadata around the event that allow you to filter and sort, e.g. host the event comes from, full path and offset within the file, you should be able to search for and retrieve the records in a specific log file within a time range and reconstruct part of the file by sorting them based on the offset. I believe Filebeat is able to supply this type of metadata for each event, but have not tried reassembling files based on it.
There is also a related issue for Kibana available on GitHub.
I realise this is an old post, but it comes up on google, so my workaround for this is:
You can reconstruct the logs using the hostname, full filename and the "offset" that gets added as metadata to the log entry itself. The "offset" is the byte offset in the file that was shipped. If you query specifying an offset of +- a sensible amount,, then sort by offset you'll end up with the log file reconstructed. It's a pain to do though, and file rotation screws it up a bit (filter by timespan as well in that case).
I wrote (hacked) some custom panels for Kibana 3 to do this (GUI for reading logs) , but am hoping elastic might deliver it built into Kibana 5 instead...
The ELK stack badly needs an easy way to do this IMO - ironically given it's at heart a search engine, it's awesome at drawing graphs, but not at actually returning the text that has been indexed!