Store directory tree of files in ES


(Tim Ward) #1

Suppose I have a directory tree of a modest number of small text files (where "modest" is currently hundreds, won't ever be more than tens of thousands) and "small" is hundreds of bytes, won't ever be more than thousands. These files (both the set of files that exist and their contents) will change occasionally rather than frequently.

We would like to store these files in a database keyed by path name rather than as files in a filing system. We happen to have ES in use in the project so that's the most convenient database to use.

The main operation is to find a file by its path, which the application will have in "some/path/components.ext" form. Other operations needed are

  • Copy an existing directory tree into ES
  • Dump out the data from ES into a directory tree
  • Allow a use to edit individual files

So I can use a single index for this, with the _id being the path.

Getting a record is

GET indexname/_doc/some%2Fpath%2Fcomponents.ext

Allowing a user to edit the occasional individual record is telling them how to use Kibana (the users are already used to this).

Loading from and dumping to a directory tree is a couple of tedious but straightforward Python scripts.

OK? Have I missed anything? Is that the right URL encoding? Has anyone done this? Feel like giving away your Python scripts? :rofl:


(David Pilato) #2

Would that help?

http://david.pilato.fr/blog/2015/12/10/building-a-directory-map-with-elk/


(Tim Ward) #3

What fun! - too clever for what I need though (and turns out the Python scripts are more trivial than I thought, and that the URL encoding does seem to work exactly like that)


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.