I have an application that maintains many xml files under a folder and set of sub-folders. The xml files are added and removed from these folders. Is there a way to use logstash to send these files to Elastic Search allowing users to search through them. We would also need to remove them from elastic search when they get removed from the folders.
Sure, that's possible with the file input. Just use a filename pattern that selects all possible XML files. You'll have to use a multiline codec to join the lines of each XML file into a single events. This is a bit clunky.
However, to remove the documents from ES once the XML files are gone is something you'll have to do outside of Logstash (or with a custom plugin).
What if a file changes. Will the record in elastic search get replaced, or will I get two?
I'm trying to replace an antiquated installation of Microsoft Indexing Service. We have it set up to just monitor a folder structure and it just "works" and indexes what's there. It seems to handle add, updates and deletes all on it's own.
Is there anyway to easily get and ELK stack to do this?
What if a file changes. Will the record in Elasticsearch get replaced, or will I get two?
(...or will nothing happen at all?) It depends on how the file is updated. Is it updated in place, i.e. is the same inode (or equivalent) reused, or is it replaced with a new file that happens to have the same name? The primary purpose of the file input is to monitor log files. Once you deviate from that use case things will get bumpy.
I'm trying to replace an antiquated installation of Microsoft Indexing Service. We have it set up to just monitor a folder structure and it just "works" and indexes what's there. It seems to handle add, updates and deletes all on it's own.
As it should, being an indexing service and all. Logstash doesn't do this well out of the box (but writing a plugin that does it wouldn't be too hard). You should probably look into GitHub - dadoonet/fscrawler: Elasticsearch File System Crawler (FS Crawler) and similar helpers.