We have a simple test configuration of Logstash sending to a MapR instance using WebHDFS (via the Logstash WebHDFS output plugin). Messages are successfully appended to existing files in MapR, but new files cannot be created.
I found this block of code in the Output plugin:
begin # Try to append to already existing file, which will work most of the times. @client.append(path, data) # File does not exist, so create it. rescue WebHDFS::FileNotFoundError # Add snappy header if format is "file". if @compression == "snappy" and @snappy_format == "file" @client.create(path, get_snappy_header! + data) elsif @client.create(path, data) end
What this does is attempt to append messages, without checking if the file already exists. It is expecting a 404 (file not found) error if the file does not exist. If it receives that error it creates it, ready for the next attempt at appending the message. Using tcpdump, I can see that the MapR instance returns a 500 (server error) rather than a 404 (file not found error). If I change the code above to look for a 500 (server error) rather than a 404 (file not found error), then files are created as needed. This is not an ideal solution though, not least because it would mask a 500 (server error) caused by something else.
Has anybody else come across this problem, and if so, how did you solve it please?