We have a simple test configuration of Logstash sending to a MapR instance using WebHDFS (via the Logstash WebHDFS output plugin). Messages are successfully appended to existing files in MapR, but new files cannot be created.
I found this block of code in the Output plugin:
begin
# Try to append to already existing file, which will work most of the times.
@client.append(path, data)
# File does not exist, so create it.
rescue WebHDFS::FileNotFoundError
# Add snappy header if format is "file".
if @compression == "snappy" and @snappy_format == "file"
@client.create(path, get_snappy_header! + data)
elsif
@client.create(path, data)
end
What this does is attempt to append messages, without checking if the file already exists. It is expecting a 404 (file not found) error if the file does not exist. If it receives that error it creates it, ready for the next attempt at appending the message. Using tcpdump, I can see that the MapR instance returns a 500 (server error) rather than a 404 (file not found error). If I change the code above to look for a 500 (server error) rather than a 404 (file not found error), then files are created as needed. This is not an ideal solution though, not least because it would mask a 500 (server error) caused by something else.
Has anybody else come across this problem, and if so, how did you solve it please?