WebHDFS output plugin fails to create new files

We have a simple test configuration of Logstash sending to a MapR instance using WebHDFS (via the Logstash WebHDFS output plugin). Messages are successfully appended to existing files in MapR, but new files cannot be created.

I found this block of code in the Output plugin:

begin   
  # Try to append to already existing file, which will work most of the times.                               
  @client.append(path, data)
  # File does not exist, so create it.
rescue WebHDFS::FileNotFoundError
  # Add snappy header if format is "file".                                                                   
  if @compression == "snappy" and @snappy_format == "file"                                                   
    @client.create(path, get_snappy_header! + data)                                                          
  elsif
    @client.create(path, data)                                                                               
  end

What this does is attempt to append messages, without checking if the file already exists. It is expecting a 404 (file not found) error if the file does not exist. If it receives that error it creates it, ready for the next attempt at appending the message. Using tcpdump, I can see that the MapR instance returns a 500 (server error) rather than a 404 (file not found error). If I change the code above to look for a 500 (server error) rather than a 404 (file not found error), then files are created as needed. This is not an ideal solution though, not least because it would mask a 500 (server error) caused by something else.

Has anybody else come across this problem, and if so, how did you solve it please?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.