WebHDFS output plugin fails to create new files


#1

We have a simple test configuration of Logstash sending to a MapR instance using WebHDFS (via the Logstash WebHDFS output plugin). Messages are successfully appended to existing files in MapR, but new files cannot be created.

I found this block of code in the Output plugin:

begin   
  # Try to append to already existing file, which will work most of the times.                               
  @client.append(path, data)
  # File does not exist, so create it.
rescue WebHDFS::FileNotFoundError
  # Add snappy header if format is "file".                                                                   
  if @compression == "snappy" and @snappy_format == "file"                                                   
    @client.create(path, get_snappy_header! + data)                                                          
  elsif
    @client.create(path, data)                                                                               
  end

What this does is attempt to append messages, without checking if the file already exists. It is expecting a 404 (file not found) error if the file does not exist. If it receives that error it creates it, ready for the next attempt at appending the message. Using tcpdump, I can see that the MapR instance returns a 500 (server error) rather than a 404 (file not found error). If I change the code above to look for a 500 (server error) rather than a 404 (file not found error), then files are created as needed. This is not an ideal solution though, not least because it would mask a 500 (server error) caused by something else.

Has anybody else come across this problem, and if so, how did you solve it please?


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.