# WebHDFS output plugin fails to create new files

**URL:** <https://discuss.elastic.co/t/webhdfs-output-plugin-fails-to-create-new-files/151494>\
**Category:** Logstash\
**Created:** [October 8, 2018, 4:20pm UTC](https://discuss.elastic.co/t/webhdfs-output-plugin-fails-to-create-new-files/151494 "2018-10-08T16:20:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nich](https://avatars.discourse-cdn.com/v4/letter/n/e79b87/32.png) [@nich](https://discuss.elastic.co/u/nich)\
**Post date:** [October 8, 2018, 4:20pm UTC](https://discuss.elastic.co/t/webhdfs-output-plugin-fails-to-create-new-files/151494/1 "2018-10-08T16:20:08Z")

</div>

We have a simple test configuration of Logstash sending to a MapR instance using WebHDFS (via the Logstash WebHDFS output plugin). Messages are successfully appended to existing files in MapR, but new files cannot be created.

I found this block of code in the Output plugin:

```
begin   
  # Try to append to already existing file, which will work most of the times.                               
  @client.append(path, data)
  # File does not exist, so create it.
rescue WebHDFS::FileNotFoundError
  # Add snappy header if format is "file".                                                                   
  if @compression == "snappy" and @snappy_format == "file"                                                   
    @client.create(path, get_snappy_header! + data)                                                          
  elsif
    @client.create(path, data)                                                                               
  end

```

What this does is attempt to append messages, without checking if the file already exists. It is expecting a 404 (file not found) error if the file does not exist. If it receives that error it creates it, ready for the next attempt at appending the message. Using tcpdump, I can see that the MapR instance returns a 500 (server error) rather than a 404 (file not found error). If I change the code above to look for a 500 (server error) rather than a 404 (file not found error), then files are created as needed. This is not an ideal solution though, not least because it would mask a 500 (server error) caused by something else.

Has anybody else come across this problem, and if so, how did you solve it please?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 5, 2018, 4:20pm UTC](https://discuss.elastic.co/t/webhdfs-output-plugin-fails-to-create-new-files/151494/2 "2018-11-05T16:20:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.