Cannot get Logstash to read in a file on Windows

Elastic Stack Logstash
1

My colleague and I have both installed ELK using Bitnami and are both having issues getting Logstash to be able to successfully read an input file - we're hoping someone here may be able to spot an issue with our setup.

We're using Windows 10 and Logstash 6.5.4.

We're trying to have Logstash read in some simple input files. In my case, it's just a one-line text file containing a string. However, we cannot get Logstash to process the file and write any output to either stdout or a file.

The logstash.conf config file is:

    input
    {
    file{
    path => "C:/Bitnami/elk-6.5.4-0/logstash/InputFiles/test_input.txt"
    sincedb_path => "NUL"
    start_position => "beginning"
    }
    }

    filter
    {
    }

    output
    {
    file{
    path => "C:/Bitnami/elk-6.5.4-0/logstash/testing_logstash_output.txt"
    }
    }

I'm launching Logstash from within PowerShell as an administrator using the following command within the logstash directory:
.\bin\logstash -r -f conf\logstash.conf

That results in the following output to the console:
Sending Logstash logs to C:/Bitnami/elk-6.5.4-0/logstash/logs which is now configured via log4j2.properties
[2019-01-17T22:41:51,426][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-17T22:41:51,454][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-17T22:41:54,540][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-01-17T22:41:55,139][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x245b955a sleep>"}
[2019-01-17T22:41:55,201][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-01-17T22:41:55,211][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-01-17T22:41:55,656][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

From that point, nothing happens. That is, my output file is not generated. When I try to right to stdout, nothing happens there either.

I'll note that I've also run it with the --debug flag. The log indicates no errors. The only odd entry when using the --debug flag is:
[DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu

If anyone has any suggestions as to why we cannot get our test input to be processed, we are all ears. Thanks in advance for your assistance!

1 Like
2

I tried to recreate.

In Google Cloud Platform Compute:
Server: Microsoft Windows Server 2016 Datacenter
LS: 6.5.4

input {
        file {
                path => "C:/Users/guy/elastic/logstash-6.5.4/NOTICE.TXT"
                start_position => "beginning"
                sincedb_path => "NUL"
        }
}

output {
  stdout {
    codec => rubydebug
  }
}

Produces many events and the last one is the last line...

{
          "host" => "euro-testing-win-newest",
          "path" => "C:/Users/guy/elastic/logstash-6.5.4/NOTICE.TXT",
    "@timestamp" => 2019-01-18T14:32:04.295Z,
       "message" => "THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.",
      "@version" => "1"
}

Do you have a permissions issue perhaps?

3

guyboerje: Thanks very much for taking a look! I did three things that, somehow in combination, helped clear my issue. I'll include them here in case other folks are having the issues I had:

  1. I set permissions on the folder to allow reads and writes from all users
  2. I made sure that my simple log file had a new line/carriage return at the end
  3. I started using the DELETE _all command within the Dev section in order to clear out any weirdness that was in there

With each of those, everything looks to now work!

2 Likes
4

Nice to know.

Yes, the file input is indeed line oriented, meaning that it expects a "logical line" to be terminated with a newline. OTOH, there is a "read" mode available which treats the EOF as the end-of-line in case no final newline is present. In "tail" mode, the default, the file input can read a partial line in the expectation that the rest of the line (and the newline) will be written in the near future.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.