Logstash file input to file output is not working. No errors seen in the logs

Elastic Stack Logstash
1

Hello,
I am new to logstash. I am using 8.2.3. I dont have Elasticsearch running. I wanted to see if logstash is able to take log file as input and convert into json and store it in a file . This is my code below .

input {
    file {
        path => "/home/rraghuna/test/debug1.log"
        start_position => "beginning"
        ignore_older => 0  
    }
}
output {
    stdout {}
    file {
        path => "/home/rraghuna/test/output.txt"
    }
}

I start the logstash using

~/logstash-8.2.3/bin$ ./logstash -f logstash.conf

It gets struck here... No errors...

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /home/rraghuna/logstash-8.2.3/logs which is now configured via log4j2.properties
[2022-06-15T17:49:30,261][INFO ][logstash.runner          ] Log4j configuration path used is: /home/rraghuna/logstash-8.2.3/config/log4j2.properties
[2022-06-15T17:49:30,270][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.2.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-06-15T17:49:30,271][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-06-15T17:49:30,455][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-06-15T17:49:31,203][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-06-15T17:49:31,568][INFO ][org.reflections.Reflections] Reflections took 61 ms to scan 1 urls, producing 120 keys and 395 values
[2022-06-15T17:49:32,203][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-06-15T17:49:32,286][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["/home/rraghuna/logstash-8.2.3/logstash.conf"], :thread=>"#<Thread:0x331a6abd run>"}
[2022-06-15T17:49:32,813][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.52}
[2022-06-15T17:49:32,862][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/home/rraghuna/logstash-8.2.3/data/plugins/inputs/file/.sincedb_5b64a3d1731c6c35ae0fd4e425d30883", :path=>["/home/rraghuna/test"]}
[2022-06-15T17:49:32,878][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-06-15T17:49:32,911][INFO ][filewatch.observingtail  ][main][fa44b673899ee8f18a2ea8d5a0406fc43981e50d5bf9cb6dfd41cedd6d520779] START, creating Discoverer, Watch with file and sincedb collections
[2022-06-15T17:49:32,921][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
^C[2022-06-15T17:55:18,094][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2022-06-15T17:55:18,118][INFO ][filewatch.observingtail  ] QUIT - closing all files and shutting down.
[2022-06-15T17:55:18,253][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-06-15T17:55:19,142][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}

I dont see output.txt. That is my desired output.

2

For filebeat, setting ignore_older to zero turns off age based filtering. For a file input, it causes the filter to ignore any files more than zero seconds old, which means it ignores any files that are not created at the moment that the scheduled poll for new files runs.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.