I am getting to get started with logstash and simply copy a csv file to another file using the following conf:
input {
file{
path => "/Users/test/Desktop/project/test.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["_id", "status", "test", ...]
}
}
output {
file{
path => "/Users/test/Desktop/project/out"
}
stdout {}
}
This is what I am seeing in the terminal:
Using bundled JDK: /Users/test/Desktop/project/elastic_stack/logstash-8.11.1/jdk.app/Contents/Home
Sending Logstash logs to /Users/test/Desktop/project/elastic_stack/logstash-8.11.1/logs which is now configured via log4j2.properties
[2023-12-01T16:42:27,123][INFO ][logstash.runner ] Log4j configuration path used is: /Users/test/Desktop/project/elastic_stack/logstash-8.11.1/config/log4j2.properties
[2023-12-01T16:42:27,128][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2023-12-01T16:42:27,128][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.11.1", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.9+9 on 17.0.9+9 +indy +jit [arm64-darwin]"}
[2023-12-01T16:42:27,129][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-12-01T16:42:27,148][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-12-01T16:42:27,659][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-12-01T16:42:27,775][INFO ][org.reflections.Reflections] Reflections took 50 ms to scan 1 urls, producing 132 keys and 464 values
[2023-12-01T16:42:27,976][INFO ][logstash.codecs.jsonlines] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-12-01T16:42:28,100][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2023-12-01T16:42:28,105][INFO ][logstash.filters.csv ][main] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-12-01T16:42:28,110][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1250, "pipeline.sources"=>["/Users/test/Desktop/project/elastic_stack/logstash-8.11.1/try-pipeline.conf"], :thread=>"#<Thread:0x69cbf2df /Users/test/Desktop/project/elastic_stack/logstash-8.11.1/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-12-01T16:42:28,457][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.35}
[2023-12-01T16:42:28,465][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/Users/test/Desktop/project/elastic_stack/logstash-8.11.1/data/plugins/inputs/file/.sincedb_b0fae6fd9aa08ab66e65567f33cbdd2b", :path=>["/Users/test/Desktop/project/test"]}
[2023-12-01T16:42:28,467][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-12-01T16:42:28,469][INFO ][filewatch.observingtail ][main][3dd4f02e694ce39e820c6ee4e5dd66ed6c143bb37649afe1faf0c53f13c461a8] START, creating Discoverer, Watch with file and sincedb collections
[2023-12-01T16:42:28,482][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
And it is stuck there, when I run with --debug I get a loop of this:
[2023-12-01T16:45:59,397][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Young Generation"}
[2023-12-01T16:45:59,398][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Old Generation"}
[2023-12-01T16:46:00,308][DEBUG][logstash.outputs.file ][main] Starting flush cycle
[2023-12-01T16:46:00,611][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-12-01T16:46:00,625][DEBUG][logstash.outputs.file ][main][d073bbdd72cc1402cd8974ecd5e0b09daadcc20742b6385b991a97edc5afa767] Starting stale files cleanup cycle {:files=>{}}
[2023-12-01T16:46:00,626][DEBUG][logstash.outputs.file ][main][d073bbdd72cc1402cd8974ecd5e0b09daadcc20742b6385b991a97edc5afa767] 0 stale files found {:inactive_files=>{}}
[2023-12-01T16:46:02,310][DEBUG][logstash.outputs.file ][main] Starting flush cycle
[2023-12-01T16:46:04,315][DEBUG][logstash.outputs.file ][main] Starting flush cycle
[2023-12-01T16:46:04,394][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2023-12-01T16:46:04,402][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Young Generation"}
[2023-12-01T16:46:04,402][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Old Generation"}
[2023-12-01T16:46:05,610][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-12-01T16:46:06,320][DEBUG][logstash.outputs.file ][main] Starting flush cycle
[2023-12-01T16:46:06,674][DEBUG][filewatch.sincedbcollection][main][3dd4f02e694ce39e820c6ee4e5dd66ed6c143bb37649afe1faf0c53f13c461a8] writing sincedb (delta since last write = 15)
I am not sure if this is important or not but I am running logstash on a macbook with M1 so I do not have the cgroup files or directories.