I'm trying to create an index and loading one log file to Elasticsearch using logstash using below config:
input {
file {
path => ["/mnt/c/databalanceInfo_0.log"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200"]
index => "balance_no_grok"
}
stdout { }
}
but I get pipeline is running and no index is getting created:
sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -r
Using LS_JAVA_HOME defined java: /usr/lib/jvm/java-17-openjdk-amd64.
WARNING: Using LS_JAVA_HOME while Logstash distribution comes with a bundled JDK.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-05-21 12:42:10.008 [main] runner - The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[INFO ] 2023-05-21 12:42:10.010 [main] runner - Starting Logstash {"logstash.version"=>"7.17.10", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 17.0.6+10-Ubuntu-0ubuntu120.04.1 on 17.0.6+10-Ubuntu-0ubuntu120.04.1 +indy +jit [linux-x86_64]"}
[INFO ] 2023-05-21 12:42:10.011 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED]
[WARN ] 2023-05-21 12:42:10.126 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-05-21 12:42:10.633 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2023-05-21 12:42:10.977 [Converge PipelineAction::Create<main>] Reflections - Reflections took 41 ms to scan 1 urls, producing 119 keys and 419 values
[WARN ] 2023-05-21 12:42:11.191 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-05-21 12:42:11.222 [Converge PipelineAction::Create<main>] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-05-21 12:42:11.255 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-05-21 12:42:11.279 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-05-21 12:42:11.376 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ] 2023-05-21 12:42:11.516 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2023-05-21 12:42:11.686 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2023-05-21 12:42:11.695 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (8.7.1) {:es_version=>8}
[WARN ] 2023-05-21 12:42:11.696 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[INFO ] 2023-05-21 12:42:11.722 [Ruby-0-Thread-10: :1] elasticsearch - Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[INFO ] 2023-05-21 12:42:11.724 [[main]-pipeline-manager] elasticsearch - Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[INFO ] 2023-05-21 12:42:11.749 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[INFO ] 2023-05-21 12:42:11.763 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>20, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x5f91ab90 run>"}
[INFO ] 2023-05-21 12:42:12.225 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.46}
[INFO ] 2023-05-21 12:42:12.258 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2023-05-21 12:42:12.285 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2023-05-21 12:42:12.296 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
)