Logstash stuck at stage pipelines running

Elastic Stack Logstash
1

I am trying to ingest a csv file to include data on the elasticsearch server on the existing index. The csv file contains whitespaces in header column names.
Here is an example "MOBILE NO" should be renamed to "MOBILE_NO". Can I achieve this in logstash import?
This is my pipeline configuration(on windows)

input{
file{
    path => "E:\abc\xyz\johny.*csv"
    start_position => "beginning"
    sincedb_path => "NUL"
}
}
filter{
csv{
    separator => ","
    columns => ["STATUS_TYPE","GENDER","AGE","MOBILE NO","HOUSE NO"]
}    
ruby  {
    code => "
            hash = event.to_hash
            hash.each{|k,v| event.set(k.gsub(/ /,'_'),v) && event.remove(k) if k.include?(' ')  }
        "
}
}
output{
elasticsearch{
    hosts => "http://localhost:9200"
    index => "heaven_search"
}
stdout{
    codec => rubydebug
}
}

Whenever I am running the above config file, I got logs and at the end i got this message [2022-07-19T00:40:07,063][INFO ][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} but nothing changed after the message. And no data indexing into elasticsearch.

2

Do not use backslash in the path option of a file input, it is interpreted as an escape, so logstash is waiting for lines to be written to a file called E:abcxyzjohny.*csv. Change it to path => "E:/abc/xyz/johny.*csv"

3

@Badger I removed the backslash in the path option of file input and ran again. Still stuck at Pipelines running
Here are the complete logs, you can see at the end of the log is "Pipeline running" after this nothing changing.

F:\elastic-stack\logstash\bin>logstash -f config/uploads_vo_search.conf
"Using bundled JDK: F:\elastic-stack\logstash\jdk\bin\java.exe"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to F:/elastic-stack/logstash/logs which is now configured via log4j2.properties
[2022-07-19T08:53:36,020][INFO ][logstash.runner          ] Log4j configuration path used is: F:\elastic-stack\logstash\config\log4j2.properties
[2022-07-19T08:53:36,028][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-07-19T08:53:36,030][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.3.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [mswin32-x86_64]"}
[2022-07-19T08:53:36,032][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-07-19T08:53:36,172][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-07-19T08:53:51,829][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-07-19T08:53:53,344][INFO ][org.reflections.Reflections] Reflections took 397 ms to scan 1 urls, producing 124 keys and 408 values
[2022-07-19T08:53:57,048][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-07-19T08:53:57,099][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2022-07-19T08:53:58,139][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2022-07-19T08:53:59,631][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2022-07-19T08:53:59,649][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.3.2) {:es_version=>8}
[2022-07-19T08:53:59,652][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-07-19T08:53:59,703][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-07-19T08:53:59,705][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-07-19T08:53:59,708][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-07-19T08:53:59,726][INFO ][logstash.filters.csv     ][main] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2022-07-19T08:53:59,828][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-07-19T08:53:59,910][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["F:/elastic-stack/logstash/config/uploads_vo_search.conf"], :thread=>"#<Thread:0x2d004e33 run>"}
[2022-07-19T08:54:03,402][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>3.49}
[2022-07-19T08:54:03,469][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-07-19T08:54:03,558][INFO ][filewatch.observingtail  ][main][7944c404515ede1b585732c36def63393e3b69570faf5bc4b24190a42b9e4dbf] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-19T08:54:03,601][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
4

Did you change the path in the file input to path => "E:/abc/xyz/johny.*csv" ?

Are the files that match this glob single line files?

Can you try to run logstash again enabling the debug log level? Just use the --debug parameter in the command line.

5

@leandrojmp I changed the path in the file input to path => "E:/abc/xyz/johny/myvochers.csv" instead of the match creteria. This time it showing different errors like Marking url as dead. Last error: ... Attempted to send a bulk request but Elasticsearch appears to be unreachable or down

F:\elastic-stack\logstash\bin>logstash -f config/upload_vo_search.conf
"Using bundled JDK: F:\elastic-stack\logstash\jdk\bin\java.exe"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to F:/elastic-stack/logstash/logs which is now configured via log4j2.properties
---
[2022-07-19T09:32:08,123][WARN ][logstash.outputs.elasticsearch][main][a3ea8faafe94b340ef08fc998c52f5acacb8602ca1b1d4407331e739bb17abab]Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://localhost:9200/_bulk][Manticore::SocketTimeout] Read timed out {:url=>http://localhost:9200/, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200/_bulk][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.