Failed to start logstash

Elastic Stack Logstash
1 
input {
  kinesis {
    application_name => "stat_events"
    kinesis_stream_name => "events"
    region => "us-east-1"
    codec => gzip_lines {}
    initial_position_in_stream => "LATEST"
  }
}

output {
  stdout { 
    codec  => rubydebug {
    }
  }
}

hi guys:
i am new to logstash, here is the conf file, it just read the aws kinesis and print on the screen. but it fail with some errors, i really don't know what's wrong with it, here is the debug log:

[2021-09-27T13:06:23,428][DEBUG][org.logstash.config.ir.CompiledPipeline][stat_events] Compiled output
 P[output-stdout{"codec"=>#<Java::OrgLogstashConfigIrImperative::PluginStatement:0x7ea9db02>}|[file]/Users/xxx/logstash/pipeline/events_s3.conf:38:3:```
stdout { 
    codec  => rubydebug {
    }
  }
```] 
 into 
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@3fdc2b9c
[2021-09-27T13:06:23,444][INFO ][logstash.javapipeline    ][stat_events] Pipeline Java execution initialization time {"seconds"=>0.62}
[2021-09-27T13:06:23,480][INFO ][logstash.inputs.kinesis  ][stat_events] Registering logstash-input-kinesis
[2021-09-27T13:06:23,587][INFO ][logstash.inputs.kinesis  ][stat_events] Registered logstash-input-kinesis
[2021-09-27T13:06:23,593][INFO ][logstash.javapipeline    ][stat_events] Pipeline started {"pipeline.id"=>"stat_events"}
[2021-09-27T13:06:23,604][DEBUG][logstash.javapipeline    ] Pipeline started successfully {:pipeline_id=>"stat_events", :thread=>"#<Thread:0x548dfcac run>"}
[2021-09-27T13:06:23,672][WARN ][com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker][stat_events][919af55b3f37814336080234cce36d04ee67a90a3c1a3f3d47da4d521e224696] Received configuration for region as us-east-1.
[2021-09-27T13:06:23,678][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:stat_events], :non_running_pipelines=>[]}
[2021-09-27T13:06:23,834][WARN ][com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker][stat_events][919af55b3f37814336080234cce36d04ee67a90a3c1a3f3d47da4d521e224696] Received configuration for region as us-east-1.
[2021-09-27T13:06:23,862][WARN ][com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker][stat_events][919af55b3f37814336080234cce36d04ee67a90a3c1a3f3d47da4d521e224696] Received configuration for region as us-east-1.
[2021-09-27T13:06:23,948][WARN ][com.amazonaws.http.apache.utils.ApacheUtils][stat_events][919af55b3f37814336080234cce36d04ee67a90a3c1a3f3d47da4d521e224696] NoSuchMethodException was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
[2021-09-27T13:06:25,656][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2021-09-27T13:06:27,330][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-09-27T13:06:27,331][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2021-09-27T13:06:30,660][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2021-09-27T13:06:32,334][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-09-27T13:06:32,334][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2021-09-27T13:06:35,667][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2021-09-27T13:06:37,338][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-09-27T13:06:37,339][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2021-09-27T13:06:37,739][DEBUG][logstash.outputs.s3      ] Periodic check for stale files

anyone who can help with it ?

2

Welcome to our community! :smiley:

There's nothing in that log that shows Logstash failing to start, where are you seeing that exactly?

3

hi Mark:

yeah, the problem is the log did not throw error, it just stuck, repeat print the following message, what does it mean ?

[2021-09-27T13:06:35,667][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2021-09-27T13:06:37,338][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-09-27T13:06:37,339][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
4

They are not relevant, and based on the log you have provided Logstash has started up fine

So why do you think that Logstash hasn't started? What's happening, or not, that you would expect?

5

i expect it will print the message in the aws kinesis topic "events", but it just print

[2021-09-27T13:06:35,667][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2021-09-27T13:06:37,338][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-09-27T13:06:37,339][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}[2021-09-27T13:06:35,667][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2021-09-27T13:06:37,338][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-09-27T13:06:37,339][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}[2021-09-27T13:06:35,667][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2021-09-27T13:06:37,338][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-09-27T13:06:37,339][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}[2021-09-27T13:06:35,667][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2021-09-27T13:06:37,338][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-09-27T13:06:37,339][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}

over and over. so i want to know what happens on my logstash, or is that something wrong with the aws kinesis ?

thanks
tony

6

Is there more being added to the stream?

7

hi wark:

i found the root cause, there is other logstash pipeline on aws already consume these message, that's why it stucked.

thanks
tony

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.