Trying to use logstash-input-google_pubsub

[dinaWork (dina)]

here is conf/logstash.conf

input {
    google_pubsub {
        project_id => "myProjectId"
        topic => "logstash-input-dev"
        subscription => "logstash-sub"
        json_key_file => "/usr/share/logstash/key-file.json"
    }
}

output {	
  stdout {}
}

here is my logstash.yaml

path.config: /usr/share/logstash/pipeline

config.reload.automatic: true

config.debug: false

running:

docker run --rm -it -v  /key-file.json:/usr/share/logstash/key-file.json  -v ~/conf/:/usr/share/logstash/pipeline/ -v ~/logstash.yml:/usr/share/logstash/config/logstash.yml sphereio/logstash-input-google_pubsub

here is my output

Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

10:38:26.430 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
10:38:26.496 [LogStash::Runner] INFO  logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"dc6bcc6a-dd8c-49c9-a15c-cb4a745713ff", :path=>"/var/lib/logstash/uuid"}
The stdin plugin is now waiting for input:
10:38:27.366 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
10:38:27.457 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
10:38:27.760 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
{
    "@timestamp" => 2017-03-29T10:38:28.359Z,
      "@version" => "1",
          "host" => "5b087dbebdf5",
       "message" => "",
          "type" => "stdin"
}

ok now publishing to pubsub

gcloud beta pubsub topics publish projects/myProjectId/topics/logstash-input-dev "hello-all"

nothing is happening, cant see the logs.

trying to pull message myself:
gcloud beta pubsub subscriptions pull --auto-ack projects/myProjectId/subscriptions/logstash-sub

I get it (which means the plugin hasnt pulled the message)

┌──────┬─────────────────┬────────────┐
│ DATA │ MESSAGE_ID │ ATTRIBUTES │
├──────┼─────────────────┼────────────┤
│ hello-all │ 114103460666990 │ │
└──────┴─────────────────┴────────────┘

I connected to the docker running- and all seems ok

docker exec -ti 5b087dbebdf5 bash

cat /usr/share/logstash/config/logstash.yml
I see my yml is there

cat /usr/share/logstash/pipeline/logstash.conf
the above conf is there

cat /usr/share/logstash/key-file.json
json file is there

how can I track whats wrong here?

[ZonOrti (Sergey Melnik)]

Hi,

When container starts and connects to Google PubSub successfully you should see something like:

"07:31:17.791 [[main]<google_pubsub] INFO  logstash.inputs.googlepubsub - Pulling messages from sub 'projects/projectID/subscriptions/logstash-sub'

This line is not here, and in general, it looks like your config is ignored.
I've checked my configuration and it looks like https://hub.docker.com/_/logstash/ image reads config from /etc/logstash/logstash.yml and not from /usr/share/logstash/config/logstash.yml
Try to change mount path.
Hope it helps

[dinaWork (dina)]

Thanks! I've changed the mount path now it's working fine!!