No modules or inputs enabled - GCP vpcflow

Elastic Stack Beats
1

hey there,
I am really confused about the correct configuration of Filebeat v8.8.1 GCP module.
I need to enable the vpcflow module, so I configured the gcp.yml file in this way:

- module: gcp
  vpcflow:
    enabled: true
    var.project_id: clf-lab-ops
    var.topic: test-vpc-flow
    var.subscription_name: log-vpc-flow

and my filebeat.yml config file is almost empy. I configured only the output section.
I am facing this error:

{"log.level":"error","@timestamp":"2023-10-02T16:15:25.000Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: no modules or inputs enabled and configuration reloading disabled. What files do you want me to watch?","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: no modules or inputs enabled and configuration reloading disabled. What files do you want me to watch?

If I try to specify into filebeat.yml file the following section:

filebeat.modules:
  - module: gcp
    vpcflow:
    enabled: true

I got this error:

{"log.level":"warn","@timestamp":"2023-10-02T16:17:59.766Z","log.logger":"gcp.pubsub","log.origin":{"file.name":"gcppubsub/input.go","file.line":160},"message":"Restarting failed Pub/Sub input worker.","service.name":"filebeat","pubsub_project":"SET_PROJECT_NAME","pubsub_topic":"stackdriver-vpcflow","pubsub_subscription":{"Name":"filebeat-gcp-vpcflow","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":true},"error":{"message":"failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = InvalidArgument desc = You have passed an invalid argument to the service (argument=SET_PROJECT_NAME)."},"ecs.version":"1.6.0"}

So I am really confused. What should I do? Where should I specify which module I would enable?

2

So you ran

filebeat modules enable gcp

correct?

What installation method did you use?

And super silly question you actually saved that gcp.yml

are there any other modules enabled or .yml in the modules.d directory

Turns out they all get concatenated together so I have seen when there was another .yml that was cancelling out the other.

3

Nope I don't use that command.
I am using Docker.
Yes I saved that file and no other are into the modules.d folder.

4

@rschirin

There is something basic going on...

Like your docker is not mounting that modules.d directory or something.

5

I thought that so I checked it inside. Modules.d folder contains only my file. So really, I cannot understand what is happening

6

Is it readable? Something simple that message it that is can find not enabled configs ... you can see when you put it in the filebeat.yml it sees it... you can always put it in there...

AHHH wait ... show me your full filebeat.yml ...

Do you have the path to the modules.d defined in that?

7 
logging.level: debug

# ================================== Logging ===============================
# If enabled, Filebeat periodically logs its internal metrics that have changed
# in the last period. For each metric that changed, the delta from the value at
# the beginning of the period is logged. Also, the total values for
# all non-zero internal metrics are logged on shutdown. The default is true.
logging.metrics.enabled: true

# The period after which to log the internal metrics. The default is 30s.
logging.metrics.period: 300s


#========================= Filebeat Processor ================================#
#
#
processors:
  - add_id: ~
#
#========================== Filebeat Elasticsearch output ===============================#
output.elasticsearch:
  hosts: ["es-rally:9200"]
  protocol: http
  client_authentication: none

that's all :slight_smile:
so, probably, you find the issue: I haven't the path to the modules.d folder. is it mandatory? I thought there was a default value usable.

8

Yup! otherwise they are not added.. :slight_smile:
So there you have it...

Usually looks like

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.