Deploying a new filebeat module

Elastic Stack Beats
1

Sorry for peppering the boards here with countless questions. You guys have been extremely helpful and I successfully created my module and it passed all tests. Thanks so much!

Now, I'm trying to deploy my module to an existing Elastic stack installation. I haven't found any documentation (on your site or though Google) as to how to do that. So, here's what I've tried:

Before I started, I made sure elasticsearch and kibana were running.

  1. I copied my local .../filebeat/module/my_module directory to the remote server at /usr/share/filebeat/module. After filebeat complained, I changed ownership to root.

  2. I copied my local .../filebeat/modules.d/my_module.yml.disabled file to the server at /etc/filebeat/modules.d. After filebeat complained, I changed ownership to root here too.

  3. I ran sudo filebeat modules enable my_module then sudo service filebeat restart.

  4. In the logs, I saw this error: Exiting: Can only start an input when all related states are finished. I did some research and ended up just deleting the registry: sudo rm -v /var/lib/filebeat/registry/filebeat/data.json

  5. After restarting filebeat, everything seemed to work -- harvesters were started for the logs my module parses.

  6. I looked at Kibana to see if any of my fields were showing up. They weren't. I unchecked the "Hide missing fields" box. I picked the last 90 days as the time frame. Most of the logs I'm interested in are from a month ago. I clicked the "Refresh" button several times over about 30 minutes.

  7. I checked /etc/filebeat/fields.yml on the server and none of the fields from my module were there. I had forgotten about that. But I also kinda thought enabling the module would have updated that file. I copied/pasted from the auto-generated fields.yml in my dev environment. I cleared the registry and restarted filebeat.

  8. Again, harvesters for the logs I care about started up and closed after log inactivity the way they should (I assume). Still, no joy on Kibana.

  9. I'm sure I'm seeing the remote Kibana in my browser and not the local Kibana. The remote Kibana doesn't have the grok debugger.

Any ideas as to what might be going wrong? Again, many thanks!

2

Hmmm did you check Eleasticsearch to make sure there are events sent there by Filebeat? Maybe you can try running Filebeat in debug mode by using ./filebeat -e -d "*". This will be very noisy but it will show you the events that Filebeat generates.

3

Thanks! I ran in debug mode and collected about 2.5G of text. I'm not sure what I'm looking for. Here are a few things I'm seeing:

I'm seeing a number of publish events that look promising:

2020-02-05T17:32:56.956Z	DEBUG	[processors]	processing/processors.go:186	Publish event: {
  "@timestamp": "2020-02-05T17:32:56.956Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.1",
    "pipeline": "filebeat-7.5.1-my_module-My_fileset-pipeline"
  },
  "host": {
    "containerized": false,
    "hostname": "my_host",
    "name": "my_host",
    "architecture": "x86_64",
    "os": {
      "name": "Ubuntu",
      "kernel": "3.13.0-161-generic",
      "codename": "trusty",
      "platform": "ubuntu",
      "version": "14.04.6 LTS, Trusty Tahr",
      "family": "debian"
    },
    "id": "dd246e57f8...9d75de7f970"
  },
  "cloud": {
    "account": {
      "id": "1234567890"
    },
    "image": {
      "id": "ami-0..."
    },
    "instance": {
      "id": "i-0123456..."
    },
    "provider": "aws",
    "machine": {
      "type": "r4.4xlarge"
    },
    "region": "us-east-1",
    "availability_zone": "us-east-1b"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/log/module/instance/2020_01_10.stderrout.log.234"
    },
    "flags": [
      "multiline"
    ]
  },
  "input": {
    "type": "log"
  },
  "event": {
    "module": "my_module",
    "dataset": "my_module.my_fileset"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "message": "Jan 10, 2020 8:16:26 PM com.xxxxxx.servers.Server <clinit>\nINFO: Running vpc FEATURES, in environment LOCAL, region is USEAST1_DEV, host is https://xxxxxxxxxx.test.xxxx.com",
  "service": {
    "type": "my_module"
  },
  "fileset": {
    "name": "my_fileset"
  },
  "agent": {
    "version": "7.5.1",
    "type": "filebeat",
    "ephemeral_id": "5d...7987c",
    "hostname": "my_host",
    "id": "..."
  }
}

I'm seeing some of these for empty logs with some lines that are excluded:

2020-02-05T17:32:56.974Z	DEBUG	[publisher]	pipeline/client.go:220	Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id: Finished:false Fileinfo:0xc... Source:/log/module/instance/2019_12_04.gc.203 Offset:0 Timestamp:2020-02-05 17:32:56.951794071 +0000 UTC m=+0.096943364 TTL:-1ns Type:log Meta:map[] FileStateOS:1447942-51713} TimeSeries:false}

Any clues as to what I should be looking for? Again, the issue is that I can't find any of the parsed fields in Kibana.

Thanks!

4

I figured it out. My /etc/filebeat/filebeat.yml had the Logstash output configured and not the Elasticsearch output. For now, we're bypassing Logstash and I had left it in an inoperative, experimental state, so this was wrong.

Thanks for all your help!

5

That was only part of the problem. Only one of my filesets is showing up in Kibana at all. I'm pouring over the huge log file(s) to see what might have happened. I'm not sure what to be looking for.

For example, looking for any mention of any fields in my fileset, I try this: egrep -o '\.fileset\.\w+' filebeat_2020-02-06@21:18:30.log | sort | uniq. That confirms that the only mention of any fileset.* is fileset.log, referring to the log file being harvested.

Is there any way to get information about what's happening? I'd really like to see the JSON that my filebeat module is sending to elasticsearch, or error messages in the attempt to do so.

Thanks.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.