Logstash show error "undefined method `length' for nil:NilClass"

Elastic Stack Logstash
1

Hi there,

I have an issue with the title. When I send the JSON log to Filebeat and send it ti my logstash, sometimes it passes the filter, but sometimes it fails. Would you please help me check my configuration to see where adjustments are needed?

Thanks!

Here is the filebeat.yml configuration:

name: 172.16.47.200
output:
  logstash:
    enabled: true
    hosts:
      - 172.16.47.200:5044
    index: "172.16.47.200"

filebeat.inputs:
  - type: log
    tags: iron
    paths:
      - /root/Iron/*
    codec: json
    processors:
    - decode_json_fields:
        fields: ["creationInfo", "package", "relationships"]
        target: "json"

Here is the logstash.conf :

input {
  beats {
    port => 5044
    codec => "json_lines"
  }
}

filter {
    json {
      source => "message"
      remove_field => [ "message" ]
    }
    mutate {
      remove_field => ["package.copyrightText", "package.externalRefs", "packages.licenseDeclared"]
    }
    ruby {
      code => '
        packages = event.get("[packages]")
        if packages
          event.set("total_package_count", packages.length)
          packages.each_with_index do |package, index|
            event.set("package_name-#{index}", package["name"])
            event.set("package_version-#{index}", package["versionInfo"])
            event.set("package_license-#{index}", package["licenseConcluded"])
          end
        end
      '
    }
}

output {
  stdout {
    codec => "rubydebug"
  }
  elasticsearch {
    hosts => "172.16.47.200:9200"
    index => "test-4"
    user => "elastic"
    password => "123123"
  }
}

Best Regards,

Wesley

2

Hello,

Please share the log line you are receiving, not just part of it.

3

Here is the example for the JSON LOG, and I want to show some info. in the package.

{
 "spdxVersion": "SPDX-2.3",
 "dataLicense": "CC0-1.0",
 "SPDXID": "SPDXRef-DOCUMENT",
 "name": "iron-appmgr-develop.tar.gz",
 "documentNamespace": "https://anchore.com/syft/file/iron-appmgr-develop.tar.gz-ec61ad8f-7e7a-4b4e-98a7-b7afe84213da",
 "creationInfo": {
  "licenseListVersion": "3.20",
  "creators": [
   "Organization: Anchore, Inc",
   "Tool: syft-0.75.0"
  ],
  "created": "2023-03-16T06:43:15Z"
 },
 "packages": [
  {
   "name": "libarchive",
   "SPDXID": "SPDXRef-Package-rpm-libarchive-2f6f0114cfc5e5a8",
   "versionInfo": "0:3.6.1-1.el7",
   "downloadLocation": "NOASSERTION",
   "sourceInfo": "acquired package info from RPM DB: iron-appmgr-develop/rpms.libarchive.7/libarchive-3.6.1-1.el7.x86_64.rpm",
   "licenseConcluded": "LicenseRef-BSD",
   "licenseDeclared": "LicenseRef-BSD",
   "copyrightText": "NOASSERTION",
   "externalRefs": [
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive:libarchive:0\\:3.6.1-1.el7:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "PACKAGE-MANAGER",
     "referenceType": "purl",
     "referenceLocator": "pkg:rpm/libarchive@3.6.1-1.el7?arch=x86_64&epoch=0&upstream=libarchive-3.6.1-1.el7.src.rpm"
    }
   ]
  },
  {
   "name": "libarchive",
   "SPDXID": "SPDXRef-Package-rpm-libarchive-505ec624e21867c8",
   "versionInfo": "0:3.6.1-1.el8",
   "downloadLocation": "NOASSERTION",
   "sourceInfo": "acquired package info from RPM DB: iron-appmgr-develop/rpms.libarchive.8/libarchive-3.6.1-1.el8.x86_64.rpm",
   "licenseConcluded": "LicenseRef-BSD",
   "licenseDeclared": "LicenseRef-BSD",
   "copyrightText": "NOASSERTION",
   "externalRefs": [
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive:libarchive:0\\:3.6.1-1.el8:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "PACKAGE-MANAGER",
     "referenceType": "purl",
     "referenceLocator": "pkg:rpm/libarchive@3.6.1-1.el8?arch=x86_64&epoch=0&upstream=libarchive-3.6.1-1.el8.src.rpm"
    }
   ]
  },
  {
   "name": "libarchive-devel",
   "SPDXID": "SPDXRef-Package-rpm-libarchive-devel-65049410ade5a32b",
   "versionInfo": "0:3.6.1-1.el7",
   "downloadLocation": "NOASSERTION",
   "sourceInfo": "acquired package info from RPM DB: iron-appmgr-develop/rpms.libarchive.7/libarchive-devel-3.6.1-1.el7.x86_64.rpm",
   "licenseConcluded": "LicenseRef-BSD",
   "licenseDeclared": "LicenseRef-BSD",
   "copyrightText": "NOASSERTION",
   "externalRefs": [
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive-devel:libarchive-devel:0\\:3.6.1-1.el7:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive-devel:libarchive_devel:0\\:3.6.1-1.el7:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive_devel:libarchive-devel:0\\:3.6.1-1.el7:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive_devel:libarchive_devel:0\\:3.6.1-1.el7:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive:libarchive-devel:0\\:3.6.1-1.el7:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive:libarchive_devel:0\\:3.6.1-1.el7:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "PACKAGE-MANAGER",
     "referenceType": "purl",
     "referenceLocator": "pkg:rpm/libarchive-devel@3.6.1-1.el7?arch=x86_64&epoch=0&upstream=libarchive-3.6.1-1.el7.src.rpm"
    }
   ]
  },
  {
   "name": "libarchive-devel",
   "SPDXID": "SPDXRef-Package-rpm-libarchive-devel-2c9aa4099f7fe8f",
   "versionInfo": "0:3.6.1-1.el8",
   "downloadLocation": "NOASSERTION",
   "sourceInfo": "acquired package info from RPM DB: iron-appmgr-develop/rpms.libarchive.8/libarchive-devel-3.6.1-1.el8.x86_64.rpm",
   "licenseConcluded": "LicenseRef-BSD",
   "licenseDeclared": "LicenseRef-BSD",
   "copyrightText": "NOASSERTION",
   "externalRefs": [
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive-devel:libarchive-devel:0\\:3.6.1-1.el8:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive-devel:libarchive_devel:0\\:3.6.1-1.el8:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive_devel:libarchive-devel:0\\:3.6.1-1.el8:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive_devel:libarchive_devel:0\\:3.6.1-1.el8:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive:libarchive-devel:0\\:3.6.1-1.el8:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:libarchive:libarchive_devel:0\\:3.6.1-1.el8:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "PACKAGE-MANAGER",
     "referenceType": "purl",
     "referenceLocator": "pkg:rpm/libarchive-devel@3.6.1-1.el8?arch=x86_64&epoch=0&upstream=libarchive-3.6.1-1.el8.src.rpm"
    }
   ]
  }
 ],
 "hasExtractedLicensingInfos": [
  {
   "licenseId": "LicenseRef-BSD",
   "extractedText": "NONE",
   "name": "BSD"
  }
 ],
 "relationships": [
  {
   "spdxElementId": "SPDXRef-DOCUMENT",
   "relatedSpdxElement": "SPDXRef-DOCUMENT",
   "relationshipType": "DESCRIBES"
  }
 ]
}

Thanks!

4

You need to share the error log line, you didn't share it, just part of the error.

5

Hi sir,

Let me give you the error log later, it is in my the other laptop. :frowning:

Thanks

6

Hi Sir,

Please according to this error message:

Error Log

Thanks!

7

The error you mentioned is not present in the file you shared, are you sure this is the correct file?

Also, you can disable DEBUG level for now, it adds too much noise, almost every time is possible to know what is the issue only with WARN and ERROR logs.

8

Hi sir,

"I found the solution. Since I was running ELK on a virtual machine, I think the performance of the VM may have caused this issue. I modified the VM settings and reduced the some strings of JSON logs, and now it works. Thank you for your assistance. :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.