Logstash fails to parse valid json, I need a "split" but I get "Only String and Array types are splittable."

Elastic Stack Elasticsearch
1

The logfile I need to ingest:

{
  "policy": {
    "name": "account-cloudtrail-enabled",
    "resource": "account",
    "description": "Checks to make sure CloudTrail is enabled on the account\nfor all regions.\n",
    "filters": [
      {
        "type": "check-cloudtrail",
        "global-events": false,
        "multi-region": false,
        "running": false,
        "file-digest": false
      }
    ]
  },
  "version": "0.9.13",
  "execution": {
    "id": "1ebc9860-6d1a-4e42-b809-0fad544479fe",
    "start": 1638815388.1077602,
    "end_time": 1638815388.935413,
    "duration": 0.8276526927947998
  },
  "config": {
    "region": "us-east-2",
    "regions": [
      "us-east-2"
    ],
    "cache": "~/.cache/cloud-custodian.cache",
    "profile": "CCAdmin",
    "account_id": "353563186465",
    "assume_role": null,
    "external_id": null,
    "log_group": null,
    "tracer": null,
    "metrics_enabled": null,
    "metrics": null,
    "output_dir": "s3://testcclog/custodian/",
    "cache_period": 15,
    "dryrun": false,
    "authorization_file": null,
    "subparser": "run",
    "config": null,
    "configs": [
      "./policies/root_account-compliance.yml"
    ],
    "policy_filters": [],
    "resource_types": [],
    "verbose": null,
    "quiet": null,
    "debug": false,
    "skip_validation": false,
    "command": "c7n.commands.run",
    "vars": null
  },
  "sys-stats": {},
  "api-stats": {
    "iam.ListAccountAliases": 1,
    "cloudtrail.DescribeTrails": 1
  },
  "metrics": [
    {
      "MetricName": "ResourceCount",
      "Timestamp": "2021-12-06T11:29:48.934903",
      "Value": 0,
      "Unit": "Count"
    },
    {
      "MetricName": "ResourceTime",
      "Timestamp": "2021-12-06T11:29:48.934920",
      "Value": 0.8265008926391602,
      "Unit": "Seconds"
    }
  ]
}

but logstash chokes on it:

[2021-12-16T22:26:37,524][DEBUG][logstash.filters.json    ][main][5760067d2c61b3b7732f165643696a8b23c1d8f10e61ade0441a188868bbd967] Running json filter {:event=>{"@version"=>"1", "path"=>"/etc/logstash/sample/raw.log", "@timestamp"=>2021-12-16T22:26:37.324Z, "host"=>"ip-172-31-29-221.us-east-2.compute.internal", "message"=>"{\n  \"policy\": {\n    \"name\": \"account-cloudtrail-enabled\",\n    \"resource\": \"account\",\n    \"description\": \"Checks to make sure CloudTrail is enabled on the account\\nfor all regions.\\n\",\n    \"filters\": [\n      {\n        \"type\": \"check-cloudtrail\",\n        \"global-events\": false,\n        \"multi-region\": false,\n        \"running\": false,\n        \"file-digest\": false\n      }\n    ]\n  },\n  \"version\": \"0.9.13\",\n  \"execution\": {\n    \"id\": \"1ebc9860-6d1a-4e42-b809-0fad544479fe\",\n    \"start\": 1638815388.1077602,\n    \"end_time\": 1638815388.935413,\n    \"duration\": 0.8276526927947998\n  },\n  \"config\": {\n    \"region\": \"us-east-2\",\n    \"regions\": [\n      \"us-east-2\"\n    ],\n    \"cache\": \"~/.cache/cloud-custodian.cache\",\n    \"profile\": \"CCAdmin\",\n    \"account_id\": \"353563186465\",\n    \"assume_role\": null,\n    \"external_id\": null,\n    \"log_group\": null,\n    \"tracer\": null,\n    \"metrics_enabled\": null,\n    \"metrics\": null,\n    \"output_dir\": \"s3://testcclog/custodian/\",\n    \"cache_period\": 15,\n    \"dryrun\": false,\n    \"authorization_file\": null,\n    \"subparser\": \"run\",\n    \"config\": null,\n    \"configs\": [\n      \"./policies/root_account-compliance.yml\"\n    ],\n    \"policy_filters\": [],\n    \"resource_types\": [],\n    \"verbose\": null,\n    \"quiet\": null,\n    \"debug\": false,\n    \"skip_validation\": false,\n    \"command\": \"c7n.commands.run\",\n    \"vars\": null\n  },\n  \"sys-stats\": {},\n  \"api-stats\": {\n    \"iam.ListAccountAliases\": 1,\n    \"cloudtrail.DescribeTrails\": 1\n  },\n  \"metrics\": [\n    {\n      \"MetricName\": \"ResourceCount\",\n      \"Timestamp\": \"2021-12-06T11:29:48.934903\",\n      \"Value\": 0,\n      \"Unit\": \"Count\"\n    },\n    {\n      \"MetricName\": \"ResourceTime\",\n      \"Timestamp\": \"2021-12-06T11:29:48.934920\",\n      \"Value\": 0.8265008926391602,\n      \"Unit\": \"Seconds\"\n    }\n  ]\n}"}}
[2021-12-16T22:26:37,540][DEBUG][logstash.filters.json    ][main][5760067d2c61b3b7732f165643696a8b23c1d8f10e61ade0441a188868bbd967] Event after json filter {:event=>{"message"=>"{\n  \"policy\": {\n    \"name\": \"account-cloudtrail-enabled\",\n    \"resource\": \"account\",\n    \"description\": \"Checks to make sure CloudTrail is enabled on the account\\nfor all regions.\\n\",\n    \"filters\": [\n      {\n        \"type\": \"check-cloudtrail\",\n        \"global-events\": false,\n        \"multi-region\": false,\n        \"running\": false,\n        \"file-digest\": false\n      }\n    ]\n  },\n  \"version\": \"0.9.13\",\n  \"execution\": {\n    \"id\": \"1ebc9860-6d1a-4e42-b809-0fad544479fe\",\n    \"start\": 1638815388.1077602,\n    \"end_time\": 1638815388.935413,\n    \"duration\": 0.8276526927947998\n  },\n  \"config\": {\n    \"region\": \"us-east-2\",\n    \"regions\": [\n      \"us-east-2\"\n    ],\n    \"cache\": \"~/.cache/cloud-custodian.cache\",\n    \"profile\": \"CCAdmin\",\n    \"account_id\": \"353563186465\",\n    \"assume_role\": null,\n    \"external_id\": null,\n    \"log_group\": null,\n    \"tracer\": null,\n    \"metrics_enabled\": null,\n    \"metrics\": null,\n    \"output_dir\": \"s3://testcclog/custodian/\",\n    \"cache_period\": 15,\n    \"dryrun\": false,\n    \"authorization_file\": null,\n    \"subparser\": \"run\",\n    \"config\": null,\n    \"configs\": [\n      \"./policies/root_account-compliance.yml\"\n    ],\n    \"policy_filters\": [],\n    \"resource_types\": [],\n    \"verbose\": null,\n    \"quiet\": null,\n    \"debug\": false,\n    \"skip_validation\": false,\n    \"command\": \"c7n.commands.run\",\n    \"vars\": null\n  },\n  \"sys-stats\": {},\n  \"api-stats\": {\n    \"iam.ListAccountAliases\": 1,\n    \"cloudtrail.DescribeTrails\": 1\n  },\n  \"metrics\": [\n    {\n      \"MetricName\": \"ResourceCount\",\n      \"Timestamp\": \"2021-12-06T11:29:48.934903\",\n      \"Value\": 0,\n      \"Unit\": \"Count\"\n    },\n    {\n      \"MetricName\": \"ResourceTime\",\n      \"Timestamp\": \"2021-12-06T11:29:48.934920\",\n      \"Value\": 0.8265008926391602,\n      \"Unit\": \"Seconds\"\n    }\n  ]\n}", "@timestamp"=>2021-12-16T22:26:37.324Z, "@version"=>"1", "cc-data"=>{"api-stats"=>{"cloudtrail.DescribeTrails"=>1, "iam.ListAccountAliases"=>1}, "version"=>"0.9.13", "config"=>{"profile"=>"CCAdmin", "command"=>"c7n.commands.run", "region"=>"us-east-2", "cache_period"=>15, "metrics"=>nil, "dryrun"=>false, "configs"=>["./policies/root_account-compliance.yml"], "resource_types"=>[], "authorization_file"=>nil, "assume_role"=>nil, "verbose"=>nil, "policy_filters"=>[], "vars"=>nil, "log_group"=>nil, "output_dir"=>"s3://testcclog/custodian/", "regions"=>["us-east-2"], "quiet"=>nil, "external_id"=>nil, "skip_validation"=>false, "cache"=>"~/.cache/cloud-custodian.cache", "tracer"=>nil, "subparser"=>"run", "config"=>nil, "debug"=>false, "account_id"=>"353563186465", "metrics_enabled"=>nil}, "execution"=>{"duration"=>0.8276526927947998e0, "start"=>0.16388153881077602e10, "id"=>"1ebc9860-6d1a-4e42-b809-0fad544479fe", "end_time"=>0.1638815388935413e10}, "metrics"=>[{"Unit"=>"Count", "MetricName"=>"ResourceCount", "Timestamp"=>"2021-12-06T11:29:48.934903", "Value"=>0}, {"Unit"=>"Seconds", "MetricName"=>"ResourceTime", "Timestamp"=>"2021-12-06T11:29:48.934920", "Value"=>0.8265008926391602e0}], "policy"=>{"name"=>"account-cloudtrail-enabled", "resource"=>"account", "filters"=>[{"file-digest"=>false, "type"=>"check-cloudtrail", "running"=>false, "global-events"=>false, "multi-region"=>false}], "description"=>"Checks to make sure CloudTrail is enabled on the account\nfor all regions.\n"}, "sys-stats"=>{}}, "host"=>"ip-172-31-29-221.us-east-2.compute.internal", "path"=>"/etc/logstash/sample/raw.log"}}
[2021-12-16T22:26:37,548][WARN ][logstash.filters.split   ][main][9222132c50512e9057c2b8b64c03cd0c4160ce44e794e7b898d3c819d3fdf7de] Only String and Array types are splittable. field:metrics is of type = NilClass

Configs
input:

input {
    file {
        start_position => "beginning"
        path => "/etc/logstash/sample/cctest1.log"
        sincedb_path => "/dev/null"
        codec => multiline {
            pattern => "^({|\[)\s*$"
            negate => true
            auto_flush_interval => 1
            multiline_tag => ""
            what => "previous"
        }
    }
}

filter:

filter {
    json {
        source => "message"
        target => "cc-data"
    }

    split {
        field => "metrics"
    }

    mutate {
        remove_field => ["@timestamp", "@version", "host"]

    }
}

I know that if I remove either of the objects in "metrics", it works. I was hoping that a "split" would fix this... but maybe I did it wrong?

2

You are using a target in your json filter, so your metrics field won't be in the root of your document, but will be inside the target field cc-data.

Try this:

split {
    field => "[cc-data][metrics]"
}
3

You are my new favorite person.

4

well... I am still getting errors... :frowning:

[2021-12-17T16:40:11,679][WARN ][logstash.outputs.elasticsearch][main][3d8eaac10327140fa39306871fc75d5aa89516e9529a22174ecc9bc37492193b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"art", :routing=>nil}, {"cc-data"=>{"metrics"=>[{"Timestamp"=>"2021-11-24T15:28:21.589209", "Unit"=>"Count", "Value"=>0, "MetricName"=>"ResourceCount"}, {"Timestamp"=>"2021-11-24T15:28:21.589220", "Unit"=>"Seconds", "Value"=>0.9067964553833008e0, "MetricName"=>"ResourceTime"}], "execution"=>{"duration"=>0.10449738502502441e1, "start"=>0.16377677005444033e10, "end_time"=>0.16377677015893772e10, "id"=>"9d1aeddf-7e36-4f6e-9dff-39e6b068dc8e"}, "version"=>"0.9.13", "config"=>{"profile"=>nil, "log_group"=>nil, "tracer"=>"default", "cache"=>"", "dryrun"=>false, "authorization_file"=>nil, "assume_role"=>nil, "cache_period"=>0, "metrics_enabled"=>false, "metrics"=>nil, "regions"=>[], "region"=>"us-east-2", "output_dir"=>"s3://testcclog/custodian/", "external_id"=>nil, "account_id"=>"353563186465"}, "policy"=>{"name"=>"cis-iam-stale-credentials", "filters"=>[{"or"=>[{"and"=>[{"key"=>"password_enabled", "type"=>"credential", "value"=>true}, {"key"=>"password_last_used", "op"=>"gt", "type"=>"credential", "value_type"=>"age", "value"=>90}]}, {"and"=>[{"key"=>"access_keys.active", "type"=>"credential", "value"=>true}, {"key"=>"access_keys.last_used_date", "op"=>"gt", "type"=>"credential", "value_type"=>"age", "value"=>90}]}]}], "actions"=>[{"transport"=>{"type"=>"sqs", "queue"=>"c7nMessageQueue"}, "violation_desc"=>"IAM user password and/or access keys are older than 90 days.\nCIS Amazon Web Services Foundations v1.1.0 (1.3)\n", "action_desc"=>"Disable stale credentials.", "type"=>"notify", "to"=>["slack"]}], "resource"=>"aws.iam-user", "comment"=>"CIS Amazon Web Services Foundations v1.1.0 (1.3)", "region"=>"us-east-2", "mode"=>{"schedule"=>"rate(24 hours)", "role"=>"arn:aws:iam::353563186465:role/CCLam", "type"=>"periodic", "tags"=>{"custodian-info"=>"mode=periodic:version=0.9.13"}}}, "api-stats"=>{"iam.ListUsers"=>1, "iam.GetUser"=>2, "iam.GetCredentialReport"=>4}, "sys-stats"=>{}}}], :response=>{"index"=>{"_index"=>"art", "_type"=>"_doc", "_id"=>"K6ZDyX0BMkLmKYp97GOc", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [cc-data.metrics.Value] cannot be changed from type [float] to [long]"}}}}
5

OK, after looking into this new error, this seems to be unrelated.

maybe: Metricbeat pct fields can be float and long which causes elasticsearch to throw an exception · Issue #5032 · elastic/beats · GitHub

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.