Logstash split filter issue

Elastic Stack Logstash
1

Hi...

I have a Azure CosmosDB log message in this format:


{
"records":
[
{
"time": "Fri, 23 Jun 2017 19:29:50.266 GMT",
"resourceId": "contosocosmosdb",
"category": "DataPlaneRequests",
"operationName": "Query",
"resourceType": "Database",
"properties": {"activityId": "05fcf607-6f64-48fe-81a5-f13ac13dd1eb","userAgent": "documentdb-dotnet-sdk/1.12.0 Host/64-bit MicrosoftWindowsNT/6.2.9200.0 AzureSearchIndexer/1.0.0",
"resourceType": "Database","statusCode": "200","documentResourceId": "","clientIpAddress": "13.92.241.0","requestCharge": "2.260","collectionRid": "",
"duration": "9250","requestLength": "72","responseLength": "209", "resourceTokenUserRid": ""}
}
]
}

I'm trying to split each field but facing some issues with the logstash split filter.
Error: [WARN ][logstash.filters.split ] Only String and Array types are splittable. field:records is of type = NilClass

This is my logstash config file:
input
{
azureblob
{
storage_account_name => "XXXX"
storage_access_key => "XXX"
container => "dataplanerequests"
registry_create_policy => "resume"
type => "XXX"
codec => json
}

filter {
if [type] == "XXX"
{
split
{
field => "records"
add_field => {
"logtime" => "%{[records][time]}"
"operationName" => "%{[records][operationName]}"
"category" => "%{[records][category]}"
"activityId" => "%{[records][properties][activityId]}"
"requestResourceType" => "%{[records][properties][requestResourceType]}"
"requestResourceId" => "%{[records][properties][requestResourceId]}"
"collectionRid" => "%{[records][properties][collectionRid]}"
"statusCode" => "%{[records][properties][statusCode]}"
"duration" => "%{[records][properties][duration]}"
"userAgent" => "%{[records][properties][userAgent]}"
"clientIpAddress" => "%{[records][properties][clientIpAddress]}"
"requestCharge" => "%{[records][properties][requestCharge]}"
"requestLength" => "%{[records][properties][requestLength]}"
"responseLength" => "%{[records][properties][responseLength]}"
"region" => "%{[records][properties][region]}"
}
remove_field => ["records"]
}
}
}
output
{
if [type] == "XXX"
{
kafka
{
codec => json_lines
topic_id => "XXX"
bootstrap_servers => "XXX"
id => "XXX"
}
}
}

Few logs are able to parse properly but few are unable to parse.

2

Error: [WARN ][logstash.filters.split ] Only String and Array types are splittable. field:records is of type = NilClass

This indicates that the event didn't have a records field. Never mind what the input looks like, what does an event look like to Logstash? Use a stdout { codec => rubydebug } output to dump raw events.

3

Hi Magnus,

I tried the codec => rubydebug but unable to find the issue. how to split each field if i have a log message like the one below. FYI: I'm using logstash as a service.

{
"records":
[

	{
		 "time": "---------------",
		 "resourceId": "-----------------",
		 "category": "---------------",
		 "operationName": "------",
		 "properties": {"activityId": "---","requestResourceType": "----","requestResourceId": "----------","collectionRid": "----","statusCode": "---","duration": "----","userAgent": "---","clientIpAddress": "---","requestCharge": "----","requestLength": "----","responseLength": "---","resourceTokenUserRid": "---","region": "----","partitionId": "-----"}
	}
	,
	{
		 "time": "---------------",
		 "resourceId": "-----------------",
		 "category": "---------------",
		 "operationName": "------",
		 "properties": {"activityId": "---","requestResourceType": "----","requestResourceId": "----------","collectionRid": "----","statusCode": "---","duration": "----","userAgent": "---","clientIpAddress": "---","requestCharge": "----","requestLength": "----","responseLength": "---","resourceTokenUserRid": "---","region": "----","partitionId": "-----"}
	}
	,
	{
		 "time": "---------------",
		 "resourceId": "-----------------",
		 "category": "---------------",
		 "operationName": "------",
		 "properties": {"activityId": "---","requestResourceType": "----","requestResourceId": "----------","collectionRid": "----","statusCode": "---","duration": "----","userAgent": "---","clientIpAddress": "---","requestCharge": "----","requestLength": "----","responseLength": "---","resourceTokenUserRid": "---","region": "----","partitionId": "-----"}
	}
]

}

4

This is how my filter section looks like;

filter {
split
{
field => "records"
add_field => {
"logtime" => "%{[records][time]}"
"operationName" => "%{[records][operationName]}"
"category" => "%{[records][category]}"
"activityId" => "%{[records][properties][activityId]}"
"requestResourceType" => "%{[records][properties][requestResourceType]}"
"requestResourceId" => "%{[records][properties][requestResourceId]}"
"collectionRid" => "%{[records][properties][collectionRid]}"
"statusCode" => "%{[records][properties][statusCode]}"
"duration" => "%{[records][properties][duration]}"
"userAgent" => "%{[records][properties][userAgent]}"
"clientIpAddress" => "%{[records][properties][clientIpAddress]}"
"requestCharge" => "%{[records][properties][requestCharge]}"
"requestLength" => "%{[records][properties][requestLength]}"
"responseLength" => "%{[records][properties][responseLength]}"
"region" => "%{[records][properties][region]}"
"partitionId" => "%{[records][properties][partitionId]}"
}
remove_field => ["records"]
}
mutate
{
add_field => { "token" => "XXXX" }
}
}
}

5

I tried the codec => rubydebug but unable to find the issue.

No, but maybe we can. Post the stdout output.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.