# Logstash split filter issue

**URL:** <https://discuss.elastic.co/t/logstash-split-filter-issue/119291>\
**Category:** Logstash\
**Created:** [February 9, 2018, 6:49pm UTC](https://discuss.elastic.co/t/logstash-split-filter-issue/119291 "2018-02-09T18:49:15Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shyam\_Burla](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Shyam\_Burla](https://discuss.elastic.co/u/Shyam_Burla)\
**Post date:** [February 9, 2018, 6:49pm UTC](https://discuss.elastic.co/t/logstash-split-filter-issue/119291/1 "2018-02-09T18:49:15Z")

</div>

Hi...

I have a Azure CosmosDB log message in this format:

> **[Azure Cosmos DB diagnostic logging](https://docs.microsoft.com/en-us/azure/cosmos-db/logging)**
>
> Use this tutorial to help you get started with Azure Cosmos DB logging.

  
{  
"records":  
[  
{  
"time": "Fri, 23 Jun 2017 19:29:50.266 GMT",  
"resourceId": "contosocosmosdb",  
"category": "DataPlaneRequests",  
"operationName": "Query",  
"resourceType": "Database",  
"properties": {"activityId": "05fcf607-6f64-48fe-81a5-f13ac13dd1eb",`"userAgent": "documentdb-dotnet-sdk/1.12.0 Host/64-bit MicrosoftWindowsNT/6.2.9200.0 AzureSearchIndexer/1.0.0",`  
"resourceType": "Database","statusCode": "200","documentResourceId": "",`"clientIpAddress": "13.92.241.0","requestCharge": "2.260","collectionRid": "",`  
"duration": "9250","requestLength": "72","responseLength": "209", "resourceTokenUserRid": ""}  
}  
]  
}

I'm trying to split each field but facing some issues with the logstash split filter.  
Error: [WARN][logstash.filters.split] Only String and Array types are splittable. field:records is of type = NilClass

This is my logstash config file:  
input  
{  
azureblob  
{  
storage\_account\_name =\> "XXXX"  
storage\_access\_key =\> "XXX"  
container =\> "dataplanerequests"  
registry\_create\_policy =\> "resume"  
type =\> "XXX"  
codec =\> json  
}

filter {  
if [type] == "XXX"  
{  
split  
{  
field =\> "records"  
add\_field =\> {  
"logtime" =\> "%{[records][time]}"  
"operationName" =\> "%{[records][operationName]}"  
"category" =\> "%{[records][category]}"  
"activityId" =\> "%{[records][properties][activityId]}"  
"requestResourceType" =\> "%{[records][properties][requestResourceType]}"  
"requestResourceId" =\> "%{[records][properties][requestResourceId]}"  
"collectionRid" =\> "%{[records][properties][collectionRid]}"  
"statusCode" =\> "%{[records][properties][statusCode]}"  
"duration" =\> "%{[records][properties][duration]}"  
"userAgent" =\> "%{[records][properties][userAgent]}"  
"clientIpAddress" =\> "%{[records][properties][clientIpAddress]}"  
"requestCharge" =\> "%{[records][properties][requestCharge]}"  
"requestLength" =\> "%{[records][properties][requestLength]}"  
"responseLength" =\> "%{[records][properties][responseLength]}"  
"region" =\> "%{[records][properties][region]}"  
}  
remove\_field =\> ["records"]  
}  
}  
}  
output  
{  
if [type] == "XXX"  
{  
kafka  
{  
codec =\> json\_lines  
topic\_id =\> "XXX"  
bootstrap\_servers =\> "XXX"  
id =\> "XXX"  
}  
}  
}

Few logs are able to parse properly but few are unable to parse.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 12, 2018, 8:52pm UTC](https://discuss.elastic.co/t/logstash-split-filter-issue/119291/2 "2018-02-12T20:52:05Z")

</div>

> Error: [WARN][logstash.filters.split] Only String and Array types are splittable. field:records is of type = NilClass

This indicates that the event didn't have a `records` field. Never mind what the input looks like, what does an event look like to Logstash? Use a `stdout { codec => rubydebug }` output to dump raw events.

---

<div class="post-metadata">

**Author:** ![Shyam\_Burla](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Shyam\_Burla](https://discuss.elastic.co/u/Shyam_Burla)\
**Post date:** [February 12, 2018, 11:33pm UTC](https://discuss.elastic.co/t/logstash-split-filter-issue/119291/3 "2018-02-12T23:33:08Z")

</div>

Hi Magnus,

I tried the codec =\> rubydebug but unable to find the issue. how to split each field if i have a log message like the one below. FYI: I'm using logstash as a service.

{  
"records":  
[

```
	{
		 "time": "---------------",
		 "resourceId": "-----------------",
		 "category": "---------------",
		 "operationName": "------",
		 "properties": {"activityId": "---","requestResourceType": "----","requestResourceId": "----------","collectionRid": "----","statusCode": "---","duration": "----","userAgent": "---","clientIpAddress": "---","requestCharge": "----","requestLength": "----","responseLength": "---","resourceTokenUserRid": "---","region": "----","partitionId": "-----"}
	}
	,
	{
		 "time": "---------------",
		 "resourceId": "-----------------",
		 "category": "---------------",
		 "operationName": "------",
		 "properties": {"activityId": "---","requestResourceType": "----","requestResourceId": "----------","collectionRid": "----","statusCode": "---","duration": "----","userAgent": "---","clientIpAddress": "---","requestCharge": "----","requestLength": "----","responseLength": "---","resourceTokenUserRid": "---","region": "----","partitionId": "-----"}
	}
	,
	{
		 "time": "---------------",
		 "resourceId": "-----------------",
		 "category": "---------------",
		 "operationName": "------",
		 "properties": {"activityId": "---","requestResourceType": "----","requestResourceId": "----------","collectionRid": "----","statusCode": "---","duration": "----","userAgent": "---","clientIpAddress": "---","requestCharge": "----","requestLength": "----","responseLength": "---","resourceTokenUserRid": "---","region": "----","partitionId": "-----"}
	}
]

```

}

---

<div class="post-metadata">

**Author:** ![Shyam\_Burla](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Shyam\_Burla](https://discuss.elastic.co/u/Shyam_Burla)\
**Post date:** [February 12, 2018, 11:36pm UTC](https://discuss.elastic.co/t/logstash-split-filter-issue/119291/4 "2018-02-12T23:36:24Z")

</div>

This is how my filter section looks like;

filter {  
split  
{  
field =\> "records"  
add\_field =\> {  
"logtime" =\> "%{[records][time]}"  
"operationName" =\> "%{[records][operationName]}"  
"category" =\> "%{[records][category]}"  
"activityId" =\> "%{[records][properties][activityId]}"  
"requestResourceType" =\> "%{[records][properties][requestResourceType]}"  
"requestResourceId" =\> "%{[records][properties][requestResourceId]}"  
"collectionRid" =\> "%{[records][properties][collectionRid]}"  
"statusCode" =\> "%{[records][properties][statusCode]}"  
"duration" =\> "%{[records][properties][duration]}"  
"userAgent" =\> "%{[records][properties][userAgent]}"  
"clientIpAddress" =\> "%{[records][properties][clientIpAddress]}"  
"requestCharge" =\> "%{[records][properties][requestCharge]}"  
"requestLength" =\> "%{[records][properties][requestLength]}"  
"responseLength" =\> "%{[records][properties][responseLength]}"  
"region" =\> "%{[records][properties][region]}"  
"partitionId" =\> "%{[records][properties][partitionId]}"  
}  
remove\_field =\> ["records"]  
}  
mutate  
{  
add\_field =\> { "token" =\> "XXXX" }  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 13, 2018, 7:12am UTC](https://discuss.elastic.co/t/logstash-split-filter-issue/119291/5 "2018-02-13T07:12:28Z")

</div>

> I tried the codec =\> rubydebug but unable to find the issue.

No, but maybe we can. Post the stdout output.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 13, 2018, 7:12am UTC](https://discuss.elastic.co/t/logstash-split-filter-issue/119291/6 "2018-03-13T07:12:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.