# Elasticsearch on Spark - case class limit to 22 fields on Scala

**URL:** https://discuss.elastic.co/t/elasticsearch-on-spark-case-class-limit-to-22-fields-on-scala/27392
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [August 14, 2015, 12:07pm UTC](https://discuss.elastic.co/t/elasticsearch-on-spark-case-class-limit-to-22-fields-on-scala/27392 "2015-08-14T12:07:45Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![eliasah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eliasah/32/34741_2.png) [@eliasah](https://discuss.elastic.co/u/eliasah)
#### Post date: [August 14, 2015, 12:07pm UTC](https://discuss.elastic.co/t/elasticsearch-on-spark-case-class-limit-to-22-fields-on-scala/27392/1 "2015-08-14T12:07:45Z")

</div>

I'm trying to work on the [kdd99 Dataset](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html) for Fraud Detection. In the dataset, a record looks like this :

`0,tcp,http,SF,215,45076,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,0,0,0.00,0.00,0.00,0.00,0.00,.00,0.00,0.00,normal.`

A record represents a connection. For each connection, the data set contains information like the number of bytes sent, login attempts, TCP errors, and so on. Each connection is one line of CSV-formatted data, containing 38 features.

So what I am trying to do is to write the data into Elasticsearch using Spark, so I can first analyze it with Kibana on a visual level, before performing deeper computation with Spark to predict whether a record is a fraudulent action or not.

The issue is that till Scala 2.10, a case class is limited to 22 fields.

Which means that I can't create a case class to associate to a record.

How can I go around this limitation without switching to Scala 2.11 which seems that can solve the issue ([SI-7296](https://issues.scala-lang.org/browse/SI-7296))?

I appreciate your help. Thanks in advance!

---

<div class="post-metadata">

### Author: ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [August 18, 2015, 9:36am UTC](https://discuss.elastic.co/t/elasticsearch-on-spark-case-class-limit-to-22-fields-on-scala/27392/2 "2015-08-18T09:36:43Z")

</div>

Case classes are just an option of types that can be serialized out of the box. You can just as well use a `Map` (whether in Scala or Java) or a JavaBean - though I would recommend the former especially considering the big number of parameters involved.

---

<div class="post-metadata">

### Author: ![eliasah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eliasah/32/34741_2.png) [@eliasah](https://discuss.elastic.co/u/eliasah)
#### Post date: [August 18, 2015, 9:58am UTC](https://discuss.elastic.co/t/elasticsearch-on-spark-case-class-limit-to-22-fields-on-scala/27392/3 "2015-08-18T09:58:31Z")

</div>

Ok thanks! So actually you'll recommend using a classic Map structure?

---

<div class="post-metadata">

### Author: ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [August 18, 2015, 10:01am UTC](https://discuss.elastic.co/t/elasticsearch-on-spark-case-class-limit-to-22-fields-on-scala/27392/4 "2015-08-18T10:01:19Z")

</div>

A case class is just that - a strongly-typed `Map`. Why not use it? Especially if the properties fall under the same type and you can add some generics to it, the `Map` should work just fine and have no size issue.

---

<div class="post-metadata">

### Author: ![eliasah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eliasah/32/34741_2.png) [@eliasah](https://discuss.elastic.co/u/eliasah)
#### Post date: [August 18, 2015, 10:02am UTC](https://discuss.elastic.co/t/elasticsearch-on-spark-case-class-limit-to-22-fields-on-scala/27392/5 "2015-08-18T10:02:53Z")

</div>

Great. Thanks! Your solution seems quite logical now. 😄

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:27pm UTC](https://discuss.elastic.co/t/elasticsearch-on-spark-case-class-limit-to-22-fields-on-scala/27392/6 "2017-07-06T13:27:39Z")

</div>


