scala - Spark Jobserver Unit Testing of jobs -

i want write unit tests spark jobs executed in spark-jobserver. works fine unless need access config e.g. check specific input values like:

try(config.getstring("mykey"))       .map(x => sparkjobvalid)       .getorelse(sparkjobinvalid("no value mykey config param")) 

the config created follows:

import com.typesafe.config.config val myconfig = configfactory.parsestring("key=value") 

then job run like:

myjob.run(sqlctx, myconfig)) 

this exception:

error utils: uncaught exception in thread driver-heartbeater   java.io.ioexception: java.lang.classcastexception: cannot assign instance of scala.collection.immutable.hashmap$serializationproxy field org.apache.spark.executor.taskmetrics._accumulatorupdates of type scala.collection.immutable.map in instance of org.apache.spark.executor.taskmetrics         @ org.apache.spark.util.utils$.tryorioexception(utils.scala:1163)         @ org.apache.spark.executor.taskmetrics.readobject(taskmetrics.scala:219)         @ sun.reflect.generatedmethodaccessor38.invoke(unknown source)         @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)         @ java.lang.reflect.method.invoke(method.java:498)         @ java.io.objectstreamclass.invokereadobject(objectstreamclass.java:1058)         @ java.io.objectinputstream.readserialdata(objectinputstream.java:1900)         @ java.io.objectinputstream.readordinaryobject(objectinputstream.java:1801)         @ java.io.objectinputstream.readobject0(objectinputstream.java:1351)         @ java.io.objectinputstream.readobject(objectinputstream.java:371)         @ org.apache.spark.util.utils$.deserialize(utils.scala:91)         @ org.apache.spark.executor.executor$$anonfun$org$apache$spark$executor$executor$$reportheartbeat$1$$anonfun$apply$6.apply(executor.scala:440)         @ org.apache.spark.executor.executor$$anonfun$org$apache$spark$executor$executor$$reportheartbeat$1$$anonfun$apply$6.apply(executor.scala:430)         @ scala.option.foreach(option.scala:236)         @ org.apache.spark.executor.executor$$anonfun$org$apache$spark$executor$executor$$reportheartbeat$1.apply(executor.scala:430)         @ org.apache.spark.executor.executor$$anonfun$org$apache$spark$executor$executor$$reportheartbeat$1.apply(executor.scala:428)         @ scala.collection.iterator$class.foreach(iterator.scala:727)         @ scala.collection.abstractiterator.foreach(iterator.scala:1157)         @ scala.collection.iterablelike$class.foreach(iterablelike.scala:72)         @ scala.collection.abstractiterable.foreach(iterable.scala:54)         @ org.apache.spark.executor.executor.org$apache$spark$executor$executor$$reportheartbeat(executor.scala:428)         @ org.apache.spark.executor.executor$$anon$1$$anonfun$run$1.apply$mcv$sp(executor.scala:472)         @ org.apache.spark.executor.executor$$anon$1$$anonfun$run$1.apply(executor.scala:472)         @ org.apache.spark.executor.executor$$anon$1$$anonfun$run$1.apply(executor.scala:472)         @ org.apache.spark.util.utils$.loguncaughtexceptions(utils.scala:1699)         @ org.apache.spark.executor.executor$$anon$1.run(executor.scala:472)         @ java.util.concurrent.executors$runnableadapter.call(executors.java:511)         @ java.util.concurrent.futuretask.runandreset(futuretask.java:308)         @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.access$301(scheduledthreadpoolexecutor.java:180)         @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.run(scheduledthreadpoolexecutor.java:294)         @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142)         @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617)         @ java.lang.thread.run(thread.java:745) caused by: java.lang.classcastexception: cannot assign instance of scala.collection.immutable.hashmap$serializationproxy field org.apache.spark.executor.taskmetrics._accumulatorupdates of type scala.collection.immutable.map in instance of org.apache.spark.executor.taskmetrics         @ java.io.objectstreamclass$fieldreflector.setobjfieldvalues(objectstreamclass.java:2133)         @ java.io.objectstreamclass.setobjfieldvalues(objectstreamclass.java:1305)         @ java.io.objectinputstream.defaultreadfields(objectinputstream.java:2006)         @ java.io.objectinputstream.defaultreadobject(objectinputstream.java:501)         @ org.apache.spark.executor.taskmetrics$$anonfun$readobject$1.apply$mcv$sp(taskmetrics.scala:220)         @ org.apache.spark.util.utils$.tryorioexception(utils.scala:1160)         ... 32 more 

i can confirm latest spark 1.6.1 issue gone. jobs not seem impacted exception.