[Rd] ClassNotFoundException when running distributed job using rJava package
Prof Brian Ripley
ripley at stats.ox.ac.uk
Fri Feb 8 08:16:53 CET 2013
This is not the rJava support list: that is at
http://www.rosuda.org/lists.shtml.
On 08/02/2013 06:57, Gaurav Dasgupta wrote:
> Hi,
>
> I have a MapReduce Java code, which I am calling from R using rJava. I have
> prepared the R package and tested that successfully. But when I deployed
> the package in a cluster and executed it, I am getting
> ClassNotFoundException. If I run the same job directly without integrating
> with R, it runs perfectly.
> Here is my R code:
>
> library(rJava)
> muMstSpark <- function(mesosMaster = NULL, input = NULL, output = NULL,
> scalaLib = NULL, sparkCore = NULL, inputSplits = 8) {
> if (is.null(mesosMaster) || is.null(input) || is.null(output) ||
> is.null(scalaLib) || is.null(sparkCore)) {
> stop("Usage: muMST(<mesosMaster>, <input>, <output>, <scalaLib>,
> <sparkCore>, [<inputSplits>]")
> }
>
> # Gets the absolute path of the external Scala and Java JARS
> pkgPath = paste(system.file(package="MuMstBig"), "/jars", sep="")
>
> # Initializes the JVM specifying the directory where the main Java class
> resides:
> .jinit("pkgPath")
>
> # Adds all the required JARs to the class path:
> .jaddClassPath(paste(pkgPath, "Prims.jar", sep="/"))
> .jaddClassPath(paste(pkgPath, "MSTInSpark.jar", sep="/"))
> .jaddClassPath(scalaLib)
> .jaddClassPath(sparkCore)
>
> # Creates the R object for the main Java class:
> obj <- .jnew("MSTInSpark")
>
> # Calls the Java main class
> .jcall(obj, "V", "mst", c(mesosMaster, input, output, inputSplits))
> }
> Here is the error log:
>
> 13/02/08 00:54:48 INFO cluster.TaskSetManager: Loss was due to
> java.lang.ClassNotFoundException: Prims$$anonfun$PrimsExecute$1
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> spark.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:20)
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:435)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:435)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> at spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
> at spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45)
> at spark.executor.Executor$TaskRunner.run(Executor.scala:93)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> I think R is unable to find the classpath. But I have specified that in the
> script by taking the absolute path of the JARs in the package. The
> package's installed across the cluster. Any idea, whats going wrong?
>
> Thanks,
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
--
Brian D. Ripley, ripley at stats.ox.ac.uk
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UK Fax: +44 1865 272595
More information about the R-devel
mailing list