-
-
Save shivaram/3a2fecce60768a603dac to your computer and use it in GitHub Desktop.
SparkR packages Jenkins error
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ===== TEST OUTPUT FOR o.a.s.deploy.SparkSubmitSuite: 'correctly builds R packages included in a jar with --packages' ===== | |
| 15/11/16 11:52:14.898 redirect stderr for command ./bin/spark-submit INFO Utils: Using properties file: null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: Parsed arguments: | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: master local-cluster[2,1,1024] | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: deployMode null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: executorMemory null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: executorCores null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: totalExecutorCores null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: propertiesFile null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: driverMemory null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: driverCores null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: driverExtraClassPath null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: driverExtraLibraryPath null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: driverExtraJavaOptions null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: supervise false | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: queue null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: numExecutors null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: files null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: pyFiles null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: archives null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: mainClass null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: primaryResource file:/home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/R/pkg/inst/tests/packageInAJarTest.R | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: name testApp | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: childArgs [] | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: jars null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: packages my.great.lib:mylib:0.1 | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: packagesExclusions null | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: repositories file:/home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/core/target/tmp/1447703534004-0/ | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: verbose true | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: Spark properties used, including those specified through | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: --conf and those from the properties file null: | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: spark.ui.enabled -> false | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: | |
| 15/11/16 11:52:14.921 redirect stderr for command ./bin/spark-submit INFO Utils: | |
| 15/11/16 11:52:15.044 redirect stderr for command ./bin/spark-submit INFO Utils: Ivy Default Cache set to: /home/jenkins/.ivy2/cache | |
| 15/11/16 11:52:15.044 redirect stderr for command ./bin/spark-submit INFO Utils: The jars for the packages stored in: /home/jenkins/.ivy2/jars | |
| 15/11/16 11:52:15.096 redirect stderr for command ./bin/spark-submit INFO Utils: file:/home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/core/target/tmp/1447703534004-0/ added as a remote repository with the name: repo-1 | |
| 15/11/16 11:52:15.148 redirect stderr for command ./bin/spark-submit INFO Utils: :: loading settings :: url = jar:file:/home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/assembly/target/scala-2.10/spark-assembly-1.6.0-SNAPSHOT-hadoop2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml | |
| 15/11/16 11:52:15.403 redirect stderr for command ./bin/spark-submit INFO Utils: my.great.lib#mylib added as a dependency | |
| 15/11/16 11:52:15.405 redirect stderr for command ./bin/spark-submit INFO Utils: :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 | |
| 15/11/16 11:52:15.405 redirect stderr for command ./bin/spark-submit INFO Utils: confs: [default] | |
| 15/11/16 11:52:16.458 redirect stderr for command ./bin/spark-submit INFO Utils: found my.great.lib#mylib;0.1 in repo-1 | |
| 15/11/16 11:52:16.476 redirect stderr for command ./bin/spark-submit INFO Utils: downloading file:/home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/core/target/tmp/1447703534004-0/my/great/lib/mylib/0.1/mylib-0.1.jar ... | |
| 15/11/16 11:52:16.477 redirect stderr for command ./bin/spark-submit INFO Utils: [SUCCESSFUL ] my.great.lib#mylib;0.1!mylib.jar (2ms) | |
| 15/11/16 11:52:16.478 redirect stderr for command ./bin/spark-submit INFO Utils: :: resolution report :: resolve 1069ms :: artifacts dl 4ms | |
| 15/11/16 11:52:16.478 redirect stderr for command ./bin/spark-submit INFO Utils: :: modules in use: | |
| 15/11/16 11:52:16.478 redirect stderr for command ./bin/spark-submit INFO Utils: my.great.lib#mylib;0.1 from repo-1 in [default] | |
| 15/11/16 11:52:16.478 redirect stderr for command ./bin/spark-submit INFO Utils: --------------------------------------------------------------------- | |
| 15/11/16 11:52:16.478 redirect stderr for command ./bin/spark-submit INFO Utils: | | modules || artifacts | | |
| 15/11/16 11:52:16.478 redirect stderr for command ./bin/spark-submit INFO Utils: | conf | number| search|dwnlded|evicted|| number|dwnlded| | |
| 15/11/16 11:52:16.478 redirect stderr for command ./bin/spark-submit INFO Utils: --------------------------------------------------------------------- | |
| 15/11/16 11:52:16.479 redirect stderr for command ./bin/spark-submit INFO Utils: | default | 1 | 1 | 1 | 0 || 1 | 1 | | |
| 15/11/16 11:52:16.479 redirect stderr for command ./bin/spark-submit INFO Utils: --------------------------------------------------------------------- | |
| 15/11/16 11:52:16.481 redirect stderr for command ./bin/spark-submit INFO Utils: :: retrieving :: org.apache.spark#spark-submit-parent | |
| 15/11/16 11:52:16.481 redirect stderr for command ./bin/spark-submit INFO Utils: confs: [default] | |
| 15/11/16 11:52:16.489 redirect stderr for command ./bin/spark-submit INFO Utils: 1 artifacts copied, 0 already retrieved (1kB/8ms) | |
| 15/11/16 11:52:16.496 redirect stderr for command ./bin/spark-submit INFO Utils: /home/jenkins/.ivy2/jars/my.great.lib_mylib-0.1.jar contains R source code. Now installing package. | |
| 15/11/16 11:52:16.526 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: Class path contains multiple SLF4J bindings. | |
| 15/11/16 11:52:16.526 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: Found binding in [jar:file:/home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/assembly/target/scala-2.10/spark-assembly-1.6.0-SNAPSHOT-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| 15/11/16 11:52:16.526 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: Found binding in [jar:file:/home/jenkins/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| 15/11/16 11:52:16.526 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. | |
| 15/11/16 11:52:16.531 redirect stderr for command ./bin/spark-submit INFO Utils: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] | |
| 15/11/16 11:52:16.612 redirect stderr for command ./bin/spark-submit INFO Utils: Extracting R/pkg/R/mylib.R to /home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/spark-84a8faa5-3811-45b4-adc8-ec2eaa5e45ae/R/pkg/R/mylib.R | |
| 15/11/16 11:52:16.615 redirect stderr for command ./bin/spark-submit INFO Utils: Extracting R/pkg/DESCRIPTION to /home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/spark-84a8faa5-3811-45b4-adc8-ec2eaa5e45ae/R/pkg/DESCRIPTION | |
| 15/11/16 11:52:16.615 redirect stderr for command ./bin/spark-submit INFO Utils: Extracting R/pkg/NAMESPACE to /home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/spark-84a8faa5-3811-45b4-adc8-ec2eaa5e45ae/R/pkg/NAMESPACE | |
| 15/11/16 11:52:16.617 redirect stderr for command ./bin/spark-submit INFO Utils: Building R package with the command: List(R, CMD, INSTALL, -l, /tmp/spark-ac2e1ede-ae7a-414e-b1f4-2fe4c2330aeb, /home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/spark-84a8faa5-3811-45b4-adc8-ec2eaa5e45ae/R/pkg) | |
| 15/11/16 11:52:16.812 redirect stderr for command ./bin/spark-submit INFO Utils: ERROR: dependency 'SparkR' is not available for package 'sparkPackageTest' | |
| 15/11/16 11:52:16.812 redirect stderr for command ./bin/spark-submit INFO Utils: * removing '/tmp/spark-ac2e1ede-ae7a-414e-b1f4-2fe4c2330aeb/sparkPackageTest' | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: ERROR: Failed to build R package in /home/jenkins/.ivy2/jars/my.great.lib_mylib-0.1.jar. | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: In order for Spark to build R packages that are parts of Spark Packages, there are a few | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: requirements. The R source code must be shipped in a jar, with additional Java/Scala | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: classes. The jar must be in the following format: | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: 1- The Manifest (META-INF/MANIFEST.mf) must contain the key-value: Spark-HasRPackage: true | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: 2- The standard R package layout must be preserved under R/pkg/ inside the jar. More | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: information on the standard R package layout can be found in: | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: http://cran.r-project.org/doc/contrib/Leisch-CreatingPackages.pdf | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: An example layout is given below. After running `jar tf $JAR_FILE | sort`: | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: META-INF/MANIFEST.MF | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: R/ | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: R/pkg/ | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: R/pkg/DESCRIPTION | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: R/pkg/NAMESPACE | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: R/pkg/R/ | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: R/pkg/R/myRcode.R | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: org/ | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: org/apache/ | |
| 15/11/16 11:52:16.829 redirect stderr for command ./bin/spark-submit INFO Utils: ... | |
| 15/11/16 11:52:16.859 redirect stderr for command ./bin/spark-submit INFO Utils: Main class: | |
| 15/11/16 11:52:16.859 redirect stderr for command ./bin/spark-submit INFO Utils: org.apache.spark.deploy.RRunner | |
| 15/11/16 11:52:16.859 redirect stderr for command ./bin/spark-submit INFO Utils: Arguments: | |
| 15/11/16 11:52:16.859 redirect stderr for command ./bin/spark-submit INFO Utils: file:/home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/R/pkg/inst/tests/packageInAJarTest.R | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: System properties: | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: SPARK_SUBMIT -> true | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: spark.ui.enabled -> false | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: spark.files -> file:/home/jenkins/workspace/Spark-Master-Maven-with-YARN/HADOOP_PROFILE/hadoop-2.4/label/spark-test/R/pkg/inst/tests/packageInAJarTest.R | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: spark.app.name -> testApp | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: spark.jars -> file:/home/jenkins/.ivy2/jars/my.great.lib_mylib-0.1.jar | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: spark.submit.deployMode -> client | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: spark.master -> local-cluster[2,1,1024] | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: Classpath elements: | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: /home/jenkins/.ivy2/jars/my.great.lib_mylib-0.1.jar | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: | |
| 15/11/16 11:52:16.860 redirect stderr for command ./bin/spark-submit INFO Utils: | |
| 15/11/16 11:52:17.339 ScalaTest-main-running-SparkSubmitSuite INFO SparkSubmitSuite: | |
| ===== FINISHED o.a.s.deploy.SparkSubmitSuite: 'correctly builds R packages included in a jar with --packages' ===== |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment