Skip to content
This repository was archived by the owner on Nov 28, 2025. It is now read-only.

Support scala 2.12.10 and spark 2.4.4 #155

Merged
jhseu merged 2 commits into from
May 5, 2020
Merged

Support scala 2.12.10 and spark 2.4.4 #155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ffb7264
Support scala 2.12.10 and spark 2.4.4
vikatskhay Apr 24, 2020
aa33e9c
Merge branch 'master' into support-scala-2.12-spark-2.4
jhseu May 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions spark/spark-tensorflow-connector/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.tensorflow</groupId>
<artifactId>spark-tensorflow-connector_2.11</artifactId>
<artifactId>spark-tensorflow-connector_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<version>1.10.0</version>
<name>spark-tensorflow-connector</name>
Expand All @@ -27,13 +27,14 @@

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<scala.maven.version>3.2.2</scala.maven.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.maven.version>4.3.0</scala.maven.version>
<scala.version>2.12.10</scala.version>
<scala.binary.version>2.12</scala.binary.version>
<scalatest.maven.version>1.0</scalatest.maven.version>
<scala.test.version>2.2.6</scala.test.version>
<scala.test.version>3.0.8</scala.test.version>
<maven.compiler.version>3.0</maven.compiler.version>
<java.version>1.8</java.version>
<spark.version>2.3.1</spark.version>
<spark.version>2.4.4</spark.version>
<yarn.api.version>2.7.3</yarn.api.version>
<junit.version>4.11</junit.version>
</properties>
Expand Down Expand Up @@ -86,7 +87,7 @@
<configuration>
<recompileMode>incremental</recompileMode>
<useZincServer>true</useZincServer>
<scalaVersion>${scala.binary.version}</scalaVersion>
<scalaVersion>${scala.version}</scalaVersion>
<checkMultipleScalaVersions>false</checkMultipleScalaVersions>
</configuration>
</plugin>
Expand Down
4 changes: 2 additions & 2 deletions ...w-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/DefaultSource.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ class DefaultSource extends DataSourceRegister
}
}

object DefaultSource {
object DefaultSource extends scala.Serializable {
Copy link
Contributor Author

@vikatskhay vikatskhay Apr 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/tensorflow/ecosystem/pull/141/files#r332816724

// The function run on each worker.
// Writes the partition to a file and returns the number of records output.
private def writePartitionLocal(
Expand All @@ -189,7 +189,7 @@ object DefaultSource {
// Make the directory if it does not exist
dir.mkdirs()
// The path to the partition file.
val filePath = localPath + s"/part-" + String.format("%05d", new java.lang.Integer(index))
val filePath = localPath + s"/part-" + String.format("%05d", java.lang.Integer.valueOf(index))
Copy link
Contributor Author

@vikatskhay vikatskhay Apr 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constructor of Integer is deprecated, with -Xfatal-warnings the build fails.

val fos = new DataOutputStream(new FileOutputStream(filePath))
var count = 0
try {
Expand Down