Skip to content

chore: fix shading problems and slightly update CI #615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SemyonSinchenko merged 6 commits into from
Jul 2, 2025
Merged

chore: fix shading problems and slightly update CI #615

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8e96e6c
**Update Scala CI workflows and build configurations**
SemyonSinchenko Jun 30, 2025
1ba0b2e
Add missing developer email
SemyonSinchenko Jun 30, 2025
0975916
Specify the scope for protobuf-java
SemyonSinchenko Jul 2, 2025
6ddd8c7
Merge remote-tracking branch 'graphframes/master' into 614-sbt-shadin…
SemyonSinchenko Jul 2, 2025
780d8f7
Take everything from #617
SemyonSinchenko Jul 2, 2025
e4c6221
main -> master
SemyonSinchenko Jul 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
**Update Scala CI workflows and build configurations** 
- Refactor `scala-publish.yml` to clarify release and snapshot publishing conditions.
- Adjust `docs.yml` trigger to specifically include the `main` branch.
- Remove unused Sonatype import from `build.sbt`.
- Enhance developer metadata and maintainers list in `build.sbt`.
- Update dependencies and assembly configuration to address shading and exclude non-connect classes for the Uber JAR.
- Introduce custom POM post-processing for correct dependency scope adjustments.
  • Loading branch information
@SemyonSinchenko
SemyonSinchenko committed Jun 30, 2025
commit 8e96e6cd6efdca86f69766e5ad4dd3f8ef7ee103
3 changes: 2 additions & 1 deletion .github/workflows/docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ name: Deploy Docs

on:
push:
branches: [$default-branch]
branches:
- main
workflow_dispatch:

permissions:
Expand Down
9 changes: 7 additions & 2 deletions .github/workflows/scala-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ on:
tags:
- "*.*.*"

# In the case of pushing the tag, it will publish the release.
# In the case of pushing to the main, it will publish the SNAPSHOT

# From the docs (https://github.com/sbt/sbt-ci-release):
# - git tag pushes are published as regular releases to Maven Central
# - merge into main commits are published as -SNAPSHOT with a unique version number for every commit

jobs:
release:
strategy:
Expand All @@ -25,14 +32,12 @@ jobs:
fetch-depth: 0 # sbt-git requires the full history to determine the version

- name: Set up JDK
if: startsWith(github.ref, 'refs/tags/')
Comment thread
SemyonSinchenko marked this conversation as resolved.
uses: actions/setup-java@v4
with:
java-version: "${{ matrix.java-version }}"
distribution: 'zulu'

- name: Run the release of core
if: startsWith(github.ref, 'refs/tags/')
run: build/sbt -Dspark.version=${{ matrix.spark-version }} +ci-release "project connect" +ci-release
env:
PGP_PASSPHRASE: ${{ secrets.PGP_PASSPHRASE }}
Expand Down
57 changes: 44 additions & 13 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import xerial.sbt.Sonatype.sonatypeCentralHost

lazy val sparkVer = sys.props.getOrElse("spark.version", "3.5.5")
lazy val sparkMajorVer = sparkVer.substring(0, 1)
lazy val sparkBranch = sparkVer.substring(0, 3)
Expand Down Expand Up @@ -38,6 +36,7 @@ ThisBuild / scmInfo := Some(
ScmInfo(
url("https://github.com/graphframes/graphframes"),
"scm:git@github.com:graphframes/graphframes.git"))
// The list of active maintainers with Write/Maintain/Admin access
ThisBuild / developers := List(
Developer(
id = "rjurney",
Expand All @@ -48,7 +47,12 @@ ThisBuild / developers := List(
id = "SemyonSinchenko",
name = "Sem",
email = "ssinchenko@apache.org",
url = url("https://github.com/SemyonSinchenko")))
url = url("https://github.com/SemyonSinchenko")),
Developer(
id = "james-willis",
name = "James Willis",
email = "???",
Comment thread
SemyonSinchenko marked this conversation as resolved.
Outdated
url = url("https://github.com/james-willis")))
ThisBuild / sonatypeCredentialHost := "s01.oss.sonatype.org"
ThisBuild / sonatypeRepository := "https://s01.oss.sonatype.org/service/local"
ThisBuild / sonatypeProfileName := "io.graphframes"
Expand Down Expand Up @@ -113,7 +117,6 @@ lazy val root = (project in file("."))
Global / concurrentRestrictions := Seq(Tags.limitAll(1)),
autoAPIMappings := true,
coverageHighlighting := false,

Compile / unmanagedSourceDirectories += (Compile / baseDirectory).value / "src" / "main" / s"scala-spark-$sparkMajorVer",

// Assembly settings
Expand All @@ -134,7 +137,7 @@ lazy val root = (project in file("."))
Compile / packageSrc / publishArtifact := true)

lazy val connect = (project in file("graphframes-connect"))
.dependsOn(root)
.dependsOn(root % "provided") // It will be provided in the final POM
Comment thread
SemyonSinchenko marked this conversation as resolved.
Outdated
.settings(
commonSetting,
name := s"graphframes-connect",
Expand All @@ -152,17 +155,45 @@ lazy val connect = (project in file("graphframes-connect"))
assembly / assemblyShadeRules := Seq(
ShadeRule.rename("com.google.protobuf.**" -> protobufShadingPattern).inAll),
assembly / assemblyMergeStrategy := {
case PathList("google", "protobuf", xs @ _*) => MergeStrategy.discard
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x if x.endsWith("module-info.class") => MergeStrategy.discard
case x => MergeStrategy.first
case _ => MergeStrategy.first
},
assembly / assemblyExcludedJars := (Compile / fullClasspath).value.filter { className =>
className.data
.getName()
.contains("scala-library-") || className.data
.getName()
.contains("slf4j-api-")
// I have zero ideas how to apply shading and exclude everything except connect classes in the right way.
// This looks terrible, but it works:
assembly / assemblyExcludedJars := (Compile / fullClasspath).value.filterNot { className =>
Comment thread
SemyonSinchenko marked this conversation as resolved.
Outdated
className.data.getName.contains("GraphFramesConnect") || className.data.getPath.contains(
"graphframes/connect/proto")
},
Comment thread
SemyonSinchenko marked this conversation as resolved.
Outdated
// This magic is required to have graphframes-core as a runtime dependency
// at the same time depends on it as provided to avoid packing it to the Uber JAR we are publishing
pomPostProcess := {
Comment thread
SemyonSinchenko marked this conversation as resolved.
Outdated
val rootModuleName = (root / moduleName).value
node =>
import scala.xml._
import scala.xml.transform._

val rewriteRule = new RewriteRule {
override def transform(n: Node): Seq[Node] = n match {
case e: Elem
if e.label == "dependency" && (e \ "artifactId").text.contains(rootModuleName) =>
val scope = e \ "scope"
if (scope.text == "provided") {
val children = e.child.filter(_.label != "scope") ++ <scope>runtime</scope>
Elem(
e.prefix,
e.label,
e.attributes,
e.scope,
minimizeEmpty = false,
children: _*)
} else {
e
}
case _ => n
}
}
new RuleTransformer(rewriteRule).transform(node).head
},
publish / skip := false,
Compile / packageBin := assembly.value,
Expand Down