Skip to content

feat: initial commit with a copy of GraphX code #680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SemyonSinchenko merged 2 commits into from
Sep 12, 2025
Merged

feat: initial commit with a copy of GraphX code #680

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1035724
Initial commit with a copy of GraphX code
SemyonSinchenko Sep 11, 2025
f22f34f
fix python tests
SemyonSinchenko Sep 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
fix python tests
  • Loading branch information
@SemyonSinchenko
SemyonSinchenko committed Sep 11, 2025
commit f22f34f0d2088caa4068483c5d3e4014da76b131
2 changes: 2 additions & 0 deletions python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ pip install graphframes-py

**NOTE!** *Python distribution does not include JVM-core. You need to add it to your cluster or Spark-Connect server!*

**NOTE!** *GraphFrames depends on its own version of GraphX. While in case of installation from the Maven Central repository, all the runtime dependencies will be resolved automatically, you may need to add them manually in case of installation from the local repository.*

## Running `graphframes-py`

You should use GraphFrames via the `--packages` argument to `pyspark` or `spark-submit`, but this package is helpful in development environments.
Expand Down
19 changes: 15 additions & 4 deletions python/tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,20 +22,31 @@ def is_remote() -> bool:
scala_version = os.environ.get("SCALA_VERSION", "2.12" if __version__ < "4" else "2.13")


def get_gf_jar_locations() -> Tuple[str, str]:
def get_gf_jar_locations() -> Tuple[str, str, str]:
"""
Returns a location of the GraphFrames JAR and GraphFrames Connect JAR.

In the case your version of PySpark is not compatible with the version of GraphFrames,
this function will raise an exception!
"""
project_root = pathlib.Path(__file__).parent.parent.parent
graphx_dir = project_root / "graphx" / "target" / f"scala-{scala_version}"
core_dir = project_root / "core" / "target" / f"scala-{scala_version}"
connect_dir = project_root / "connect" / "target" / f"scala-{scala_version}"

graphx_jar: Optional[str] = None
core_jar: Optional[str] = None
connect_jar: Optional[str] = None

for pp in graphx_dir.glob(f"graphframes-graphx-spark{spark_major_version}*.jar"):
assert isinstance(pp, pathlib.PosixPath) # type checking
graphx_jar = str(pp.absolute())

if graphx_jar is None:
raise ValueError(
f"Failed to find graphframes jar for Spark {spark_major_version} in {graphx_dir}"
)

for pp in core_dir.glob(f"graphframes-spark{spark_major_version}*.jar"):
assert isinstance(pp, pathlib.PosixPath) # type checking
core_jar = str(pp.absolute())
Expand All @@ -54,22 +65,22 @@ def get_gf_jar_locations() -> Tuple[str, str]:
f"Failed to find graphframes connect jar for Spark {spark_major_version} in {connect_dir}"
)

return (core_jar, connect_jar)
return core_jar, connect_jar, graphx_jar


@pytest.fixture(scope="module")
def spark():
warnings.filterwarnings("ignore", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning)

(core_jar, connect_jar) = get_gf_jar_locations()
(core_jar, connect_jar, graphx_jar) = get_gf_jar_locations()

with tempfile.TemporaryDirectory() as tmp_dir:
builder = (SparkSession.Builder()
.appName("GraphFramesTest")
.config("spark.sql.shuffle.partitions", 4)
.config("spark.checkpoint.dir", tmp_dir)
.config("spark.jars", f"{core_jar},{connect_jar}")
.config("spark.jars", f"{core_jar},{connect_jar},{graphx_jar}")
)

if spark_major_version == "3":
Expand Down