Skip to content

1 of 3: Build a graphframes Python package during the build process #512

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rjurney merged 43 commits into from
Feb 20, 2025
Merged

1 of 3: Build a graphframes Python package during the build process #512

Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
f4e9cdb
Converted tests to pytest. Build a Python package. Update requirement…
rjurney Feb 16, 2025
c256244
Restore Python .gitignore
rjurney Feb 16, 2025
6c3df0b
Extra newline removed
rjurney Feb 16, 2025
b2838d2
Merge branch 'master' of github.com:graphframes/graphframes into rjur…
rjurney Feb 16, 2025
caf5091
Added VERSION file set to 0.8.5
rjurney Feb 16, 2025
7cfa2d1
isort; fiex edgesDF variable name.
rjurney Feb 16, 2025
2ca9a15
Merge branch 'master' of github.com:graphframes/graphframes into rjur…
rjurney Feb 16, 2025
a8bf0be
Back out Dockerfile changes
rjurney Feb 16, 2025
54a942d
Back out version change in build.sbt
rjurney Feb 16, 2025
8b0e346
Backout changes to config and run-tests
rjurney Feb 16, 2025
46c2b93
Back out pytest conversion
rjurney Feb 16, 2025
18b5da0
Back out version changes to make nose tests pass
rjurney Feb 16, 2025
8eca097
Remove changes to requirements
rjurney Feb 16, 2025
277c06f
Put nose back in requirements.txt
rjurney Feb 16, 2025
b55ee48
Remove version bump to version.sbt
rjurney Feb 16, 2025
f8a8fd9
Remove packages related to testing
rjurney Feb 16, 2025
bc2cb36
Remove old setup.py / setup.cfg
rjurney Feb 16, 2025
728be33
New pyproject.toml and poetry.lock
rjurney Feb 16, 2025
3cea1a8
Short README for Python package, poetry won't allow a ../README.md path
rjurney Feb 16, 2025
87cc975
Remove requirements files in favor of pyproject.toml
rjurney Feb 16, 2025
6f84a5a
Try to poetrize CI build
rjurney Feb 16, 2025
9a8eef0
pyspark min 3.4
rjurney Feb 16, 2025
75ecd99
Local python README in pyproject.toml
rjurney Feb 16, 2025
80231d0
Trying to remove he working folder to debug scala issue
rjurney Feb 16, 2025
2a9170b
Set Python working directory again
rjurney Feb 16, 2025
3de2263
Accidental newline
rjurney Feb 16, 2025
4662717
Install Python for test...
rjurney Feb 17, 2025
1b7b9f8
Run tests from python/ folder
rjurney Feb 17, 2025
58da493
Try running tests from python/
rjurney Feb 17, 2025
9f4aa24
poetry run the unit tests
rjurney Feb 17, 2025
11b2782
poetry run the tests
rjurney Feb 17, 2025
9772344
Try just using 'python' instead of a path
rjurney Feb 17, 2025
d55dbfe
poetry run the last line, graphframes.main
rjurney Feb 17, 2025
2fc4d08
Remove test/ folder from style paths, it doesn't exist
rjurney Feb 17, 2025
8297a13
Remove .vscode
rjurney Feb 17, 2025
2035d98
VERSION back to 0.8.4
rjurney Feb 17, 2025
f9f4bd7
Remove tutorials reference
rjurney Feb 17, 2025
9ddd6b2
VERSION is a Python thing, it belongs in python/
rjurney Feb 17, 2025
7065647
Include the README.md and LICENSE in the Python package
rjurney Feb 17, 2025
a6c7e91
Some classifiers for pyproject.toml
rjurney Feb 17, 2025
51e3e6d
Trying poetry install action instead of manual install
rjurney Feb 17, 2025
272be06
Removing SPARK_HOME
rjurney Feb 17, 2025
4587999
Returned SPARK_HOME settings
rjurney Feb 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
poetry run the unit tests
  • Loading branch information
@rjurney
rjurney committed Feb 17, 2025
commit 9f4aa24e6d77ccb45c42b4ea8bf02b1905e826a1
2 changes: 1 addition & 1 deletion .github/workflows/python-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,5 @@ jobs:
- name: Test
working-directory: ./python
run: |
export SPARK_HOME=$(python -c "import os; from importlib.util import find_spec; print(os.path.join(os.path.dirname(find_spec('pyspark').origin)))")
export SPARK_HOME=$(poetry run python -c "import os; from importlib.util import find_spec; spec = find_spec('pyspark'); print(os.path.join(os.path.dirname(spec.origin)))")

@SemyonSinchenko SemyonSinchenko Feb 17, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need it? Tests will work even without SPARK_HOME

./run-tests.sh