Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
3dcd569
edges sampling API (scala)
SemyonSinchenko Oct 16, 2025
06f6af6
add seed to z-estimation
SemyonSinchenko Oct 16, 2025
e4ede62
wip
SemyonSinchenko Oct 27, 2025
08ee0eb
Merge remote-tracking branch 'graphframes/main' into 726-sampling-api
SemyonSinchenko Oct 30, 2025
d050a7a
WIP
SemyonSinchenko Nov 20, 2025
4a0a14f
scalfix
SemyonSinchenko Nov 21, 2025
579a5c3
docstrings to RandomWalkBase and RandomWalkWithRestart
SemyonSinchenko Dec 1, 2025
16dd46c
Fix RandomWalk implementation bugs and add example
SemyonSinchenko Dec 2, 2025
965dea8
Add Word2VecHashingTrick implementation for graph embeddings
SemyonSinchenko Dec 3, 2025
3260cc9
Implement reservoir sampling for neighbor selection in random walks
SemyonSinchenko Dec 4, 2025
aafcbc6
fix scalastyle?
SemyonSinchenko Dec 5, 2025
f27462d
fix reservoir
SemyonSinchenko Dec 5, 2025
28b8ca5
add hash2vec
SemyonSinchenko Dec 8, 2025
b8255ac
fixes
SemyonSinchenko Dec 9, 2025
2113376
docstrings + scalfix
SemyonSinchenko Dec 9, 2025
6244ec2
remove sampling as not needed
SemyonSinchenko Dec 9, 2025
a9ce665
fixes in build and code
SemyonSinchenko Dec 10, 2025
0d5da92
Big update
SemyonSinchenko Dec 30, 2025
84306b6
Tests and updates
SemyonSinchenko Dec 30, 2025
5084d88
workaround scala 2.13 deprecation of Searching.search
SemyonSinchenko Dec 30, 2025
0a9dc90
Fixes
SemyonSinchenko Dec 30, 2025
f7a9aeb
Fix access
SemyonSinchenko Dec 30, 2025
acc5a2c
fallback to java Serialization
SemyonSinchenko Dec 30, 2025
8b20f00
Fix some bugs
SemyonSinchenko Dec 30, 2025
538a72d
Sampling Convolution tests and docstrings
SemyonSinchenko Dec 31, 2025
3548bb1
docstrings for RW Emebddings
SemyonSinchenko Dec 31, 2025
3df9282
Python API
SemyonSinchenko Jan 2, 2026
af97cac
hash2vec tests
SemyonSinchenko Jan 2, 2026
5d724b5
Explicit types
SemyonSinchenko Jan 2, 2026
526330c
hash2vec and random walks with restart tests
SemyonSinchenko Jan 2, 2026
061fab3
Merge remote-tracking branch 'graphframes/main' into 726-sampling-api
SemyonSinchenko Jan 2, 2026
7e77bc9
performance
SemyonSinchenko Jan 7, 2026
02c22f8
ignore unused nowarn
SemyonSinchenko Jan 7, 2026
97d529a
performance + cached walks support + continous mode
SemyonSinchenko Jan 7, 2026
cc7f908
Merge remote-tracking branch 'graphframes/main' into 726-sampling-api
SemyonSinchenko Jan 7, 2026
48e981a
fix rw and update the branch
SemyonSinchenko Jan 7, 2026
de63a45
fix
SemyonSinchenko Jan 7, 2026
e1cf2a4
protobuf & connect
SemyonSinchenko Jan 7, 2026
d2780a4
public API for embeddings and small refactoring of methods
SemyonSinchenko Jan 7, 2026
a294c03
initial Py API for embeddings
SemyonSinchenko Jan 9, 2026
60ca7e5
fix
SemyonSinchenko Jan 9, 2026
6583c20
tests + docs
SemyonSinchenko Jan 19, 2026
54ee3d7
fix connect tests
SemyonSinchenko Jan 19, 2026
867ae47
Merge remote-tracking branch 'graphframes/main' into 726-sampling-api
SemyonSinchenko Feb 11, 2026
c7ef2d9
decrease GC pressure
SemyonSinchenko Feb 12, 2026
ff107a2
further optimizations
SemyonSinchenko Feb 12, 2026
7f01b9c
refactor: optimize Hash2Vec string hashing and partitioning logic
SemyonSinchenko Feb 12, 2026
b7d95f8
refactor: replace generic hash function with type-specific implementa…
SemyonSinchenko Feb 12, 2026
5def505
refactor: simplify hash function logic and improve performance in Has…
SemyonSinchenko Feb 12, 2026
5f27b92
refactor: inline generic processPartitionGeneric into specialized Str…
SemyonSinchenko Feb 12, 2026
ed07856
test: add tests for PagedMatrixDouble helper covering page extension,…
SemyonSinchenko Feb 12, 2026
5cc9f1a
refactor: replace unsafe hash functions with MurmurHash3 and optimize…
SemyonSinchenko Feb 12, 2026
895a319
refactor: update processStringPartition to use PagedMatrixDouble for …
SemyonSinchenko Feb 12, 2026
66809bd
docs: add internal documentation for PagedMatrixDouble explaining mem…
SemyonSinchenko Feb 12, 2026
548a17a
chore: remove commented helper section from Hash2Vec
SemyonSinchenko Feb 12, 2026
70d96cb
refactor: replace case class with class for PagedMatrixDouble and rem…
SemyonSinchenko Feb 12, 2026
842cb7e
fix: correct LongMap type parameter in Hash2Vec vocabIndex initializa…
SemyonSinchenko Feb 12, 2026
c2c6ff4
refactor: reduce PAGE_BITS from 16 to 12 and update related constants…
SemyonSinchenko Feb 12, 2026
54c32b3
feat: add max vectors per partition limit and batched processing for …
SemyonSinchenko Feb 12, 2026
e689b86
refactor: process long partitions in batches respecting max vectors l…
SemyonSinchenko Feb 12, 2026
6d92b2c
test: add Hash2Vec tests for co-occurrence patterns and cosine simila…
SemyonSinchenko Feb 12, 2026
ae60afd
chore: clean up code formatting and improve test readability in Hash2Vec
SemyonSinchenko Feb 12, 2026
c0f3ca3
fix: correct typo in error message from 'gor' to 'got' in Hash2Vec ex…
SemyonSinchenko Feb 12, 2026
b0dc568
docs: add docstrings for Hash2Vec setters setDoNormalization and setM…
SemyonSinchenko Feb 12, 2026
95d73c3
fix: correct typo in NOTICE and KMinSampling, update Hash2Vec default…
SemyonSinchenko Feb 12, 2026
de2d6a3
fix: skip seeds for previous batches to maintain consistency when sta…
SemyonSinchenko Feb 18, 2026
3a0a302
fix: add overwrite mode when writing batch results to allow re-runnin…
SemyonSinchenko Feb 18, 2026
af7f8f4
feat: add cleanUp method to remove temporary files for a walk ID usin…
SemyonSinchenko Feb 18, 2026
579a2b5
docs: improve documentation for cleanUp method in RandomWalkBase trait
SemyonSinchenko Feb 18, 2026
5e2a462
test: add cleanUp call in RandomWalkWithRestart test
SemyonSinchenko Feb 18, 2026
d3cb1a2
test: move walks execution inside try block in RandomWalkWithRestartS…
SemyonSinchenko Feb 18, 2026
f5bb5ca
refactor: set walkID default to UUID and remove redundant runID variable
SemyonSinchenko Feb 18, 2026
4b25649
docs: update comment to clarify walkID retrieval method behavior
SemyonSinchenko Feb 18, 2026
0c9262d
refactor: rename walkID to runID for clarity in random walk operations
SemyonSinchenko Feb 18, 2026
c3b8d86
refactor: remove runId parameter from cleanUp method in RandomWalkBase
SemyonSinchenko Feb 18, 2026
3309269
refactor: move cleanUp method to companion object with parameters and…
SemyonSinchenko Feb 18, 2026
a9ff184
refactor: improve documentation formatting and remove redundant log s…
SemyonSinchenko Feb 18, 2026
9eb4dea
test: verify temporary files are deleted after RandomWalkWithRestart …
SemyonSinchenko Feb 18, 2026
5b349c8
refactor: use numBatches variable and fix run path in RandomWalkWithR…
SemyonSinchenko Feb 18, 2026
5c12113
test: add test for RandomWalkWithRestart resuming from middle iteration
SemyonSinchenko Feb 18, 2026
483c3be
style: Format RandomWalkWithRestartSuite with consistent spacing and …
SemyonSinchenko Feb 18, 2026
5efc5dd
refactor: use getSeq instead of getAs[Seq[String]] in RandomWalkWithR…
SemyonSinchenko Feb 18, 2026
001a2bf
fix: correct typo in error message and parameter name for gaussian sigma
SemyonSinchenko Feb 26, 2026
64aee65
feat: add clean-up option for temporary random walk files in embeddin…
SemyonSinchenko Feb 26, 2026
e45381d
docs: improve formatting and consistency in graph-ml documentation ta…
SemyonSinchenko Feb 26, 2026
d4f9f50
feat: add clean_up_after_run field to RandomWalkEmbeddings proto defi…
SemyonSinchenko Feb 26, 2026
79be29d
chore: add clean_up_after_run parameter to _RandomWalksEmbeddingsPara…
SemyonSinchenko Feb 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add seed to z-estimation
  • Loading branch information
SemyonSinchenko committed Oct 16, 2025
commit 06f6af683ced9a9f87e42d2fee367f4f062be6f7
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,13 @@ abstract class EdgesSamplingStrategy extends Serializable with WithIntermediateS
* @return
* the estimated normalization factor, with a minimum value to avoid division by zero
*/
protected def estimateNormalizeFactor(edges: DataFrame, sampleSize: Double): Double = {
protected def estimateNormalizeFactor(
edges: DataFrame,
sampleSize: Double,
seed: Long): Double = {
val estimatedSum = edges
.select(edgeKeepProbability)
.sample(sampleSize)
.sample(sampleSize, seed)
.select(sum(edgeKeepProbability))
.head()
.getDouble(0)
Expand Down Expand Up @@ -185,7 +188,7 @@ abstract class EdgesSamplingStrategy extends Serializable with WithIntermediateS
case None => None
}))

val zFactor = estimateNormalizeFactor(triplets, zSampleSize)
val zFactor = estimateNormalizeFactor(triplets, zSampleSize, seed)

val probs = triplets
.withColumn(
Expand Down