Repository navigation
Circular dependencies between pmd-designer, pmd-core, pmd-cli #4446
Description
Is your feature request related to a problem? Please describe.
We have a workaround for this problem (#4445), but I'd like to explore, whether we can make a real solution instead of allowing snapshot dependencies (and make the build not repeatable) for release candidates. Also, for the final release 7.0.0, we would need a pmd-designer, that depends on e.g. 7.0.0-rc3.
So, the dependencies look like this:
- pmd (that's what we release in one run)
- pmd-core
- pmd-cli
- pmd-designer
- pmd-core
- pmd-designer
- pmd-dist
- pmd-designer
- pmd-core
- pmd-designer
- pmd-designer (the designer is released separately)
- pmd-core
If we could release pmd-core separately first, then we could release pmd-ui second, and the rest of pmd as last.
pmd-core uses pmd as the parent project, so that would be needed to be released as well. Currently, we use our parent project as parent project and additionally as aggregator. That would need to change.
Describe the solution you'd like
I don't think, I have a favorite yet...
Describe alternatives you've considered
a) No change
We could just accept, that we need snapshot dependencies for the first release of a major version. However, the same problem will occur again for 8.0.0.
b) Release pmd-core first, then the rest
So, maybe we can have the following structure:
pmd/pom.xml (artifactId=pmd-aggregator): Just an aggregator to easily build everything at once.
Won't be released.
pmd/parent/pom.xml (pmd): the parent pom, without modules. So, not an aggregator anymore.
pmd/pmd-core/pom.xml (pmd-core): no changes - uses the same parent
We could then release parent and pmd-core first. That could be done by just setting the version of parent to a non-SNAPSHOT version, updating the reference in pmd-core and in other modules to the new parent. Commit this as "prepare release pmd-core x.y.z", and tag this as "pmd-core-x.y.z". Pushing that tag would build the aggregator pom, which would build all modules, while parent and core a released versions, all other are still SNAPSHOT. Not sure, how the nexus-staging-maven-plugin handles this situation (deploying a mix of SNAPSHOT and non-SNAPSHOT artifacts). If that causes a problem, we could create a separate build-script/workflow, that runs only on these tags (pmd-core-*) - since pmd-core is just a library, we don't need to do anything else (like creating release notes, etc.). The script can be very simple (just run "mvn deploy -f parent/pom.xml && mvn deploy -f pmd-core/pom.xml").
Once pmd-core is available in maven-central, we can simply update pmd-designer to use this new pmd-core version and release pmd-designer.
Once pmd-designer is available in maven-central, we can update the other modules and set everything to the same version, commit this as "prepare release x.y.z", and tag this as usual as "pmd_releases/x.y.z". Push the tag and let it build. We need to make sure, we don't build/deploy pmd-core again, as it is already in maven-central - redeploying it will definitively result in failure. The build script however would do all the magic we currently do (create documentation, upload binaries, publish release with release notes).
This as definitively an impact on reproducible builds - as reproducing a pmd release is then not anymore a single build.
What I don't like here is: it looks very complicated, we don't have a single tag, but two tags. The release process might take a while, because we need to wait between the steps.
c) Move pmd-core into its own repository
Maybe we should separate pmd-core into its own repository. That would solve the tagging issue and makes the build scripts simpler. However, it might have an impact on the developer experience - but you can e.g. import an additional module in IntelliJ, so that you have pmd-core integrated during development.
d) Move pmd-designer back into pmd
Move pmd-designer back into the main repository. That way, we have all the moving parts in one place and can simply release everything in one go. Affects issues (designer bugs are separate), raises the question, why pmd-eclipse-plugin is separate, etc.
e) Use git submodules
Alternatively we could use git submodules
f) Consider pmd-designer a separate product
- Don't include pmd-designer in pmd-dist anymore
- Don't provide a CLI command to start the designer
- The designer will need to be installed always additionally and in parallel to the PMD installation
g) pmd-designer as a separate product, soft dependency only
That kind of a compromise:
- We don't include pmd-designer in the pmd-dist anymore. That's not possible, because the designer is not available yet
- We provide a CLI command to start the designer, but that figures out at runtime, whether the designer jar is there. If so: start the designer as usual. If not: ask the user, whether to download the jar. The jar can also be placed manually into the lib folder.
- In addition to the jar, we also provide a "fat" pmd-designer binary distribution, which includes also PMD. And javafx.
h) pmd-cli/pmd-dist in separate repository
- Name of the repository: maybe "https://github.com/pmd/pmd-package"?
- pmd-designer is still separate
- Drop the statically generated shell completion for languages in the cli module (just affects the
--use-versionand--force-languageflags IIRC). The shell completion does anyway not reflect the actual modules that are present on the classpath at runtime. This removes the dependency of pmd-cli on other language modules. PMD should give a error message if a wrong language is used (e.g. "did you mean..."). - Since pmd-cli doesn't have dependencies on language modules anymore, we can move it out of the main repo together with pmd-dist. They can then depend on pmd modules and the designer freely.
Dependency hierarchy now:
- pmd (single release unit)
- pmd-core
- pmd-java
- pmd-cli
- pmd-core
- pmd-ui (references only, only old release available)
- pmd-core
- pmd-dist
- pmd-core
- pmd-cli
- pmd-ui (references only, only old release available)
- pmd-ui (released separately)
- pmd-core (references only, only old release available)
Dependency hierarchy after:
- pmd (single release unit)
- pmd-core
- pmd-java
- pmd-ui (released afterwards)
- pmd-core (can reference the just released pmd-core)
- pmd-package (final release unit)
- pmd-cli
- pmd-ui (can reference the just released pmd-ui)
- pmd-core (can reference the just released pmd-core)
- pmd-dist
- pmd-core (can reference the just released pmd-core)
- pmd-ui (can reference the just released pmd-ui)
- pmd-cli (in the same reactor of pmd-package)
- pmd-cli
Consequences:
- Releasing a new PMD version will require multiple steps. At least two steps (pmd itself and pmd-dist), if an updated pmd-designer is needed, 3 steps.
- The Release publishing step with release notes etc. will be done when pmd-package is built/released. The release of "pmd" itself will only deploy the jars into maven central. It could however prepare a draft release, to which pmd-dist can then add the assets.
- Automation: we could build a pipeline, so that the release build of pmd triggers a workflow on pmd-designer which triggers a workflow on pmd-package.
Additional context
- I think, this problem is new. We moved out the pmd-designer with 6.13.0. In PMD 6.13.0, we shipped designer 6.12.0. Since we didn't do a major release since then, we not had this problem yet.
- We once seem to had a similar problem: 6.21.0 (with designer): PMD 6.21.0 shipped designer 6.19.0. PMD 6.21.0-full is actually PMD 6.22.0-SNAPSHOT with designer 6.21.0. However, with a major release change, we can't ship PMD 7 with designer 6.x...
Activity
jsotuyod commentedon Apr 3, 2023
I seriously dislike this one… working on a major release, doing refactors to public APIs is going to be very painful, with PRs split in two and needing to be in sync. pmd-ui has suffered of this in PMD 7 development, even though it doesn't dive as deep into the PMD internals as language modules need to do. Figuring out which branch / commit is compatible with what you have adds overhead and pain.
I think this could be a viable option actually… we moved it away mostly because as a new tool it was iterating fast and unbounded by the lifecycle of PMD itself, but for the last year it has been really stable. Since May 2022, there has only been 1 set of changes that are not dependencies upgrades / compatibility changes for PMD 7, with it being bug fixes. Maybe this is a sign it doesn't require that freedom anymore.
As for PMD Eclipse, for me it's easy, as it's not accessible from the CLI.
Alternatively we could use git submodules, but I would rather not… they are cumbersome and requires extra commands along the way, it will probably cause more trouble than good…
adangel commentedon Apr 13, 2023
I've added it to the list above. It might actually be a solution, given that you always correctly issue
git submodule initetc. after a clone. The release would then happen only on the pmd repo, which includes pmd-designer.I was trying to figure out, why we have this problem with the designer, but not with the eclipse plugin. One reason is: The eclipse plugin is shipped separately. It is not included in the "standard" pmd binary distribution. So the eclipse plugin is always a separate release, which occurs after PMD. The eclipse plugin includes all the pmd libraries.
On the other side, the designer is part of the binary distribution. And hence, it needs to be released at the very instant (or before, but that's not possible because of the cyclic dependency).
Maybe it's time that we consider one more alternative: Really considering pmd-designer as a separate product. We would ship pmd binary dist without the designer, only the CLI for PMD itself and CPD and treeexporter. But not the designer. If a user wants to use the designer, it downloads it from pmd-designer/releases and that's it. That would mean: remove the CLI command for Designer. And maybe add a run-script directly in the designer. There is one advantage: We can then e.g. say that for the designer, you need at least Java 17 and we can directly ship the javafx libraries - no need to set JAVAFX_HOME etc. The downside is, that the user needs two download PMD twice: once for the CLI and again for the Designer. In that scenario, we wouldn't publish pmd-designer (or pmd-ui) into maven central, as the pmd-designer is not a library anymore. Overall, this alternative sounds for me currently the most sensible.
There is a compromise alternative to the above: We could change
DesignerCommandin that way, that we dynamically figure out (reflection, load class on demand, catch ClassNotFoundErrors, etc.) whether pmd-designer.jar/pmd-ui.jar is on the classpath. If not, we say - please download pmd-ui.jar and place it into the folder. We could also ask whether to download it for the user automatically. We don't ship the designer anymore in the pmd-dist package. We release pmd-designer in two forms: standalone fat jar (which includes pmd, maybe we should also produce a distribution binary zip file with a similar layout) and pmd-ui.jar drop-in.We would then replace the hard compile-time dependency in pmd-cli with a soft runtime-dependency (which creates its own risks).
I'll try to summarize the options in the issue description....
adangel commentedon Apr 13, 2023
One consequence of considering pmd-designer as a separate (aka standalone) product would be, that it always requires a new release, when a new PMD version is released. That's currently the case with the pmd-eclipse-plugin.
oowekyala commentedon May 29, 2023
For reference, last meeting we discussed the following solution:
--use-versionand--force-languageflags IIRC). The shell completion does anyway not reflect the actual modules that are present on the classpath at runtime. This removes the dependency of pmd-cli on other language modules.I think this is a good compromise and it would also let us release different assemblies without too much trouble. We could eg optionally package the JavaFX dependencies, or make a distribution with a specific set of languages.
FWIW I don't think we should be going the route of dowloading files at runtime like #4476 does...
adangel commentedon Jun 1, 2023
@oowekyala Thanks for updating the summary here. I've added it as the option "h)" in the issue description.
Some additional thoughts and notes from my side:
Release notes and doc will stay where they are: in pmd main repo under "/docs/"
do-release.shand the ci build scripts need to be adjustedI think, pmd-cli and pmd-dist really belong together. E.g. we have in pmd-dist the shell scripts to start the cli...
The issue, that the bash completion script won't recognize languages, that are added at runtime is IMHO a separate (additional) problem. The problem here is that pmd-cli needs pmd-ui. IMHO, removing language dependencies from pmd-cli should be done in a separate step. That pmd-cli depends on all languages (as does pmd-dist) doesn't hurt atm.
The artifacts of pmd-package/pmd-dist don't need to be in maven central (at the moment, the binary distribution are not in central). I think, we should keep it that way.
I like the idea, that this opens up the possibility to build different flavors of pmd's binary distribution, maybe one even with a Java Runtime or an installer. That's however still up in the air.
I think, we should really do this already for PMD 7 - as this currently is hurting us. I don't like to ship an old pmd-designer, when we actually would require a new version.
Yes, that was just an experiment, whether this would be technically possible.
jsotuyod commentedon Jun 1, 2023
So, if I understand correctly, the idea would be to split a release in 3 separate steps (with their own repos) with the given order:
I acknowledge this would work and solve most of our issues. The APIs being used by pmd-cli are really limited and are effectively public APIs meant for programmatic integration that should be stable through the major release cycle. Also pmd-cli is unlikely to change often.
I do agree with @adangel, this is independent on wether we do language completion on cli or not. It's also worth noting that removing the dependency from pmd-cli to the languages would prevent autocomplete for other features in the pipeline such as #2947 which I feel will be significantly harder to consume / discover if relying exclusively on people digging into the language-specific documentation.
Moving the completion to pmd-dist is doable. It does require to leak some implementation details (ie: the internal root command class name), but nothing unmanageable, specially if cli and dist are part of the same repo. I feel this is a better compromise than outright removing it.
Samewise, I'd notice that building multiple distributions with different subsets of languages, shipping JavaFX platform specific dependencies, or a selfcontained JRE would all still be plausible regardless of where pmd-dist lives. We could actually be doing that right now if we wanted.
Overall, I'd be ok with moving down this path, even if it is still not my preferred one (that being option d).
oowekyala commentedon Jun 1, 2023
Yeah I think you're right, the main problem we are trying to solve here is the pmd-designer dependency.
Very true.
About completion, maybe a fix to the "freshness" problem of the generated completions would be to add the ability to regenerate the completion scripts at runtime based on the current classpath. I know some other programs I use do that, eg just: you run eg
just --completions zsh > somewhere-in-your-plugin-path. I'm not sure what dependencies that would bring in... but picocli is very small anyway. With such a feature it doesn't matter whether the completions are generated in pmd-dist or pmd-cli. Maybe it becomes useless to generate the completions at build time altogether.jsotuyod commentedon Jun 1, 2023
That is actually an interesting option. We don't need any additional dependencies, this is done by the very same picocli jar we already use to actually run…
There is already a
picocli.AutoComplete.GenerateCompletioncommand we can hook up as per https://picocli.info/autocomplete.html#_distributionadangel commentedon Sep 28, 2023
I'd like to tackle this for the upcoming rc4 release. I have in mind to do a very, very low impact change only - that means: not moving any projects around into new repositories etc.
My plan is - which I want to try out with rc4 - the following:
do-release.shto release all modules except pmd-cli and pmd-dist. Probably by moving these modules into a profile, that is not active during releasing.mvn deploy- meaning the artifacts are deployed to maven centralfatshaded jar that is attached to the release.do-release.shto release only modules pmd-cli and pmd-dist. Probably we need to create a second tag (e.g. pmd_releases/7.0.0-rc4-dist).The main goal is:
No goals for the first attempt:
[-]Circular dependencies between pmd-ui, pmd-core, pmd-cli[/-][+]Circular dependencies between pmd-designer, pmd-core, pmd-cli[/+][ci] publish-release.yml: simplify
UncleOwen commentedon Apr 17, 2026
How exactly does pmd-dist depend on pmd-designer? I see the dependency in pom.xml, but nothing else? If I grep for 'fxdesigner', I only get commented out strings in scripts/pmd and scripts/pmd.bat
As for pmd-cli: Maybe we could get by with some reflection trickery? That would turn the hard dependency into a soft one.
UncleOwen commentedon Apr 17, 2026
Ah, simply by depending it somehow ends up in the released .zip file. So I guess that's my (very late) vote for g)?