Skip to content

Python rewrite - #4

Merged
Qonfused merged 249 commits into
mainfrom
python-rewrite
Jul 28, 2023
Merged

Python rewrite#4
Qonfused merged 249 commits into
mainfrom
python-rewrite

Conversation

@Qonfused

@Qonfused Qonfused commented Jun 20, 2023 •
edited
Loading

Copy link
Copy Markdown
Owner

Test coverage:

Tasks:

CI (Python)

  • Handle semver versioning schemes
    • Support automatic versioning schemes (e.g. via git tags)
  • Add environment checks for future versions of Python.
  • Ensure namespace and module resolution are not mangled
    • Automatically generate API exports for codegen
  • Sort import order for packages and modules

CLI (Python)

  • Add CLI support for --version flag
  • Add shared context for verbosity and debug flags
  • Add shared context for config and build file paths
  • Commands
    • Build command
    • Lock command
    • Patch command

Library (C++, Python)

  • Errors
    • Traceback frame introspection and suppression
    • Custom error classes (runtime/tests)
    • Runtime validation methods
  • Filesystem
    • Handle extracting remote archives to temporary directories
    • Expose missing POSIX filesystem operations
  • Parsers
    • Tokenizer/AST
      • (Native)
      • Stringified (asserting all values as string)
      • Flat (stringifying all nested keys)
    • Plist - Apple PLIST 1.0
      • Dictionary (root)
      • Array (root)
    • YAML - YAML 1.2
      • (Native)
      • Annotated (see example config.yml)
      • Stringified (asserting all values as string)
      • Flat (stringifying all nested keys)
  • Pipeline
    • Build file/configuration
      • Read and parse build file
      • Normalize entries
      • Validate build file against schema
      • Generate a template build file from an existing EFI structure
    • Config
      • Read and parse config file
      • Parse schema from Sample.plist/config.plist file
        • Validate config.plist against a schema
      • Merge and patch config.plist
        • Handle config inheritance
    • Kexts
      • Extract kexts from archives
      • Read and parse kext manifest
      • Validate kext manifest against build configuration
      • Sort kext load order by OSBundleLibraries field
    • Lockfile
      • Read and parse lockfile
      • Handle parsing build configuration specifiers
      • Handle creating or updating the lockfile
      • Validate lockfile against build configuration
      • Cache package sources and checksums
      • Validate lockfile updates against cached packages
    • OpenCore
      • Read and parse the OpenCore (build) configuration
      • Unpack the OpenCore archive to a temporary directory
      • Extract bundled OpenCore binaries, scripts + documentation
    • SSDTs
      • Extract and wrap iASL binary
      • Handle decompilation of SSDT binaries
      • Handle patching of SSDT source files
      • Handle compilation of SSDT source files
      • Sort SSDTs by resolving symbolic references
  • Sources
    • Handle remote sources
      • GitHub
        • Commits - raw + tarball sources
        • CI/CD - GitHub Actions
        • Releases
      • Dortania build repo
      • Handle arbitrary archives/URLs
    • Handle local sources
      • Filesystem - Raw path and RFC 8089 file: protocol
      • Local cache
  • Versioning

Qonfused added 16 commits June 21, 2023 16:10
@Qonfused
Update dict parser typing
10a7764
@Qonfused
plist parser: Soften edge cases for file streaming
243cba8
@Qonfused
Parsers: Fix codestyle consistency
3235b5d
@Qonfused
Ignore local test file (codegen)
8e153eb
@Qonfused
Add filesystem methods for archive formats
e63e3ca
@Qonfused
Add versioning methods for version management
d566a77
@Qonfused
Add constants file
10a5764
@Qonfused
filesystem: Fix extension extraction for automatic format detection
07227fe
@Qonfused
constants: Fix constant naming
dbca4fc
@Qonfused
versioning: Make compare method public
5167907
@Qonfused
Extend filesystem methods, move archive methods to separate archive…
6d1df8a 
… file
@Qonfused
Add build dir to gitignore
fd2a11e
@Qonfused
filesystem: refactor, add rename method
088a38d
@Qonfused
archives: Add opencore archive method
f87d9ad
@Qonfused
archives: Perform left merge of OCBinaryData resources
f3b681a
@Qonfused
parsers: Add delimiter arg to flatten_dict method
d20e5ab
Qonfused added 17 commits July 25, 2023 18:14
@Qonfused
[pipeline/schema] Fix value extraction fallback
459f59a
@Qonfused
[pipeline/schema] Improve parser code quality
990490b
@Qonfused
[parsers/schema, pipeline/config] Migrate parsing logic and remote so…
abf8070 
…urces
@Qonfused
[ocebuild_cli/commands/patch] Fetch schema based on resolved OpenCore…
3344734 
… version

In case no lockfile is provided or no OpenCorePkg dependency is found, the latest version of the Sample.plist schema will be used.
@Qonfused
[ci:python] Default to pytest sort order rather than by filename
ad31e98
@Qonfused
Revert change
d07a5e6
@Qonfused
Add missing docstrings to pipeline methods and new schema module
6aaefea
@Qonfused
[parsers/schema] Add support for outputting cleaned LaTeX entries for…
86f28c4 
… each key
@Qonfused
[parsers/schema] Add markdown parsers for generating reference docs
bd0c55e
@Qonfused
[parsers/schema] Fix regression in boundary logic for resetting cursor
cea6c55
@Qonfused
[parsers/schema] Add additional boundary conditions and checks
64c42a9
@Qonfused
Move project examples to new examples/ directory
3f6a7e4
@Qonfused
[docs/schema] Add docsgen for latest config.plist schema
9cec116
@Qonfused
[docs/base-config.yml] Add additional notes to UpdateSMBIOSMode
6e5f244
@Qonfused
[parsers/schema] Add more robust boundary logic for inconsistent form…
10e3e9c 
…atting

Handles nested boundaries (invalid formatting) and restores missing nested lists.
@Qonfused
[parsers/schema] Handle additional commands and scopes
4d6ac5e
@Qonfused
[docs] Move gh repo docs and flatten docs resources
4381116
Qonfused
Qonfused commented Jul 28, 2023
View reviewed changes

@Qonfused Qonfused left a comment •
edited
Loading

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Schema docs generation is mostly uncoupled from the main library, though there currently exists a markdown parser in the ocebuild/parsers/schema.py file that may be worth moving separately under the current docsgen pipeline.

A more robust solution worth exploring is to use pandoc (i.e. ran in CI w/ a docker image), which will more reliably convert each schema's LaTeX entry to markdown. There is currently no action needed for the current schema parser which parses and segments these entries.

Qonfused
Qonfused commented Jul 28, 2023
View reviewed changes
Comment thread docs/resources/Schema.plist
<dict>
<key>OpenCore-Version</key>
<string>0.9.3</string>
</dict>

@Qonfused Qonfused Jul 28, 2023

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Schema docs will need to be updated on a regular cadence following each OpenCore release. Pre-release builds of OpenCore will also need to be smoke-tested against any schema validation in case new schema entries are added that must be specially handled.

Additionally, the usage of the '#Revision' dict mimics the same usage by OCLP in its payload config.plist here. Special care must be taken when introducing metadata keys that may conflict with other projects.

@Qonfused

Qonfused commented Jul 28, 2023

Copy link
Copy Markdown
Owner Author

Unfortunately due to a bug in the paths filtering behavior for GitHub actions, there is currently a block preventing progress on the CI for orchestrating integration testing. This will force current progress in this PR to be merged pre-maturely in order to continue.

The core library is mostly stable and tested, however, what is remaining regarding development and test coverage is primarily the pipeline supporting ocebuild_cli:
image

For the CLI, the build and lock commands are mostly finished and are pending the patch command to be implemented, which will handle the remaining config.plist patching for the build pipeline.

@Qonfused Qonfused mentioned this pull request Jul 28, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@Qonfused Qonfused

Labels

None yet

Projects

None yet

Milestone

v1.0.0-dev

Development

Successfully merging this pull request may close these issues.

[Feature]: Resolve config.plist schema conflicts

1 participant

@Qonfused