[iceberg] Write required Avro metadata to manifest files - #10329

Purpose

Newly written Iceberg manifests include the required Avro metadata: schema, schema-id, partition-spec, partition-spec-id, format-version, and content. Readers can resolve the schema and partition spec from the file headers. Previously, only avro.schema and avro.codec were present.

Fixes #8440.
Related: #9497.

The writer factory exposes an opt-in file-metadata capability, keeping paimon-core independent of the Avro implementation. Metadata is set before the Avro header is written. Each manifest writer receives its own data or deletes content value, and partition-spec is serialized as the required field array.

This draft retains the existing schema and metrics field IDs. Existing headerless manifests are not rewritten; legacy-manifest migration and field-ID remapping are outside this change.

Tests

• Reproduced the missing headers before the production fix: five of nine checks failed.
• JDK 17: 56 AvroFileFormatTest, 48 IcebergCompatibilityTest, 8 IcebergManifestBlockSizeTest, and 3 IcebergDVCompatibilityTest tests passed (115 total).
• Verified all six metadata entries in the physical headers, including partitioned and unpartitioned V2 data manifests and V3 data/delete manifests.
• Read generated V2 data and V3 data/delete manifests through Iceberg without an external partition-spec map.
• Checkstyle, Spotless, Maven Enforcer, and git diff --check passed for the final change.