osm-pbf-parquet

Transcode OSM PBF file to parquet files with hive-style partitioning by type

Download latest version from releases

Example for x86_64 linux system with pre-compiled binary:

curl -L "https://github.com/OvertureMaps/osm-pbf-parquet/releases/latest/download/osm-pbf-parquet-x86_64-unknown-linux-gnu.tar.gz" -o "osm-pbf-parquet.tar.gz"
tar -xzf osm-pbf-parquet.tar.gz
chmod +x osm-pbf-parquet
./osm-pbf-parquet --input your.osm.pbf --output ./parquet

OR compile and run locally:

git clone https://github.com/OvertureMaps/osm-pbf-parquet.git
cargo run --release -- --input your.osm.pbf --output ./parquet

• Local filesystem
• AWS S3 (auth read from environment, see object_store docs)

planet.osm.pbf
parquet/
  type=node/
    node_0000.zstd.parquet
    ...
  type=relation/
    relation_0000.zstd.parquet
    ...
  type=way/
    way_0000.zstd.parquet
    ...

Files are rolled once the parquet writer has emitted --file-target-mb (default 500MB), so output files land close to that size.

Reference Arrow/SQL schema

duckdb -c "SELECT * FROM read_parquet('s3://your-s3-bucket/path/') LIMIT 10;"

CREATE EXTERNAL TABLE IF NOT EXISTS `osm` (
    `id` BIGINT,
    `tags` MAP<STRING, STRING>,
    `lat` DOUBLE,
    `lon` DOUBLE,
    `nds` ARRAY<STRUCT<ref: BIGINT>>,
    `members` ARRAY<STRUCT<type: STRING, ref: BIGINT, role: STRING>>,
    `changeset` BIGINT,
    `timestamp` TIMESTAMP,
    `uid` BIGINT,
    `user` STRING,
    `version` BIGINT,
    `visible` BOOLEAN
)
PARTITIONED BY (
    `type` STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS PARQUET
LOCATION 's3://your-s3-bucket/path/';

MSCK REPAIR TABLE `osm`;

SELECT * FROM osm LIMIT 10;

1. Install rust and just
2. Clone repo git clone https://github.com/OvertureMaps/osm-pbf-parquet.git
3. Make changes
4. Run against PBF with cargo run -- --input your.osm.pbf (Geofabrik regional PBF extracts here)
5. Run just --list to see available dev commands (just test, just clippy, just ci-test, etc.)

osmpbf-async and osm-pbf-parquet are two separate crates that version independently, so there's no single tag or release that covers both.

Publishing to crates.io and cutting a CLI release are two separate, independent mechanisms:

• crates.io: bump the version in whichever crate's Cargo.toml changed (either one, both, or neither) as part of your PR. .github/workflows/publish_crates.yml runs on every merge to main and publishes a crate only if its current Cargo.toml version isn't already on crates.io, via Trusted Publishing (no token to mint or store). A merge that doesn't bump a crate's version is a no-op for that crate; nothing else to do.
• CLI binaries: cut a GitHub release with a vX.Y.Z tag to build and attach osm-pbf-parquet binaries for each supported platform, via .github/workflows/release_deploy.yaml. This is independent of the crates.io publish above; a release doesn't publish anything to crates.io, and a crates.io publish doesn't create a release or attach binaries.

osm-pbf-parquet prioritizes transcode speed over preserving element ordering.

Measured on the 2026-08-03 OSM planet PBF (94GB, 12.0B elements), transcoded by each tool on the same machine (Core Ultra 7 265K, 64GB, input on NVMe, output on a separate SATA SSD), pinned to the same 8 performance cores, with 8 worker threads and zstd level-3 parquet output, and outputs verified equivalent via aggregate checksums. CPU time is user + system time; peak memory includes child processes, so Sedona's Spark JVM is counted:

¹ ST_ReadOSM emits no metadata columns (changeset/timestamp/uid/user/version/visible), so it writes less data per row.

² Only osm-pbf-parquet targets a file size; the others write one file per output partition. See bench/README.md.

Distributed under the MIT License. See LICENSE for more information.

• osmpbf and osm2gzip for reading PBF data
• osm2orc for schema and processing ideas