Architecture
Technical overview of HPM — system design, dependency resolution, cleanup, storage, and Python integration. Intended for contributors and integrators.
Table of contents
- System overview
- Crate layout
- Core types
- Dependency resolution
- Install flow
- Project-aware cleanup
- Storage architecture
- Python integration
- Security and performance
- Extension points
System overview
HPM is a modular monolith. A single binary (hpm-cli) drives a handful of
focused library crates. Responsibilities are split along stable seams:
manifest parsing, configuration, resolution, storage, Python, and errors all
live in their own crates so they can be reused (e.g., a desktop app can
depend on the library crates and skip the CLI entirely).
┌────────────────────────────────────────────────────────────────────────┐
│ CLI (hpm-cli) │
│ • subcommands: init, add, remove, install, update, list, search, │
│ run, check, build, pack, audit, clean, registry, global, │
│ completions │
│ • output formats: human, json, json-lines, json-compact │
│ • shell completions: bash, zsh, fish, powershell, elvish │
└───────────────────────────────────┬────────────────────────────────────┘
│
┌───────────────────────────────────▼────────────────────────────────────┐
│ Core orchestration (hpm-core) │
│ • StorageManager: global package install/remove/list │
│ • ProjectDiscovery: scan configured paths for active projects │
│ • LockFile / LockedDependency / PackageSource │
│ • Registry trait + ApiRegistry, GitRegistry │
│ • ArchiveFetcher, packer │
└─────────┬──────────────────┬──────────────────┬─────────────────────────┘
│ │ │
┌─────────▼────────┐ ┌───────────────┐ ┌───────────────┐
│ hpm-package │ │ hpm-assets │ │ hpm-config │
│ PackageManifest│ │ Asset │ │ Config │
│ DependencySpec │ │ AssetKind │ │ Storage/ │
│ StageConfig │ └───────────────┘ │ Projects/ │
│ Platform │ │ Signing │
│ HoudiniPackage │ └───────────────┘
└──────────────────┘
Python tooling — bundled uv, content-addressable venvs, the
Houdini→Python ABI mapping, and per-script venv resolution — lives in
hpm_core::python. It was a separate hpm-python crate through 0.16
but folded into hpm-core since it had no consumers other than
hpm-core and hpm-cli.
hpm-package and hpm-assets are the workspace leaves — neither depends on
another HPM crate. hpm-config depends on hpm-package (it reuses the
manifest’s registry and dependency types), so it is not a leaf despite
sitting alongside them in the diagram above.
Each crate defines its own error type (e.g. StorageError, ConfigError)
via thiserror. Errors surface to the user through CliError in hpm-cli,
which converts them to exit codes and help hints.
Key non-functional properties:
- Language: Rust, edition 2024, MSRV 1.85.
- Async runtime: Tokio, multi-thread.
- Configuration: TOML throughout (manifests, lock file, global config).
- Testing: unit, integration, and property-based (proptest) tests.
Crate layout
| Crate | Responsibility |
|---|---|
hpm-cli | Command-line frontend (clap). Turns command-line invocations into calls on the library crates. |
hpm-core | Storage, project discovery, lock file, registry trait + two implementations, archive fetching/packing, Python tooling (python submodule: bundled uv, venvs, cleanup). |
hpm-package | hpm.toml parsing and validation, dependency/Python dependency types, Houdini package.json generation, platform enum. |
hpm-assets | Operator asset-index model (Asset, AssetKind) emitted by hpm pack from [[operators]] declarations. |
hpm-config | Global and project configuration loading and merging. |
Dependencies flow downward: hpm-cli depends on everything else, hpm-core
depends on package, assets, and config, and so on. No crate depends upward.
Domain errors (StorageError, ConfigError, …) live in the crate that
produces them.
Core types
Selected public types. See cargo doc --workspace --no-deps --open for the
full list.
hpm-package
#![allow(unused)]
fn main() {
pub struct PackageManifest {
pub package: PackageInfo,
pub compat: CompatConfig,
pub stage: StageConfig,
pub registries: Vec<RegistryConfig>,
pub dependencies: IndexMap<String, DependencySpec>,
pub python_dependencies: IndexMap<String, PythonDependencySpec>,
pub runtime: IndexMap<String, ManifestEnvEntry>,
pub scripts: PackageScripts,
pub operators: Vec<OperatorDecl>,
}
// All collections and section structs use `#[serde(default,
// skip_serializing_if = ...)]` so an absent TOML section round-trips to
// the same in-memory representation as an empty one. `Option` was
// removed in 0.17 — see CHANGELOG.
pub struct PackageScripts {
pub commands: IndexMap<String, ScriptEntry>, // flat entries under [scripts]
}
// Untagged: serialises as either a bare string ("cargo build") or a table
// form { cmd, python?, requirements?, label?, description?, package-env? }.
// The table form opts the script into
// a uv-managed venv resolved on demand by `hpm run`, and supports
// conditional cmd values via the same `when` grammar `[runtime]` uses.
pub enum ScriptEntry {
Plain(String),
WithEnv(ScriptEnv),
}
pub struct ScriptEnv {
pub cmd: EnvValue, // flat string or [{ when = { os }, set }, ...]
pub python: Option<String>, // e.g. "3.11"
pub requirements: Vec<String>, // e.g. ["PySide6>=6.6"]
pub label: Option<String>, // display name for UIs; hpm never acts on it
pub description: Option<String>, // one-line description for tooltips/help
pub package_env: bool, // run inside the package's full resolved env
}
pub struct PackageInfo {
pub path: PackagePath, // "creator/slug", validated kebab-case
pub name: String, // freeform display name
pub version: String, // semver
pub description: Option<String>,
pub authors: Vec<String>, // empty vec encodes "no authors declared"
pub license: Option<String>,
pub readme: Option<String>,
pub homepage: Option<String>,
pub repository: Option<String>,
pub documentation: Option<String>,
pub keywords: Vec<String>,
pub categories: Vec<String>,
}
pub enum DependencySpec {
// The bare-string shorthand `pkg = "1.0.0"` (de)serializes as
// Registry { registry: None, optional: false }.
Registry { version: String, registry: Option<String>, optional: bool },
Url { url: String, version: String, optional: bool },
Path { path: String, optional: bool, link: bool }, // link=true → symlink/junction install
}
pub enum Platform {
LinuxX86_64, // "linux-x86_64"
LinuxAarch64, // "linux-aarch64"
MacosX86_64, // "macos-x86_64"
MacosAarch64, // "macos-aarch64"
WindowsX86_64, // "windows-x86_64"
WindowsAarch64, // "windows-aarch64"
Universal, // "universal" — OS-agnostic (pure-Python / data)
}
}
hpm-core
#![allow(unused)]
fn main() {
pub struct LockFile {
pub package: LockPackageInfo,
pub dependencies: BTreeMap<String, LockedDependency>,
pub python_dependencies: BTreeMap<String, LockedPythonDependency>,
pub metadata: Option<LockMetadata>,
}
pub struct LockedDependency {
pub version: String,
pub checksum: Option<String>, // "sha256:<hex>"
pub source: LockedSource, // Url { url, version } | Path { path }
}
#[async_trait]
pub trait Registry: Send + Sync {
async fn search(&self, query: &str) -> Result<SearchResults, RegistryError>;
async fn get_versions(&self, name: &str) -> Result<Vec<RegistryEntry>, RegistryError>;
async fn get_version(&self, name: &str, version: &str) -> Result<RegistryEntry, RegistryError>;
async fn refresh(&self) -> Result<(), RegistryError>;
fn name(&self) -> &str;
}
}
hpm-core::python
#![allow(unused)]
fn main() {
pub struct PythonVersion {
pub major: u8,
pub minor: u8,
pub patch: Option<u8>,
}
pub struct ResolvedDependencySet {
pub python_version: PythonVersion,
pub packages: BTreeMap<String, String>, // name → exact version
}
pub struct VenvManager { /* … */ }
impl VenvManager {
pub async fn ensure_virtual_environment(
&self,
resolved: &ResolvedDependencySet
) -> Result<PathBuf, PythonError>;
}
}
hpm-config
#![allow(unused)]
fn main() {
pub struct Config {
pub install: InstallConfig,
pub storage: StorageConfig,
pub projects: ProjectsConfig,
pub registries: Vec<RegistrySourceConfig>,
pub signing: SigningConfig,
}
pub struct StorageConfig {
pub home_dir: PathBuf, // default: $HOME/.hpm
pub cache_dir: PathBuf,
pub packages_dir: PathBuf,
pub registry_cache_dir: PathBuf,
}
}
Configuration is layered: defaults → ~/.hpm/config.toml →
<cwd>/.hpm/config.toml. Files are parsed as presence-aware overlays
(ConfigOverlay), so each layer overrides exactly the values it sets — a
file that omits a section leaves the lower layers untouched.
Dependency resolution
HPM currently does naive per-package version selection: for each declared
registry dependency, query the registry, filter non-yanked versions, and
pick the highest that matches the spec’s VersionReq. There is no
transitive constraint solving — every dependency is treated as a direct
dependency, and registry packages don’t declare their own deps in a way
the install path consumes. A proper constraint solver (PubGrub-style or
otherwise) will be reintroduced if/when transitive resolution becomes a
real requirement.
Version requirement grammar
HPM manifest values accept the usual semver constraint operators:
| Form | Meaning |
|---|---|
=1.2.3, 1.2.3 | Exact. |
^1.2.3 | Compatible with the given major (>=1.2.3, <2.0.0). |
~1.2.3 | Patch updates allowed (>=1.2.3, <1.3.0). |
>=1.0.0, <2.0.0 | Explicit range. |
Install flow
The install command itself is a thin shell — it loads the manifest +
lockfile, builds a ProjectManager, and calls sync_dependencies().
The desktop client uses the same ProjectManager entry point, so both
clients run the same flow.
┌──────────────────────────────────────────────────────────────────────┐
│ hpm install (CLI) ── ProjectManager::sync_dependencies ── │
├──────────────────────────────────────────────────────────────────────┤
│ 1. Load hpm.toml │
│ 2. If hpm.lock exists: │
│ verify cached packages against stored checksums │
│ warn if metadata.generated_at > 90 days │
│ 3. Fetch + install in parallel (one task per dep, JoinSet): │
│ Registry → query registry, exact-version lookup │
│ → ArchiveFetcher → ~/.hpm/fetch/ │
│ → install_into_cas → ~/.hpm/packages/<slug>@<v>/ │
│ Url → ArchiveFetcher (no registry query) │
│ → install_into_cas │
│ Path → install_as_dev_copy (or install_as_dev_link │
│ if { link = true }) │
│ → ~/.hpm/packages/_dev/<slug>@<v>/ │
│ (real dir for copy, symlink/junction for │
│ link mode — both honor _dev/ CAS isolation) │
│ Already-in-CAS deps short-circuit (avoids the install_into_cas │
│ remove-and-recopy that breaks on Windows when Houdini is open). │
│ 4. Merge [python_dependencies] from root + every dep manifest │
│ Python ABI = root manifest's [compat].houdini lower bound │
│ 5. Ensure managed CPython installed under ~/.hpm/uv-python/ │
│ (uv python install <ver>) — auto-downloads on clean machines │
│ 6. Resolve with bundled uv, hash the resolved set, pick or │
│ rebuild ~/.hpm/venvs/<hash>/ │
│ 7. uv pip install --python <venv>/bin/python │
│ 8. Write <project>/.hpm/packages/{creator}.{slug}.json per dep │
│ 9. Sweep stale <project>/.hpm/packages/ entries │
│ 10. CLI step: build new lockfile from sync's InstallOutcomes │
│ (backfilled from the prior lockfile for short-circuited entries) │
│ and write hpm.lock │
└──────────────────────────────────────────────────────────────────────┘
Note: registry resolution currently uses each dep spec’s version string
as an exact lookup against the registry. Range specs like ^1.0.0 in
[dependencies] won’t resolve through hpm install today — hpm update
is the path that re-resolves ranges (querying all versions, filtering by
VersionReq, picking the highest non-yanked) and rewrites the spec to
the resolved exact version.
--frozen-lockfile aborts if loading the existing lockfile fails, if any
cached checksum mismatches, or if the freshly-resolved set differs from
the prior lockfile.
Project-aware cleanup
hpm clean removes packages and venvs no active project depends on. The
identity of “active projects” comes from the [projects] config section.
I = installed packages in ~/.hpm/packages/
P = active projects discovered via ProjectDiscovery
D(p) = direct deps declared by project p
T(I) = transitive closure over the dependency graph
orphans = I \ T( ⋃ D(p) )
Implementation outline:
#![allow(unused)]
fn main() {
// crates/hpm-core/src/graph.rs
impl DependencyGraph {
/// BFS from each root, returning every `PackageId` reachable along
/// dependency edges. The roots themselves are included.
pub fn mark_reachable_from_roots(&self, roots: &[PackageId]) -> HashSet<PackageId> {
let mut reachable = HashSet::new();
for root in roots {
let Some(&start) = self.index_of.get(root) else {
continue;
};
let mut bfs = Bfs::new(&self.graph, start);
while let Some(idx) = bfs.next(&self.graph) {
reachable.insert(self.graph[idx].id.clone());
}
}
reachable
}
}
}
DependencyGraph wraps a petgraph graph; its fields are private and it
exposes no orphans() of its own. Taking the set difference against what is
actually installed on disk is StorageManager::find_orphaned_packages
(crates/hpm-core/src/storage/cleanup.rs), which calls
mark_reachable_from_roots and subtracts the result from the installed set.
Project discovery scans explicit_paths plus recursive walks of
search_roots, respecting max_search_depth and skipping directory names
that match ignore_patterns. Only directories containing an hpm.toml are
considered projects.
Python venv cleanup uses the same principle against the metadata.json files
in each venv, which list the HPM packages using that venv.
Storage architecture
HPM stores everything under ~/.hpm/ on every supported platform. The
layout is content-addressable where it helps:
~/.hpm/
├── config.toml
├── packages/ # canonical CAS — written by StorageManager
│ ├── slug@1.0.0/ # registry / URL installs
│ │ ├── hpm.toml
│ │ ├── (package sources)
│ │ └── … (Houdini convention subdirs)
│ └── _dev/ # path-installed (dev) packages
│ └── slug@1.0.0/ # never substituted for a registry hit
│ └── … # at the same (slug, version)
├── fetch/ # ArchiveFetcher staging — extracted
│ └── creator-slug-1.0.0/ # archives live here briefly before
│ # install_into_cas copies into packages/
├── venvs/
│ └── <12-char hash>/ # hash of resolved set + Python version
│ ├── pyvenv.cfg
│ ├── bin/ # or Scripts/ on Windows
│ ├── lib/pythonX.Y/site-packages/
│ └── metadata.json
├── cache/ # download archive cache
├── registry/ # one subdir per registry
│ └── <registry name>/
├── tools/ # bundled uv binary
├── uv-cache/ # isolated uv cache
├── uv-config/
├── uv-python/ # managed CPython installs (UV_PYTHON_INSTALL_DIR)
└── fetch/ # staging for downloaded archives
hpm install routes URL/registry deps through a two-step flow:
ArchiveFetcher downloads + extracts into ~/.hpm/fetch/,
then StorageManager::install_into_cas copies into the canonical CAS at
~/.hpm/packages/<slug>@<version>/. Path deps skip the fetcher entirely
and go straight to ~/.hpm/packages/_dev/<slug>@<version>/ via
install_as_dev_copy.
Per-project:
<project>/
├── hpm.toml
├── hpm.lock
└── .hpm/
├── config.toml # optional
└── packages/
├── utility-nodes.json # generated Houdini manifests, one per dep
└── material-library.json # absolute paths into the global CAS
Global packages are shared across projects; each project holds only the per-dependency Houdini manifest and a lockfile. This keeps disk usage sane across a studio’s worth of projects. The Houdini JSON manifests carry absolute CAS paths, so the project tree contains no symlinks pointing into the global storage.
Each per-package manifest is named <creator>.<slug>.json. The creator
segment is part of the filename because the slug alone is not unique: two
creators may publish the same slug, and keying the file on the slug let the
second install silently overwrite the first, so Houdini loaded only one of
them. . is the separator rather than - because both segments may
themselves contain - (a-b/c and a/b-c would collide), while . is not
a legal segment character.
sync_dependencies sweeps stale per-package manifests at the end of every
install run: any .json in <project>/.hpm/packages/ whose stem is no
longer in the resolved dependency set is removed. Without this, a manifest
written by a previous run (e.g. for a path dependency that has since been removed)
would keep loading the package on Houdini launch even though hpm.toml no
longer asks for it. Manifests written by an older hpm under the bare
<slug>.json name are not in the current scheme, so the same sweep clears
them on the next install — otherwise the package would load twice, once per
filename. Note this makes <project>/.hpm/packages/ an hpm-owned directory:
unrecognized .json files placed there by hand do not survive a sync.
Global installs
hpm global writes the same Houdini manifest into Houdini’s per-user
preferences directory (hpm_core::houdini_prefs resolves it per platform,
honouring HOUDINI_USER_PREF_DIR and its __HVER__ placeholder) instead of
into a project. It reuses the whole install path — the same RegistrySet,
the same CAS, the same shared venvs, and the same manifest builder
(build_houdini_package, a free function precisely so both callers produce
byte-identical output).
The one structural difference is directory ownership, and it inverts the
sweep described above. <project>/.hpm/packages/ belongs to hpm, so the
project installer may delete anything it does not recognise there. Houdini’s
user packages/ directory belongs to the user: it holds files written by
SideFX, by other tools, and by hand. Applying the sweep there would delete
them. So global installs are tracked in a ledger at
~/.hpm/global/houdini-<X.Y>.json, and every operation is driven off it —
list reads it, remove deletes exactly the filename it records, and
nothing ever scans the directory to decide what to delete.
The ledger is also a GC root. find_orphaned_packages marks reachability
from discovered projects, and a globally installed package belongs to no
project; without its ledger entry it is unreferenced by construction, so
cleanup would delete the store directory while the manifest in Houdini’s
preferences kept pointing at it. An unreadable ledger aborts cleanup rather
than being treated as empty — “no global installs” and “cannot tell” must not
look the same to a collector.
Compatibility is checked before anything is written. The emitted enable
expression is evaluated by Houdini at startup, so a package whose
[compat].houdini excludes the target version would install successfully and
then load nothing, silently. check_compatible turns that into an install-
time error.
Path dependencies install into ~/.hpm/packages/_dev/<slug>@<version>/
rather than the registry CAS at ~/.hpm/packages/<slug>@<version>/.
The _dev subtree is invisible to list_installed, so a dev install of
foo@1.0.0 cannot be served as the cached install for a registry
resolution at the same coordinate from a different project.
_dev/ entries are garbage-collected by a parallel cleanup pass driven
directly off project path-dependencies (since the CAS-orphan logic
deliberately can’t see them). For each discovered project,
find_orphaned_dev_installs parses the manifest, resolves every
{ path = "..." } dep’s source manifest to its (slug, version), and
unions those into the “needed” set. Anything in _dev/ whose dir-name
encoding falls outside that set is orphan. A project with an unreadable
path-dep source logs a warning and skips that dep — a broken project
doesn’t bypass cleanup of other dev installs; re-running hpm install
re-creates whatever the project still needs. Removal goes through the
same remove_install_entry primitive used by clear_existing_install
and remove_package, so link installs are unlinked without traversal.
Path deps come in two install styles, selected by the link field on
the manifest’s { path = "...", link = ? } spec:
- Copy (default,
link = false):install_as_dev_copysnapshot-copies the workspace into a content-addressed_dev/<slug>@<version>/<source-hash>/directory (see below). Subsequent working-tree edits don’t reach the install until the nexthpm install. - Link (
link = true):install_as_dev_linkcreates a symlink (Unix) or NTFS junction (Windows) at_dev/<slug>@<version>/pointing at the canonicalized workspace. Edits are live; Houdini’s HPATH resolution follows the link transparently. Junctions on Windows side-step the Developer Mode / admin requirement that NTFS directory symlinks carry, so the workflow is viable on a stock Houdini workstation.
Native-binary packages are the exception: when a link = true path dep
declares concrete [compat].platforms (anything other than universal),
install_inner downgrades it to a copy — see
CompatConfig::declares_native_platforms. A junction/symlink would make
the workspace’s build output the very DSO/DLL a running Houdini has
memory-mapped, so an in-place rebuild fails (Windows LNK1104 /
ERROR_SHARING_VIOLATION), and link-mode buys nothing for native code
anyway — a mapped DSO can’t hot-reload into a live session; it needs a
relaunch, which re-copies and re-runs prepack. Pure-data / pure-Python /
universal-only link installs are unaffected.
Dev copies are content-addressed so a re-sync never clears a directory a
running Houdini has mapped. source_hash fingerprints the workspace (relative
path + length + mtime of every file, stat-only) and that hash names the
install directory: the copy lands at _dev/<slug>@<version>/<source-hash>/, and
the generated Houdini manifest points hpath there. install_inner stages the
copy into a hidden .stage-<pid>-<n> directory and commits it with an atomic,
race-tolerant rename (a concurrent install that loses the race drops its stage,
since the content is identical by construction). If the target hash directory is
already present and complete, the existing install is reused untouched — the
copy-mode analogue of the “already installed” short-circuits the registry/URL
specs have in install_one_dep.
The payoff is that a rebuild produces a new hash directory rather than
overwriting the old one: a concurrently-running Houdini keeps mapping the
directory it was launched from, so install_inner never remove_dir_alls a live
copy. Before content addressing, an in-place clear-and-recopy of a package a
second session still had mapped failed on Windows with os error 5 /
StorageError::PackageInUse; that failure is now structurally impossible on all
platforms. At install time only a stale link left by a prior link = true
install is cleared (clear_container_link, link-safe); the directory of live
hashes is left in place.
Superseded hash directories accumulate across a dev-iteration loop and are
reclaimed by hpm clean: cleanup_unused_dev_installs removes whole orphan
containers (tolerant of in-use) and, for still-referenced installs, prunes every
hash directory except the one matching the current source
(prune_stale_dev_hashes). Reclamation is best-effort and carries the same “run
when Houdini sessions are closed” expectation as CAS package cleanup — a copy
still mapped by a live process is skipped (on Windows the OS lock fails the
removal) rather than force-removed.
Both install replacement (clear_existing_install) and orphan cleanup
(remove_package) are symlink-safe: each checks symlink_metadata (plus
junction::exists on Windows) before deciding between remove_dir_all
(real dir) and remove_file / junction::delete (link entry), via a
shared remove_install_entry primitive. Without this, a remove_dir_all
on a Windows junction would recurse into and delete the user’s workspace
on the next sync or orphan sweep.
Both styles set InstalledPackage::is_dev = true. That flag flows
through to create_houdini_package_with_python, which forwards it to
ManifestEnvEntry::lower(.., is_dev). The install_source axis on each
conditional [runtime] variant is filtered there: variants gated to
"dev" only survive for path-installed packages, variants gated to
"registry" only survive for registry/URL installs, and variants with no
axis fire in both contexts. A registry-fetched install whose every
matching variant is install_source = "dev" produces no entry for that
key in the generated package.json, so dev-only paths never leak to a
published consumer.
Emission targets Houdini’s actual env semantics, as verified with the
hconfig conformance harness (hpm-core/src/houdini_conformance_tests.rs)
and modeled executably in hpm-core/src/houdini_env_model.rs: prepend /
append values are single-element JSON lists (a flat-string first
definition makes a custom variable non-mergeable — every later entry
overwrites it, whatever its method), set is emitted as a bare flat
string (Houdini has no set method; a list-form replace appends onto
a path-registered variable Houdini seeded flat-first, e.g. OCIO, so only a
flat string overwrites it cleanly — a genuinely conditional set still
falls back to replace since it can’t be flat), and conditional branches
are compiled mutually exclusive because
Houdini applies every matching conditional-array element, not the first
match. Project-level [runtime] entries are written once to
~hpm-project-overrides.json in the project packages dir; Houdini
processes package files in byte-wise ascending filename order and ~
sorts after every slug character, so that file always applies last — an
override lands exactly once, after (or, for set, replacing) every
package contribution.
Python integration
The Python layer runs on four ideas, in descending order of importance:
- Content-addressable venvs. Hash the resolved dependency set, use it as the venv directory name. Same hash → same venv → shared across packages.
- Bundled uv. A copy of
uvships with HPM and lives at~/.hpm/tools/uv. Its cache (~/.hpm/uv-cache/), config (~/.hpm/uv-config/), and managed CPython installs (~/.hpm/uv-python/, pinned viaUV_PYTHON_INSTALL_DIR) are isolated from any systemuvso HPM never perturbs your other Python work. - Self-bootstrapping Python.
uv pip compileanduv venvneed an interpreter. Before invoking either, HPM runsuv python install <ver>to ensure a managed CPython matching the project’s Houdini ABI exists. This is what makes a clean Windows install (no system Python anywhere) Just Work — without itpip compileerrors withNo interpreter found in virtual environments, managed installations, search path, or registry. The ensure step is process-cached, so it costs one fast filesystem probe per resolution. - Houdini manifest generation. For each HPM package that declares Python dependencies, HPM writes
<project>/.hpm/packages/{creator}.{slug}.jsonwithPYTHONPATHprepended at the shared venv’ssite-packages.
Hash function
#![allow(unused)]
fn main() {
// crates/hpm-core/src/python/types.rs
impl ResolvedDependencySet {
pub fn hash(&self) -> String {
let mut hasher = Sha256::new();
hasher.update(format!("python:{}", self.python_version));
for (name, version) in &self.packages {
hasher.update(format!("{}:{}", name, version));
}
hasher
.finalize()
.iter()
.map(|b| format!("{:02x}", b))
.collect::<String>()[..16]
.to_string()
}
}
}
packages is a BTreeMap<String, String>, so it iterates in sorted order
without an explicit sort. Only names and versions are hashed — extras are
not part of the resolved set.
Houdini→Python mapping
| Houdini | Python |
|---|---|
| 20.5+ | 3.11 |
| 21.x | 3.11 |
| 22.x | 3.13 |
The mapping is sourced from the lower bound of the root manifest’s
[compat].houdini — the project’s Houdini build is what determines the
embedded CPython ABI that wheels in the venv must match. A dependency
package’s own [compat].houdini describes a compatibility floor (the
oldest Houdini it runs on) and is not consulted for ABI selection: a
[compat].houdini = ">=21.0" package consumed by a Houdini-22 project
still gets a 3.13 venv.
Unsupported versions return an error. No silent fallback — an ABI-mismatched venv would break C-extension imports at Houdini launch instead of surfacing the mapping gap at install time. Houdini 19.x (Python 3.7) and 20.0–20.4 (Python 3.9) are unsupported: their Python interpreters are past upstream EOL.
Generated Houdini manifest
{
"hpath": ["/Users/me/.hpm/packages/studio/my-tool@1.0.0"],
"env": [
{
"PYTHONPATH": {
"method": "prepend",
"value": "/Users/me/.hpm/venvs/a1b2c3d4e5f6/lib/python3.11/site-packages"
}
}
],
"enable": "houdini_version >= '20.5'"
}
method: "prepend" delegates path-separator handling to Houdini so the same
manifest works on Windows (;) and Unix (:) without embedding an
OS-specific joiner. The generator is
hpm_core::project::houdini_emit::build_houdini_package, a free function
shared by the project installer and hpm global so both emit identical
manifests.
Security and performance
Security
- Transport: TLS through rustls (pure Rust, no OpenSSL). Certificate verification against the platform trust store.
- Integrity: SHA-256 over every installed archive, recorded in
hpm.lock, verified before every install. - Signing:
hpm pack --keysigns archives with Ed25519 over PKCS#8 PEM private keys.keyId= first 8 bytes of public key, hex-encoded. See Security for the wire format. - Isolation: bundled
uvruns out-of-process with its own cache and config, never touching system state.
Performance
- Concurrent downloads: registry-fronted installs run in parallel, bounded by
[install].parallel_downloads(default 8). - Content-addressable dedup: both HPM packages (keyed by
slug@versionin the global CAS) and Python venvs (keyed by resolved-set hash) are deduplicated globally. - Cache hits:
uv’s cache eliminates wheel re-downloads across projects; HPM’s venv cache eliminates install work when a matching set is found. - Atomic filesystem ops: packages are extracted to a temp directory and renamed into place to avoid partial states.
Extension points
Programmatic configuration
The library crates can be used without the CLI. Embedders (desktop apps,
pipeline tools) build a Config in-memory by starting from
Config::default() and setting the fields they need:
#![allow(unused)]
fn main() {
use hpm_config::{Config, RegistrySourceConfig, RegistryType};
let mut config = Config::default();
config.add_registry(RegistrySourceConfig {
name: "tumbletrove".into(),
url: "https://api.tumbletrove.com/v1/registry".into(),
registry_type: RegistryType::Api,
});
config.install.path = "packages/hpm".into();
}
Registry trait
Any type that implements the Registry async trait can be plugged into a
RegistrySet and used for resolution. The built-in ApiRegistry and
GitRegistry are reference implementations.
For visibility-gated API registries (e.g. private packages an
authenticated user is entitled to see), embedders can pass a bearer
token through RegistrySet::from_configs_with_auth — the token is
attached as Authorization: Bearer <token> on every API-registry
request and marked sensitive so reqwest won’t log it. The token is
baked into the HTTP client at construction, so callers tracking a
refreshing token rebuild the RegistrySet per operation rather than
mutating one in place. Git registries currently ignore the token.
ProjectManager mirrors that contract via
ProjectManager::new_with_auth(.., Option<String>). The token is
stashed on the manager and forwarded to every RegistrySet it builds
internally — sync_dependencies for registry-form deps and
add_dependency’s registry-resolved path both go through
from_configs_with_auth with the stashed token. ProjectManager::new
delegates with None, so anonymous use is unchanged. As with the
RegistrySet variant, refreshing tokens are handled by rebuilding the
ProjectManager per operation.
Plugin system
Not implemented. The CLI surface is currently fixed at compile time. A dynamic plugin system is on the roadmap but not prioritized — most studios are better served by wrapping HPM in a higher-level pipeline tool.