Testing Guide

HPM uses several layers of tests: traditional unit tests, integration tests, property-based tests with proptest, and a Houdini conformance test that checks generated package files against a real Houdini. This guide covers how to run them, how to write new ones, and how to debug failures.

Table of contents

• Test layers
• Running tests
• Configuration
• Writing property tests
• Regression files
• Debugging failures

Test layers

Houdini conformance test

hpm-core/src/houdini_conformance_tests.rs writes package files through the real emission path, runs HOUDINI_PACKAGE_VERBOSE=1 $HFS/bin/hconfig (license-free), and asserts the merged env values from the verbose package log. It exists because the emission layer was once validated only against its own JSON output while Houdini silently ignored method on flat-string custom variables. Two scenarios are covered:

• houdini_resolves_generated_packages_per_method — packages sharing a variable plus project overrides in every method (set / prepend / append) and conditional (version/OS-gated) values.

• houdini_set_overwrites_path_registered_ocio_seed — the OCIO crash on H22.0.367: a project set on a path-registered variable must OVERWRITE Houdini’s own $HFS/packages/ocio.json seed, not append onto it. It reads the real seed via a baseline hconfig run (skipping with a note on builds that don’t seed OCIO), then asserts set replaces it wholesale rather than resolving to the two-path builtin;project value that crashes OCIO.Config.CreateFromFile. The path-registered-append behavior is invisible to the executable model below (it starts from an empty variable set), so only this real-Houdini layer can catch it.

• The Houdini install is auto-discovered: $HFS first, then the platform-standard locations (/opt/hfs*, /Applications/Houdini/..., C:\Houdini *).

• Without an install the test skips (passing, with a SKIPPED note). Set HPM_REQUIRE_HOUDINI=1 to turn the skip into a failure — the CI check pipeline does, since the workers all have Houdini.

The semantics the emission layer targets are also captured as an executable model (hpm-core/src/houdini_env_model.rs); a property test (houdini_emission_model_tests.rs) runs randomized packages and project overrides through the real emission code and the model, asserting that package values survive overrides, overrides apply exactly once, and nothing emitted uses a method or value shape Houdini rejects. When a new question about Houdini’s package semantics comes up, extend the conformance test to settle it empirically, then encode the answer in the model.

Property test distribution

Property tests are concentrated in the crates with the most value-shaped logic: manifest parsing, Python version handling, storage types. Exact counts shift over time — run grep -rh "fn prop_" crates/*/src crates/*/tests | wc -l for the current number (the manifest strategies live in crates/hpm-package/tests/properties.rs and the CLI strategies in crates/hpm-cli/tests/cli_validation.rs).

Running tests

# everything: cli (single-threaded), rest of workspace, then doctests
just test

# doctests only — public-API examples in //! and /// blocks
just test-doc

# slow / external-dependency tests gated behind `#[ignore]`
# (currently: real-uv venv smoke tests in hpm-core::python)
just test-ignored

# raw cargo equivalents
cargo test --workspace
cargo test --workspace --doc

# one crate
cargo test -p hpm-core

# one test by name
cargo test prop_version_req_roundtrip

# property tests only
cargo test prop_

# sequential execution (required when tests touch shared filesystem paths;
# `just test` already does this for hpm-cli via the CwdGuard)
cargo test --workspace -- --test-threads=1

# more proptest cases (default 256)
PROPTEST_CASES=1000 cargo test prop_ --workspace

Configuration

Proptest environment variables

CI

There is no proptest-specific CI matrix — CI runs the suite at proptest’s default case count and never sets PROPTEST_CASES.

The only test pipeline is .woodpecker/check.yml, which is tag-gated (event: tag, ref: refs/tags/v*) and so does not run on push or pull request. It runs:

cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace

with HPM_REQUIRE_HOUDINI=1, since the build workers have Houdini installed and the hconfig conformance test must run rather than silently skip.

Raising the case count is therefore a local operation:

PROPTEST_CASES=2000 cargo test --workspace

Writing property tests

A property test defines a strategy for generating inputs and an invariant that must hold for every generated input.

Custom strategies

Constrain generators to the valid input space. Unconstrained any::<String>() mostly generates nonsense and rejects it, which wastes cycles and produces misleading shrinks.

#![allow(unused)]
fn main() {
use proptest::prelude::*;

// Good — produces valid package slugs by construction
fn slug_strategy() -> impl Strategy<Value = String> {
    "[a-z][a-z0-9-]{1,20}"
        .prop_map(|s| s.trim_end_matches('-').to_string())
        .prop_filter("non-empty", |s| !s.is_empty())
}

fn version_strategy() -> impl Strategy<Value = String> {
    (0u32..100, 0u32..100, 0u32..100)
        .prop_map(|(maj, min, pat)| format!("{maj}.{min}.{pat}"))
}
}

Invariants worth testing

Roundtrip — serialize and deserialize and compare:

#![allow(unused)]
fn main() {
proptest! {
    #[test]
    fn prop_manifest_toml_roundtrip(manifest in manifest_strategy()) {
        let toml_str = toml::to_string(&manifest).unwrap();
        let parsed: PackageManifest = toml::from_str(&toml_str).unwrap();
        prop_assert_eq!(manifest.package.path, parsed.package.path);
        prop_assert_eq!(manifest.package.version, parsed.package.version);
    }
}
}

Validation always holds — every value the strategy produces should pass validation:

#![allow(unused)]
fn main() {
proptest! {
    #[test]
    fn prop_valid_manifests_validate(manifest in valid_manifest_strategy()) {
        prop_assert!(manifest.validate().is_ok());
    }
}
}

Determinism — running the same operation twice should produce the same result:

#![allow(unused)]
fn main() {
proptest! {
    #[test]
    fn prop_content_hash_is_deterministic(set in resolved_set_strategy()) {
        prop_assert_eq!(set.hash(), set.hash());
    }
}
}

Real bug caught by property tests

VersionReq::new(" ") (whitespace-only) was incorrectly accepted as valid. A property test that fed r"\s*" into the constructor surfaced it; the fix was to trim before the empty check. Every such bug gets a regression file that fails until the fix stays in place.

Regression files

Proptest persists failing cases to crates/<crate>/proptest-regressions/. These are source-of-truth regression tests — commit them alongside the fix.

crates/hpm-core/proptest-regressions/
├── houdini_emission_model_tests.txt   # Each line is a minimized failing input.
├── project.txt
└── storage/types.txt

hpm-core is currently the only crate with recorded regressions.

Never delete these by hand unless you’re certain the bug class is gone and the input is no longer meaningful.

Debugging failures

Get a minimal failing input

Proptest automatically shrinks to the smallest input that still fails. To see the shrink trace:

PROPTEST_VERBOSE=1 cargo test failing_prop_test -- --nocapture

Reduce the case count while iterating

PROPTEST_CASES=10 cargo test failing_prop_test -- --nocapture

Inspect the regression file

cat crates/hpm-core/proptest-regressions/project.txt

Each line is a serialized form of the seed that reproduced the bug. Proptest replays these on every subsequent run, so the same minimal case is checked deterministically until you fix it.

Move/borrow issues in assertions

prop_assert_eq! moves its operands. If you want to assert on a field and then on a predicate, clone:

#![allow(unused)]
fn main() {
prop_assert_eq!(value.field.clone(), expected);
prop_assert!(value.is_valid());
}

Running coverage

cargo install cargo-tarpaulin
PROPTEST_CASES=500 cargo tarpaulin --workspace --out html

Resources

• Proptest book
• Hypothesis: what is property-based testing?