Skip to content

🧠 How the pxl-pipeline cli works

πŸš€ Pipeline lifecycle

  • init β†’ generate template
  • test β†’ runs the pipeline locally in .venv/
  • deploy β†’ builds & pushes Docker image + registers in Picsellia
  • smoke-test β†’ runs pipeline in a container before deploying

πŸ“‚ Project structure

Here is a typical pipeline folder structure:

my_pipeline/
β”œβ”€β”€ config.toml
β”œβ”€β”€ pyproject.toml
β”œβ”€β”€ uv.lock
β”œβ”€β”€ Dockerfile

β”œβ”€β”€ picsellia_pipeline.py
β”œβ”€β”€ local_pipeline.py
β”œβ”€β”€ steps.py
β”œβ”€β”€ utils/
β”‚ └── parameters.py

β”œβ”€β”€ runs/
β”‚ └── run1/
β”‚ └── run_config.toml

└── .venv/

Key files:

  • config.toml Describes the pipeline metadata, entrypoint files, requirements file, and model metadata. βž• This makes pipelines easily portable and shareable.

  • pyproject.toml / uv.lock Managed by uv to declare dependencies. You don’t need to manually install anything β€” just run the CLI.

  • picsellia_pipeline.py Entrypoint when running on Picsellia (inside Docker).

  • local_pipeline.py Entrypoint for running and testing the pipeline locally.

  • steps.py Contains @step-decorated functions that define the logic of your pipeline.

  • utils/parameters.py Contains the parameter class (TrainingHyperParameters, ProcessingParameters, etc.) used to extract configuration at runtime.

  • .venv/ Created automatically by the CLI when you run pxl-pipeline test.

πŸ” Environment variables

The CLI requires:

PICSELLIA_API_TOKEN
PICSELLIA_ORGANIZATION_NAME
PICSELLIA_HOST  # optional, defaults to https://app.picsellia.com

They are:

  • Prompted once during init, test, or deploy
  • Saved in: ~/.config/picsellia/.env
  • Automatically loaded on future runs

You can:

  • Manually edit that file
  • Or override any value in the current terminal session with export VAR=...

🧰 Dependency management with uv

Each pipeline uses uv as the dependency manager. It handles package resolution and installation via pyproject.toml, without needing pip or poetry.

πŸ“¦ What happens during pxl-pipeline test?

When you run:

pxl-pipeline test my_pipeline

The following is automatically done for you:

  • uv lock resolves all dependencies and generates/updates uv.lock
  • uv sync installs packages into .venv/ based on the lock file

You don't need to install or activate anything manually β€” the CLI ensures the right environment is built.

βž• Adding dependencies

To install a PyPI package:

uv add opencv-python --project my_pipeline

To add a Git-based package:

uv add git+https://github.com/picselliahq/picsellia-cv-engine.git --project my_pipeline

This updates the pyproject.toml and uv.lock files inside your pipeline folder.

πŸ’‘ Tip: the --project flag ensures the package is added to the correct pipeline folder.

πŸ“ How runs/ work

Each test run creates a new directory under runs/:

β”œβ”€β”€ runs/
β”‚   β”œβ”€β”€ run1/
β”‚   β”œβ”€β”€ run2/
β”‚   └── run3/
β”‚       └── run_config.toml

Inside each run folder:

  • run_config.toml stores the parameters used for that run (e.g. experiment_id, model_version_id, etc.)
  • The dataset and model will be downloaded into this folder
  • Logs, annotations, and any outputs will be saved here

Reusing configurations

  • If a previous run exists, the CLI will prompt:
πŸ“ Reuse previous config? experiment_id=... [Y/n]
  • Choosing Y reuses the last config (but creates a new folder and re-downloads assets).

  • Use the flag --reuse-dir to reuse the same directory and config, without downloading again.

Working with pipeline parameters

βž• Adding a custom parameter

Each pipeline includes a utils/parameters.py file containing a parameter class that extracts and validates values from Picsellia metadata (experiment or processing).

1. Locate your parameters file

my_pipeline/
β”œβ”€β”€ utils/
β”‚   └── parameters.py  ← edit this file

2. Edit the parameter class

Inside parameters.py, you’ll find a class that inherits from:

  • Parameters (for processing pipelines)

  • HyperParameters (for training pipelines)

Add your new fields by calling self.extract_parameter(...) in the constructor.

from picsellia_cv_engine.core.parameters import Parameters

class ProcessingParameters(Parameters):
    def __init__(self, log_data):
        super().__init__(log_data)

        # Add your custom parameters here πŸ‘‡
        self.threshold = self.extract_parameter(
            keys=["threshold"],
            expected_type=float,
            default=0.5,
        )

        self.use_filter = self.extract_parameter(
            keys=["use_filter"],
            expected_type=bool,
            default=True,
        )
  1. Link the class in config.toml

Make sure the class is declared in your pipeline’s config.toml:

[execution]
parameters_class = "utils/parameters.py:ProcessingParameters"

βœ… What you can define

Each parameter can include:

Field Description
keys One or more fallback keys (e.g. ["lr", "learning_rate"])
expected_type Type validation (int, float, bool, str, Optional[...])
default Optional default value (or ... to mark as required)
range_value Value bounds: (min, max) for numeric parameters

Advanced use cases (enums, optional types, dynamic validation) are documented in the base Parameters class via extract_parameter(...).

βœ… Summary

  • Pipelines are self-contained and shareable via config.toml

  • Dependencies are isolated and reproducible with uv

  • CLI stores runs in runs/, with config and outputs

  • Parameters are centralized and easy to extend

  • You can deploy to Picsellia with pxl-pipeline deploy ...

For template-specific usage, see: