Questions around using this with python & packages

Hi there!

I’ve been messing around with flox some at home and at work and I’m really liking what I see. For context I work at a startup that is hiring a few junior data engineers/analysts and I want to use this tool to help manage the environments they work in. Historically that’s always been a mess at other companies and I think Nix via Flox will really help out a lot with that.

One thing I’m a bit confused about is the best practices around using Flox and Python, specifically installing packages. My current solution is as follows:

  1. Set up the env for a project. In this case it’ll be project-a
  2. flox install -e project-a python310 poetry
  3. Then i do the standard poetry init and poetry add for packages. I’ve got poetry configured to install in a local .venv folder in that directory that’s ignored by .gitignore.
  4. Finally I’m adding the .venv/bin folder to the path in the .envrc file.

This has been working great in my test envs and with our dbt project. IS there a better way of doing this? Any examples I can see? If feasible I’d like to keep the package management using Poetry as that is what folks are used to, but i’m flexible. Probably having them edit/search for nix recipes(is that the right jargon) is probably out atm.

Hi, glad to hear it!

If you’re happy with poetry, that’s a great way to do it. If you want to automate steps 3 and 4, you could add those commands to a hook, e.g. flox edit -e project-a and add:

shell.hook = ''
  poetry init
  poetry add
  export PATH="$PWD/.venv/bin:$PATH"

Sounds like that’s probably best for your use case. We also support using Python packages from Nixpkgs, but it does require a little more Nix at the moment, although we’re planning to simplify it eventually. Taking the Nixpkgs route means you could cut out poetry (but it sounds like that is a non-goal for you at the moment) But if you want to experiment with doing it that way, you could flox edit -e project-a and add

inline = {
  packages = {
    myPython = {python3}:
      python3.withPackages (pythonPackages: with pythonPackages; [add packages you need here]);

(I copied that snippet from flox.nix configuration | flox:docs)

1 Like

@mkenigs Thanks! That’s super helpful :slight_smile:


I’m trying to do something and running into difficulties.
In order:

  • I have a pyproject.toml:
name = "flox_poetry"
version = "0.1.0"
description = ""
authors = ["Me"]

python = ">=3.8,<3.11"
numpy = "^1.19"
requests = "*"

pre-commit = "^3"
black = "^22"
mypy = "^0.991"
ruff = "^0.0.241"

pytest = "^7"

requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
  • I create an environment flox create -e flox_poetry
  • I run flox edit -e flox_poetry such that the contents of the file defining the declarative environment are as follows:
  packages = {
    nixpkgs-flox = {
      python3 = {};
      poetry = {};
      zlib = {};

  shell.hook = ''
     echo "Flox Environment"
     poetry install
     export PATH="$PWD/.venv/bin:$PATH"

flox activate -e flox_poetry runs without issue and the executables/modules installed by poetry appear to be on PATH.
However, python -c "import numpy" complains that it can’t load zlib. Adding zlib = {}; to the packages.nixpkgs-flox set (as shown above) doesn’t help though. ruff also does not work. I get:

bash: /tmp/flox_poetry/.venv/bin/ruff: cannot execute: required file not found

I’m not sure if this is useful, but this thread might be of interest:


Hi @robertodr,

I’ll need to get back to you on the use of poetry with your example, but in the meantime I wanted to provide you with a solution that does not use poetry, but instead employs the Nix/flox “native” way of providing a version of python configured with the packages/modules you listed in your pyproject.toml.

In the example below I’m using a Nix expression to declare a postBuild hook that renames “python*” → “mypython*” and performs a few smoke tests to make sure all the expected modules import properly, but of course you can reconfigure that part to suit your specific needs, and/or add anything (packages, env variables, hooks) as you would for any flox environment.

  packages.nixpkgs-flox.ruff = {};
  inline = {
    packages.mypython = {python3}:
        # Prefix for python executable name.
        pyprefix = "my"; # e.g. python installed as "mypython"
        # Attribute set of python modules -> packages that they come
        # from, used to drive both the build and "smoke test".
        # --> add packages to this list
        extraLibModules = {
          # Python import name   pythonPackages name
          # ================== = ===================
          # e.g. tensorflow    = "tensorflow-bin_2";
            requests           = "requests";
            numpy              = "numpy";
            black              = "black";
            mypy               = "mypy";
            pytest             = "pytest";
        (python3.buildEnv.override {
          ignoreCollisions = true;
          extraLibs = map (pyPkg: python3.pkgs.${pyPkg}) (builtins.attrValues extraLibModules);
          postBuild = ''
            # Rename the python* binaries to minimize confusion.
            ( cd $out/bin && for i in python*; do mv $i ${pyprefix}$i; done )
            # A quick "smoke test" to ensure we have all the necessary imports.
            for i in ${builtins.toString (builtins.attrNames extraLibModules)}; do
              ( set -x && $out/bin/${pyprefix}python -c "import $i" )
        }).overrideAttrs (oldAttrs: {
          name = + "-${pyprefix}python";

I’ve uploaded this environment for the x86_64-linux, x86_64-darwin and aarch64-darwin architectures and you can download/try it out with the following:

flox [flox/prerelease default] brantley@MacBook-Air ~ % flox subscribe flox-examples github:flox-examples/floxpkgs/master
subscribed channel 'flox-examples'
flox [flox/prerelease default] brantley@MacBook-Air ~ % flox pull -e flox-examples/discourse-665  
Everything up-to-date
flox [flox/prerelease default] brantley@MacBook-Air ~ % flox activate -e flox-examples/discourse-665
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % mypython
Python 3.10.9 (main, Dec 13 2022, 00:53:42) [Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> import numpy
>>> import black
>>> import mypy
>>> import pytest
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % which black
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % which mypy
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % which pytest
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % which ruff
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % 

… and if you’d like to make a copy of this environment to hack on then you can do the following:

flox [flox/prerelease default] brantley@MacBook-Air ~ % flox export -e flox-examples/discourse-665 | flox import -e mypython   
created generation 1
flox [flox/prerelease default] brantley@MacBook-Air ~ % 

Again, we’ll need to get back to you with a recipe for using poetry with flox, but wanted to pass this on in case it helps in the meantime. Thanks for your interest in flox!

1 Like

Thanks for the example. It works nicely, as long as the Python packages are in pythonPackages, which, very sadly, doesn’t cover my use case. Would it be possible to use poetry2nix in a flox expression?

Hi @robertodr - could you let us know which specific Python packages are missing and we can look to get these added?

These are the dependencies blocks in the pyproject.toml my team and I are working with:

python = ">=3.8,<3.11"
scipy = "~1.9"
qiskit = "^0.39"
cma = "^3"
joblib = "^1.1"
qiskit-nature = "^0.4"
pyscf = { version = "^2.1", optional = true, markers = "sys_platform != 'win32'" }
picos = "^2.4.11"
quimb = { version = "^1.4.2", extras = ["tensor"] }
sparse = "^0.13.0"
dask = "^2022.10.0"
cvxpy = "^1.2.2"
numba = "^0.56.4"
qiskit-experiments = "^0.4.0"
qdldl = { git = "", rev = "8211ee6" }
cotengra = { git = "", rev = "1a746c6" }
nevergrad = "^0.5.0"

pre-commit = "^3"
black = "^22"
mypy = "^0.991"
ruff = "^0.0.241"

pytest = "^7"
hypothesis = "^6.46.0"
pytest-cov = "^4"
pytest-rerunfailures = "^11.0"

pyscf = ['pyscf']

optional = true

mkdocs = "^1.4.2"
mkdocs-material = "<9"
mkdocs-jupyter = "^0.22.0"
mkdocstrings = "^0.20.0"
mkdocstrings-python = "^0.8.3"
mike = "^1.1.2"
mkdocs-gen-files = "^0.4.0"
mkdocs-literate-nav = "^0.6.0"
mkdocs-section-index = "^0.3.4"
black = "^22"

optional = true

jupyter-console = "*"
notebook = "*"
matplotlib = "*"

optional = true

scalene = "*"
line-profiler = {version = "*", extras = ["ipython"]}
memray = "*"

I am not superfond of poetry, but it’s what my team is used to: that’s why I was looking into wrapping it up with flox.

I see the dilemma - thanks for sharing the list. Yes there are several packages in that list missing from nixpkgs pythonPackages. I tried to incorporate your list of packages and found the following missing or otherwise not working (“#” indicates missing, “####” indicates requirement for custom package, others as noted):

          # Python import name      pythonPackages name
          # ===================== = ===================
          # e.g. tensorflow       = "tensorflow-bin_2";
            black                 = "black";
            cma                   = "cma";
####        cotengra              = "cotengra";
            cvxpy                 = "cvxpy";
            dask                  = "dask";
            hypothesis            = "hypothesis";
            joblib                = "joblib";
            jupyter_console       = "jupyter_console";
            line_profiler         = "line_profiler";
            matplotlib            = "matplotlib";
#           memray                = "memray";
#           mike                  = "mike";
            mkdocs                = "mkdocs";
#           mkdocs_gen_files      = "mkdocs_gen_files";
            mkdocs_jupyter        = "mkdocs-jupyter";
#           mkdocs_literate_nav   = "mkdocs_literate_nav";
            material              = "mkdocs-material";
#           mkdocs_section_index  = "mkdocs_section_index";
            mkdocstrings          = "mkdocstrings";
            mkdocstrings_handlers = "mkdocstrings-python";
            mypy                  = "mypy";
#           nevergrad             = "nevergrad";
            notebook              = "notebook";
            numba                 = "numba";
            picos                 = "picos";
            pre_commit            = "pre_commit";
###         pyscf                 = "pyscf"; # Requires linux-headers; won't work on aarch64
            pytest                = "pytest";
            pytest_cov            = "pytest-cov";
            pytest_rerunfailures  = "pytest-rerunfailures";
####        qdldl                 = "qdldl";
###         qiskit                = "qiskit"; # Requires linux-headers; won't work on aarch64
#           qiskit_experiments    = "qiskit-experiments";
##          qiskit_nature         = "qiskit-nature"; # unit tests fail
#           quimb                 = "quimb";
#           scalene               = "scalene";
            scipy                 = "scipy";
            sparse                = "sparse";

It is certainly possible and I’ve undergone similar efforts to package a similar volume of missing & broken packages in the past, but before we do I will ask @tomberek to chime in with his thoughts. I’m not terribly familiar with poetry but my understanding is that the issues are twofold:

  • need to configure poetry to compile all packages and avoid using binary wheels containing embedded interpreter and run paths not found on all systems (or if it must, to then patchelf files as required to get them to use lib dependencies provided by Nix)
  • ultimately will need to employ poetry2nix or similar to create a packaged version of the resulting application that is able to run on all system types

Many thanks again for this interesting and challenging example! Your question is coming at the perfect time for us to work with you to create and document a general solution to the problem involving flox. We’ll get back to you shortly. :+1:

1 Like

Thanks for checking which packages are and which are not in nixpkgs. I could have done due diligence myself, sorry for the laziness :sweat_smile:

I think both pyscf and qiskit should be package-able for aarch-64: my team mates build the wheels from source on their M1 mac machines. qdldl also appears to be in nixpkgs-unstable already.
For the rest, I’ve never had much luck writing derivations for Python packages, but if there’s a good workflow that you can suggest, then I’m all ears. It’s going to be easier than trying to figure out why a wheel build within poetry install suddenly starts failing and it gives me more control over which system libraries (e.g. BLAS/LAPACK) are in use. As I said, I’m not super-fond of poetry.



I’ve been chipping away at packaging the missing packages over the last few days and wanted to share my progress.

For context you’ll notice some packages on this list weren’t included in the original list because they are required to build some other package, but aren’t “runtime dependencies”. As I work my way through the list a few more are bound to appear for this reason, but here’s where I’m at so far:

  • DONE memray
  • DONE mike
  • DONE mkdocs_gen_files
  • DONE mkdocs_literate_nav
  • DONE mkdocs_section_index
  • DONE verspec
  • DONE lmfit
  • DONE ntlm-auth
  • DONE rustworkx
  • STARTED nevergrad
  • TODO quimb
  • TODO scalene
  • TODO cotengra
  • TODO qdldl
  • TODO pyscf
  • TODO qiskit
  • TODO qiskit_nature
  • TODO cv2
  • TODO cdt
  • TODO gym_anm
  • TODO qiskit-ibm-experiment
  • TODO qiskit_experiments

Currently these are published to a private channel, but I’ll be working with the team once all of them are done to get them merged into the public floxpkgs channel for you.

1 Like