I’ve been messing around with flox some at home and at work and I’m really liking what I see. For context I work at a startup that is hiring a few junior data engineers/analysts and I want to use this tool to help manage the environments they work in. Historically that’s always been a mess at other companies and I think Nix via Flox will really help out a lot with that.
One thing I’m a bit confused about is the best practices around using Flox and Python, specifically installing packages. My current solution is as follows:
Set up the env for a project. In this case it’ll be project-a
flox install -e project-a python310 poetry
Then i do the standard poetry init and poetry add for packages. I’ve got poetry configured to install in a local .venv folder in that directory that’s ignored by .gitignore.
Finally I’m adding the .venv/bin folder to the path in the .envrc file.
This has been working great in my test envs and with our dbt project. IS there a better way of doing this? Any examples I can see? If feasible I’d like to keep the package management using Poetry as that is what folks are used to, but i’m flexible. Probably having them edit/search for nix recipes(is that the right jargon) is probably out atm.
If you’re happy with poetry, that’s a great way to do it. If you want to automate steps 3 and 4, you could add those commands to a hook, e.g. flox edit -e project-a and add:
Sounds like that’s probably best for your use case. We also support using Python packages from Nixpkgs, but it does require a little more Nix at the moment, although we’re planning to simplify it eventually. Taking the Nixpkgs route means you could cut out poetry (but it sounds like that is a non-goal for you at the moment) But if you want to experiment with doing it that way, you could flox edit -e project-a and add
inline = {
packages = {
myPython = {python3}:
python3.withPackages (pythonPackages: with pythonPackages; [add packages you need here]);
};
}
flox activate -e flox_poetry runs without issue and the executables/modules installed by poetry appear to be on PATH.
However, python -c "import numpy" complains that it can’t load zlib. Adding zlib = {}; to the packages.nixpkgs-flox set (as shown above) doesn’t help though. ruff also does not work. I get:
bash: /tmp/flox_poetry/.venv/bin/ruff: cannot execute: required file not found
I’ll need to get back to you on the use of poetry with your example, but in the meantime I wanted to provide you with a solution that does not use poetry, but instead employs the Nix/flox “native” way of providing a version of python configured with the packages/modules you listed in your pyproject.toml.
In the example below I’m using a Nix expression to declare a postBuild hook that renames “python*” → “mypython*” and performs a few smoke tests to make sure all the expected modules import properly, but of course you can reconfigure that part to suit your specific needs, and/or add anything (packages, env variables, hooks) as you would for any flox environment.
{
packages.nixpkgs-flox.ruff = {};
inline = {
packages.mypython = {python3}:
let
# Prefix for python executable name.
pyprefix = "my"; # e.g. python installed as "mypython"
# Attribute set of python modules -> packages that they come
# from, used to drive both the build and "smoke test".
# --> add packages to this list
extraLibModules = {
# Python import name pythonPackages name
# ================== = ===================
# e.g. tensorflow = "tensorflow-bin_2";
requests = "requests";
numpy = "numpy";
black = "black";
mypy = "mypy";
pytest = "pytest";
};
in
(python3.buildEnv.override {
ignoreCollisions = true;
extraLibs = map (pyPkg: python3.pkgs.${pyPkg}) (builtins.attrValues extraLibModules);
postBuild = ''
# Rename the python* binaries to minimize confusion.
( cd $out/bin && for i in python*; do mv $i ${pyprefix}$i; done )
# A quick "smoke test" to ensure we have all the necessary imports.
for i in ${builtins.toString (builtins.attrNames extraLibModules)}; do
( set -x && $out/bin/${pyprefix}python -c "import $i" )
done
'';
}).overrideAttrs (oldAttrs: {
name = oldAttrs.name + "-${pyprefix}python";
});
};
}
I’ve uploaded this environment for the x86_64-linux, x86_64-darwin and aarch64-darwin architectures and you can download/try it out with the following:
flox [flox/prerelease default] brantley@MacBook-Air ~ % flox subscribe flox-examples github:flox-examples/floxpkgs/master
subscribed channel 'flox-examples'
flox [flox/prerelease default] brantley@MacBook-Air ~ % flox pull -e flox-examples/discourse-665
Everything up-to-date
flox [flox/prerelease default] brantley@MacBook-Air ~ % flox activate -e flox-examples/discourse-665
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % mypython
Python 3.10.9 (main, Dec 13 2022, 00:53:42) [Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> import numpy
>>> import black
>>> import mypy
>>> import pytest
>>>
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % which black
/Users/brantley/.local/share/flox/environments/flox-examples/x86_64-darwin.discourse-665/bin/black
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % which mypy
/Users/brantley/.local/share/flox/environments/flox-examples/x86_64-darwin.discourse-665/bin/mypy
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % which pytest
/Users/brantley/.local/share/flox/environments/flox-examples/x86_64-darwin.discourse-665/bin/pytest
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ % which ruff
/Users/brantley/.local/share/flox/environments/flox-examples/x86_64-darwin.discourse-665/bin/ruff
flox [flox-examples/discourse-665 flox/prerelease default] brantley@MacBook-Air ~ %
… and if you’d like to make a copy of this environment to hack on then you can do the following:
Again, we’ll need to get back to you with a recipe for using poetry with flox, but wanted to pass this on in case it helps in the meantime. Thanks for your interest in flox!
Thanks for the example. It works nicely, as long as the Python packages are in pythonPackages, which, very sadly, doesn’t cover my use case. Would it be possible to use poetry2nix in a flox expression?
I see the dilemma - thanks for sharing the list. Yes there are several packages in that list missing from nixpkgs pythonPackages. I tried to incorporate your list of packages and found the following missing or otherwise not working (“#” indicates missing, “####” indicates requirement for custom package, others as noted):
It is certainly possible and I’ve undergone similar efforts to package a similar volume of missing & broken packages in the past, but before we do I will ask @tomberek to chime in with his thoughts. I’m not terribly familiar with poetry but my understanding is that the issues are twofold:
need to configure poetry to compile all packages and avoid using binary wheels containing embedded interpreter and run paths not found on all systems (or if it must, to then patchelf files as required to get them to use lib dependencies provided by Nix)
ultimately will need to employ poetry2nix or similar to create a packaged version of the resulting application that is able to run on all system types
Many thanks again for this interesting and challenging example! Your question is coming at the perfect time for us to work with you to create and document a general solution to the problem involving flox. We’ll get back to you shortly.
Thanks for checking which packages are and which are not in nixpkgs. I could have done due diligence myself, sorry for the laziness
I think both pyscf and qiskit should be package-able for aarch-64: my team mates build the wheels from source on their M1 mac machines. qdldl also appears to be in nixpkgs-unstable already.
For the rest, I’ve never had much luck writing derivations for Python packages, but if there’s a good workflow that you can suggest, then I’m all ears. It’s going to be easier than trying to figure out why a wheel build within poetry install suddenly starts failing and it gives me more control over which system libraries (e.g. BLAS/LAPACK) are in use. As I said, I’m not super-fond of poetry.
I’ve been chipping away at packaging the missing packages over the last few days and wanted to share my progress.
For context you’ll notice some packages on this list weren’t included in the original list because they are required to build some other package, but aren’t “runtime dependencies”. As I work my way through the list a few more are bound to appear for this reason, but here’s where I’m at so far:
DONE memray
DONE mike
DONE mkdocs_gen_files
DONE mkdocs_literate_nav
DONE mkdocs_section_index
DONE verspec
DONE lmfit
DONE ntlm-auth
DONE rustworkx
STARTED nevergrad
TODO quimb
TODO scalene
TODO cotengra
TODO qdldl
TODO pyscf
TODO qiskit
TODO qiskit_nature
TODO cv2
TODO cdt
TODO gym_anm
TODO qiskit-ibm-experiment
TODO qiskit_experiments
Currently these are published to a private channel, but I’ll be working with the team once all of them are done to get them merged into the public floxpkgs channel for you.
Hi @robertodr! These are currently in a private repo but once done Alex will put them in github:flox-examples/floxpkgs which is a public repo. They will also be submitted to nixpkgs and eventually show up there.
Hi @robertodr - apologies, it looks like the work Alex was doing needs to be rolled in to our normal flox-examples channel. I have created a ticket to get this done for you so it will be available for you to use.
