How to search for specific packages

Developing with flox
1

I am trying to floxify one of my libraries.
However I am finding it difficult to find which are the correct dependencies for the project.
In particular I depend on tasty-hedgehog which must have a version between 1.2 and 2.0.

A flox search does not tell me much and a nixos search shows me that there’s a tasty-hedgehog_1_2_0_0 package. But then I accidentally discovered a tasty-hedgehog_1_3_1_0 package and a tasty-hedgehog_1_4_0_0 version in the unstable channel for the nixos packages.

The fact that this package had to put version numbers in its name looks very fishy to me (it gets worse since it has a dependency on a hedgehog-2_0_0 package).

My question is: what is the flox way to discover those packages?

2

Hi @etorreborre!
One of our engineers should be getting back to you on this tomorrow. :slight_smile:

1 Like
3

CLI search for libraries is more specific than the common case of someone looking binaries/tools. This example is further complicated by nixpkgs’s haskellPackages not supporting recursive search into it by default.

flox nix search nixpkgs-flox#haskellPackages tasty-hedgehog

will provide a few results with fully resolved names. You would only need the last portion, which you seem to be doing here: registry/default.nix at floxify · etorreborre/registry · GitHub

Other standard nixpkgs search mechanisms also apply (search.nixos.org, search in github, google, git grep of a clone)

4

What is important is that nix does not implement version range resolution.
In nix expressions you have to depend on one specific package that packages one specific version, hence the Pkg_1_2_3_4 names.

Many language package managers implement version resolution for in varying complexity, but most importantly in a different way, that makes it hard or at least inconvenient to implement the same in nix.

Now, you see that for your package there is no single nixpkgs yet that contains a valid combination of its dependencies.
Luckily nix allows you to provide your own versions (and can pull versions directly from hackage)

You can take registry/default.nix at floxify · ysndr/registry · GitHub for reference on how to do that.

//

However, what i would be very interested in is getting to know your goal with packaging your library.

  • would you like to create an environment to develop that library]
    • will that require nix provided dependencies?
    • or do you just need the cabal/hls/ghc/… environment to be able to work on it?
  • would you like to provide the lib to other binaries of yours as dependency and want it to be a newer version than the one shipping with nix?

//

Background:

Nix packages defines a large subset (all?) of the latest hackage releases
That is because the haskell community bought into nix quite early - stack has official nix support.

Hackage has 15644(today) packages, which are all somewhat referenced in nixpkgs (e.g, https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/development/haskell-modules/hackage-packages.nix) – that includes registry (3.2.1)

Why does flox search not find these?:

In short, for performance reasons.
flox search needs to evaluate all these 15k packages in order to show them.
The same is true for pythonPackages (less complete than haskell but still, big!) and a couple of other package sets which had to be included as well
Also flox build is still more targeted to package consumers showing what you can install to your environments rather than use in your build expressions.

Why does search.nixos.org find these packages?:

search.nixos.org indexes a greater extend of nixpkgs and stores it on an elasticsearch instance.
Searching on the website is therefore more efficient as its served by an actual search engine.
We have solutions in mind for similar approaches in flox but were not there just yet.

Why so many tasty hedgehogs?

Nixpkgs has not figured out how to encode multiple versions of the same package.
A nix expression packages one specific version of a program.
If two versions are packaged they cannot share the same name, but have to be distinguished somehow, often by the package name.
On top of that the haskellPackages are generated automatically and more versions might be added manually to agree with package constraints

5

Thanks a lot, I did not realize that I could reuse cabal2nix that way and just override some dependencies.

I think that in first approximation I want to be able to be able to develop with up to date tools: stack, hls, ormolu, ghc etc… For example right now I am trying to flox install vscode but I can’t find it.

My understand is that it is does by calling hackage directly like you did on my floxify branch so it might not be so necessary to have the most recent version on nix. Actually I realize that the latest version is there but shows up as broken. I am going to see how to fix this.

6

Now, given your explanation, I wonder if registry is broken because it requires

> Setup: Encountered missing or private dependencies:
       > tasty-hedgehog >=1.2 && <2.0

and nix does not find it because it should be looking at tasty-hedgehog-1_4_0_0.

7

Hej Eric

let me leave some notes for you, If you haven’t figured things out over the weekend

  1. nix has a whole subsystem specifically for haskell already built into nixpkgs (ot to be confused with GitHub - input-output-hk/haskell.nix: Alternative Haskell Infrastructure for Nixpkgs). unfortunately the documentation tends to be on the sparse side (nix is working on it and we’re here to help you find things in the meantime).
  1. as for registry is broken on nixpkgs:
  • registry requires the GHC2021 language.
  • nixpkgs-stable’s ghc is ghc 9.0.1, which predates 9.2.1(?) which introduced GHC2021
  • since registry does not build on the default toolchain, it was marked as broken.
  1. how to unbreak registry
  • build it with a newer version of ghc and first try to force building the package (often that already works for packages that just require a different ghc)
  • below: we modify the haskell package set based on ghc 9.2.4 by declaring a new package for registry that is the existing version, but marked as not broken using haskell.lib.markUnbroken 
let my-haskell = haskell.packages.ghc924.override {
  overrides = self: super: {
    registry = haskell.lib.markUnbroken super.registry;
  };
};

in my-haskell.registry;
  1. how to develop registry then?

firstly: if you were inclined to run flox with the unstable (i.e. most recent iteration of) nixpkgs set, no overrides will be necessary, at all

to build your local package just haskellPackages.callCabal2nix "registry" ../.. {} will suffice, on stable, the dependencies have to be figured out as demostrated in my fork.

The wiki on haskell development with nix shows how to use shellFor to get a development shell.
Inyour case that would be something like:

let my-haskell = haskell.packages.ghc924.override {
  overrides = self: super: {
    registry = haskell.lib.markUnbroken super.registry;
  };
};

in my-haskell.shellFor {
    packages = p: [p.registry];
    withHoogle = true;
    buildInputs = [ my-haskell.haskell-language-server pkgs.ormulu pkgs.ghcid /*and more*/ ];
  }

I’m on it to come up with the right command, but beware there is a lot of recompiling, since only the default packages come with prebuilts on nix (and with ghc 9.2.4 you’ll apparently need a ghc9.2.4 compile hls)

8

Thanks Yannik. I don’t have much time this week to work on any of this but I hope to come back to it soon. Just one or 2 questions about what you wrote above:

  • " 3. how to unbreak registry": what you indicate here is how to get an unbroken version of registry if I want to invoke it when building another project, right? And then, hopefully, when ghc 9.2.4 goes into nixpkgs stable everything will be fixed automatically and the package marked as unbroken?

  • if I use shellFor, can this get activated when I use flox develop? How can flox even know about shellFor?

Thanks

9

if I want to invoke it when building another project, right?

Yes that approach would not necessarily fit the usecase of developing that library, correct.

And then, hopefully, when ghc 9.2.4 goes into nixpkgs stable everything will be fixed automatically and the package marked as unbroken?

Most likely, yes

if I use shellFor , can this get activated when I use flox develop ?

Yes, flox develop will pick that package up and provide a development shell for it.
shellFor is a nix library function that produces an ordinary package, with a special build env that is adjusted for developing purposes rather than build, it should give you the usual cabal to allow incremental building using the haskell tools, but with the dependencies put in place by nix.

You can actually flox build the package, but the result wont be the built library but presumably a dev env activation script at best.
Often (e.g. for binaries) you would have two packages, one for the binary and one for the dev shell

1 Like
10

Hi Tom, I’m coming back to this while my code is compiling :-). I don’t think that using flox nix is mentioned anywhere in the user manual. It would be nice to explain:

  • what it does differently than just calling nix
  • why it might be necessary to search something
  • or maybe to just phase it out and provide more options to flox search?
11

On the subject of install, I am trying to install a linker zld.

flox nix search nixpkgs zld                                                                                                                                                                                                                                              rust
* legacyPackages.x86_64-darwin.zld (1.3.4)
  A faster version of Apple's linker

I can see that it’s available with a nixos search. What would be the recommended flox commands to search that package and install it?

I can do this

flox nix search nixpkgs zld                                                                                                                                                                                                                                              rust
* legacyPackages.x86_64-darwin.zld (1.3.4)
  A faster version of Apple's linker

but then this fails

flox install legacyPackages.x86_64-darwin.zld

I have tried other variants like flox install zld, flox install nixpkgs.zld but I don’t understand how to refer to the proper name for that package (apologies if it was already explained to me in another place :-)).

12

@etorreborre : i’ve got some time today or the weekend if you’d like to go through some blockers collaboratively, it might be a fast way to see how you are using things as well as to answer any questions you may have (Calendly - Thomas Bereknyei)

  • what it [flox nix] does differently than just calling nix

It gives you some opinionated settings and a registry. For example, it turns on some Nix experimental features.

  • why it might be necessary to search something

The normal flox search is catered+optimized for installing normal tools and binaries. Library searching is a bit more complex and we do not have that integrated into flox search yet, so the escape hatch of using flox nix search and search.nixos.org is available until we integrate these better into the tooling.

  • or maybe to just phase it out and provide more options to flox search?

Yes, that is our plan.

zld

I’m not sure why this is not available via our existing catalog; I am sure this can be fixed. As a stop-gap measure, you can install anything via normal Nix mechanisms, so in your case

flox install 'nixpkgs#zld'
1 Like
13

It works, thanks!

Thanks for your proposal to have a session. I am travelling for part of the week-end but I’ll keep that in mind.

Some questions about flox install 'nixpkgs#zld':

  • are the quotes really necessary? (it seems to work fine without)

  • when I do a flox list I end up with 1 zld.nixpkgs. in the list (note the final . in the name)

  • when an environment is activated, why is it necessary to still mention -e <currently active env> when doing a flox install. Wouldn’t it be easier to install to the currently active environment and not have to specify -e? Maybe not because what we activate is more like a list of active environments, so this would mean the latest activated and might be ambiguous?

14
  • are the quotes really necessary? (it seems to work fine without)

Some people, especially on Mac, use zsh with extended_glob turned on. This makes the # a special character in a non-POSIX-compatible way. Normal usage allows it inside of a word without being interpreted as the start of a comment. I added the quotes there just in case this was true for you, but it turns out the quotes are not necessary for you.

  • when I do a flox list I end up with 1 zld.nixpkgs. in the list (note the final . in the name)

I noticed that. This has to do with the assumption that most things will be installed via the catalog. Be aware that the install via this method might have a few issues, please let us know if you run into anything.

  • when an environment is activated, why is it necessary to still mention -e <currently active env> when doing a flox install. Wouldn’t it be easier to install to the currently active environment and not have to specify -e? Maybe not because what we activate is more like a list of active environments, so this would mean the latest activated and might be ambiguous?

We call this “modal commands” where the active environment becomes the default for all the env manipulation commands. It requires more awareness for the person to understand what had been activated, hence we’ve begun adding this to the default command-line prefix. Adding the modal-style manipulation is on our roadmap.

1 Like