🎨 Extract dominant colors in Elixir

Posted at — Nov 26, 2023

This is an article on how to create a Elixir wrapper of Rust crate. If you are more care about using the final library to extract dominant colors, head over to 29decibel/dominant_colors or reference this Livebook gist

livebook extract dominant colors

I found this very nice Rust library called kmeans-colors which can extract dominant colors from a given image. It based on k means clustering algorithm. It even comes with a binary. Which you can generate color palette like this:

kmeans_colors -i gfx/mountains.jpg --no-file --palette

Command line is great, but I wish I can use this in Elixir and Livebook.

So I go ahead wrote a simple wrapper on top of it using the awesome rustler library. In this article, I am gonna document how you can leverage the great Rust ecosystem and bring them into Elixir.

Create Elixir library

Let’s create a new library using mix new:

mix new dominant_colors

I choose to use dominant_colors rather than something like kmeans_colors_ex so that it’s more general and not bind to the internal algorithm.

After the project created. We need to add {:rustler, "~> 0.30.0"} to the dependency.

Create Rust crate inside

Then we can create a native Rust lib inside our elixir package:

mix rustler.new

Then when it prompted asking for the Elixir module the native module registered to. Type DominantColors, which is the module name mix generated.

This is the name of the Elixir module the NIF module will be registered to.
Module name > DominantColors
This is the name used for the generated Rust crate. The default is most likely fine.
Library name (dominantcolors) >
* creating native/dominantcolors/.cargo/config.toml
* creating native/dominantcolors/README.md
* creating native/dominantcolors/Cargo.toml
* creating native/dominantcolors/src/lib.rs
* creating native/dominantcolors/.gitignore
Ready to go! See /Users/mikeli/projects/dominant_colors/native/dominantcolors/README.md for further instructions.

Now let’s loading the NIF. This is what it becomes

defmodule DominantColors do
  use Rustler,
    otp_app: :dominant_colors,
    crate: :dominantcolors

  # When loading a NIF module, dummy clauses for all NIF function are required.
  # NIF dummies usually just error out when called when the NIF is not loaded, as that should never normally happen.
  def add(_arg1, _arg2), do: :erlang.nif_error(:nif_not_loaded)
end

Let’s try to add a simple test case to make sure the calling to Rust module works:

test "add" do
  assert DominantColors.add(1, 2) == 3
end

OK now we have the basic Elixir calling Rust working through rustler.

❯ mix test
.
Finished in 0.00 seconds (0.00s async, 0.00s sync)
1 test, 0 failures

Randomized with seed 733558

Integrate `kmeans-colors`

Now we can start the real work to integrate the Rust library. https://github.com/okaneco/kmeans-colors

So head over to native/dominantcolors.

Running cargo add kmeans-colors to add the dependency.

cargo add kmeans-colors
cargo add palette
cargo add image

Please notice we also added palette and image crate as well because we are going to use it to manipulate images and colors.

Here we have the first version of it. It’s heavily inspired by the kmeans-colors’s cli.

use kmeans_colors::{get_kmeans, Kmeans};
use palette::{FromColor, IntoColor, Lab, Srgb};

// reference
// https://github.com/okaneco/kmeans-colors/blob/master/src/bin/kmeans_colors/app.rs
#[rustler::nif]
fn dominant_colors(file_name: String) -> Result<Vec<String>, String> {
    // Read file into buffer
    let img = image::open(file_name);
    if img.is_err() {
        return Err("Can not open given file".to_string());
    }
    let img_vec = img.unwrap().to_rgb8().into_raw();

    // Convert RGB [u8] buffer to Lab for k-means
    let lab: Vec<Lab> = img_vec
        .chunks(3)
        .map(|chunk| {
            Srgb::new(
                chunk[0] as f32 / 255.0,
                chunk[1] as f32 / 255.0,
                chunk[2] as f32 / 255.0,
            )
            .into_format()
            .into_color()
        })
        .collect();

    // Perform k-means clustering
    let mut best_result = Kmeans::new();
    let converge = 5.0; // 5 for LAB or 0.0025 for RGB; // threshold for convergence.
    let runs = 1; // default just run once
    let cluster = 8; // number of clusters/colors returned
    let max_iter = 20; // maximum number of iterations
    let verbose = false;

    for i in 0..runs {
        // Run k-means multiple times
        let run_result = get_kmeans(cluster, max_iter, converge, verbose, &lab, 0 + i as u64);
        if run_result.score < best_result.score {
            best_result = run_result;
        }
    }

    // Convert Lab centroids back to Srgb<u8> for output
    let color_codes: Vec<String> = best_result
        .centroids
        .iter()
        .map(|x| Srgb::from_color(*x).into_format::<u8>())
        .map(|color| format!("#{:02x}{:02x}{:02x}", color.red, color.green, color.blue))
        .collect();

    Ok(color_codes)
}

rustler::init!("Elixir.DominantColors", [dominant_colors]);

I won’t go very deep on this, but I assume we will need more tweaks later on or expose more knobs to make it more customizable. But for now it should do the basic work.

Then we can register this function in our Elixir module:

defmodule DominantColors do
  use Rustler,
    otp_app: :dominant_colors,
    crate: :dominantcolors

  def dominant_colors(_arg1), do: :erlang.nif_error(:nif_not_loaded)
end

Let’s write a test to validate it.

test "dominant_colors" do
  file = "./test/fixtures/test3.png"
    assert DominantColors.dominant_colors(file) == {:ok, [
        "#bfbdbb",
        "#2f2e2e",
        "#a3a1a0",
        "#d8653a",
        "#eeedec",
        "#171515",
        "#7e7b7b",
        "#4d4b4b"] }
end

Works great 🎨.

Publish to hex

Now let’s publish it to hex and try to use it in Livebook.

In order to publish the package, we would need to add couple of info to the mix.exs

diff --git a/mix.exs b/mix.exs
index 7dd3f49..800a10c 100644
--- a/mix.exs
+++ b/mix.exs
@@ -7,7 +7,9 @@ defmodule DominantColors.MixProject do
       version: "0.1.0",
       elixir: "~> 1.15",
       start_permanent: Mix.env() == :prod,
-      deps: deps()
+      deps: deps(),
+      description: description(),
+      package: package()
     ]
   end
 
@@ -18,10 +20,23 @@ defmodule DominantColors.MixProject do
     ]
   end
 
+  defp description() do
+    "Extract dominant colors from given image (using kmeans), wrapper on top of rust kmeans_colors."
+  end
+
+  defp package() do
+    [
+      name: "dominant_colors",
+      licenses: ["MIT"],
+      links: %{"GitHub" => "https://github.com/29decibel/dominant_colors"}
+    ]
+  end
+
   # Run "mix help deps" to learn about dependencies.
   defp deps do
     [
-      {:rustler, "~> 0.30.0"}
+      {:rustler, "~> 0.30.0"},
+      {:ex_doc, ">= 0.0.0", only: :dev, runtime: false}
     ]
   end
 end

Making sure you are registered or logged in into hex:

mix hex.user auth

Now we can publish the package. If you got the error of following:

** (UndefinedFunctionError) function Hex.Crypto.decrypt/3 is undefined (module Hex.Crypto is not available)

Then you can try the following:

mix local.hex --force
mix hex.publish

If everything goes right, you should be able to see something like this:

Building dominant_colors 0.1.0
  Dependencies:
    rustler ~> 0.30.0 (app: rustler)
  App: dominant_colors
  Name: dominant_colors
  Files:
    lib
    lib/dominant_colors.ex
    priv
    priv/native
    priv/native/libdominantcolors.so
    .formatter.exs
    mix.exs
    README.md
  Version: 0.1.0
  Build tools: mix
  Description: Extract dominant colors from given image (using kmeans), wrapper on top of rust kmeans_colors.
  Licenses: MIT
  Links:
    GitHub: https://github.com/29decibel/dominant_colors
  Elixir: ~> 1.15
Before publishing, please read the Code of Conduct: https://hex.pm/policies/codeofconduct

Publishing package to public repository hexpm.

Proceed? [Yn] y
Building docs...
Generating docs...
View "html" docs at "doc/index.html"
View "epub" docs at "doc/dominant_colors.epub"
Local password:
Publishing package...
[#########################] 100%
Package published to https://hex.pm/packages/dominant_colors/0.1.0 (722129016b397b09a4a58349ab3f60ad12e4b24e09d1c5a301bd08a79ab18977)
Publishing docs...
[#########################] 100%
Docs published to https://hexdocs.pm/dominant_colors/0.1.0

Our package is now available.

Let’s create a dummy to consume it.

mix new test_colors

Then adding the dependency:

{:dominant_colors, "~> 0.1.0"}

Opps, we got an error:

spawn: Could not cd to native/dominantcolors

== Compilation error in file lib/dominant_colors.ex ==
** (RuntimeError) calling `cargo metadata` failed.

    (rustler 0.30.0) lib/rustler/compiler/config.ex:96: Rustler.Compiler.Config.metadata!/1
    (rustler 0.30.0) lib/rustler/compiler/config.ex:78: Rustler.Compiler.Config.build/1
    (rustler 0.30.0) lib/rustler/compiler.ex:8: Rustler.Compiler.compile_crate/3
    lib/dominant_colors.ex:2: (module)
could not compile dependency :dominant_colors, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile dominant_colors --force", update it with "mix deps.update dominant_colors" or clean it with "mix deps.clean dominant_colors"

Looks like we forgot the add the rust files into the package, that’s why it can’t find certain files in the published package. To fix this, make sure to include the rust files:

  defp package() do
    [
      files: [
        "lib",
        "native/dominantcolors/.cargo",
        "native/dominantcolors/src",
        "native/dominantcolors/Cargo*",
        "mix.exs"
      ],
      ...
    ]
  end

With this, we should be able to consume the package.

Rustler Pre-compilation

This works great if the consumer end have the Rust toolchain ready. However, in the context of Livebook this won’t work by default because Livebook do not have the access to the Rust toolchain like cargo.

Luckily rustler_precompiled is exactly trying to solve this.

The TLDR version of rustler_precompied is that it uses a CI server to pre-compile a bunch of binaries, then tell the Elixir to download those binary directly during installation. Think of it as a multi stage process.

Using CI server to pre compile binary. This is necessary to support multiple architecture machine (X86, ARM, Linux, PC, Mac …)
Save precompiled binary along with version number in Github artifacts.
Pointing the Elixir module to download that (with checksum).

Create Github action workflow

First let’s create a GitHub action workflow. Heavily borrowed from rustler_precompilation_example:

name: Build precompiled NIFs

on:
  push:
    branches:
      - main
    tags:
      - "*"

jobs:
  build_release:
    name: NIF ${{ matrix.nif }} - ${{ matrix.job.target }} (${{ matrix.job.os }})
    runs-on: ${{ matrix.job.os }}
    strategy:
      fail-fast: false
      matrix:
        nif: ["2.16", "2.15"]
        job:
          - {
              target: arm-unknown-linux-gnueabihf,
              os: ubuntu-20.04,
              use-cross: true,
            }
          - {
              target: aarch64-unknown-linux-gnu,
              os: ubuntu-20.04,
              use-cross: true,
            }
          # commentted this out as I am experiencing some compilation issues
          #   - {
          #       target: aarch64-unknown-linux-musl,
          #       os: ubuntu-20.04,
          #       use-cross: true,
          #     }
          - { target: aarch64-apple-darwin, os: macos-11 }
          - {
              target: riscv64gc-unknown-linux-gnu,
              os: ubuntu-20.04,
              use-cross: true,
            }
          - { target: x86_64-apple-darwin, os: macos-11 }
          - { target: x86_64-unknown-linux-gnu, os: ubuntu-20.04 }
          - {
              target: x86_64-unknown-linux-musl,
              os: ubuntu-20.04,
              use-cross: true,
            }
          - { target: x86_64-pc-windows-gnu, os: windows-2019 }
          - { target: x86_64-pc-windows-msvc, os: windows-2019 }

    steps:
      - name: Checkout source code
        uses: actions/checkout@v3

      - name: Extract project version
        shell: bash
        run: |
          # Get the project version from mix.exs
          echo "PROJECT_VERSION=$(sed -n 's/^  @version "\(.*\)"/\1/p' mix.exs | head -n1)" >> $GITHUB_ENV          

      - name: Install Rust toolchain
        uses: dtolnay/rust-toolchain@stable
        with:
          toolchain: stable
          target: ${{ matrix.job.target }}

      - name: Build the project
        id: build-crate
        uses: philss/rustler-precompiled-action@v1.0.1
        with:
          project-name: dominantcolors
          project-version: ${{ env.PROJECT_VERSION }}
          target: ${{ matrix.job.target }}
          nif-version: ${{ matrix.nif }}
          use-cross: ${{ matrix.job.use-cross }}
          project-dir: "native/dominantcolors"

      - name: Artifact upload
        uses: actions/upload-artifact@v3
        with:
          name: ${{ steps.build-crate.outputs.file-name }}
          path: ${{ steps.build-crate.outputs.file-path }}

      - name: Publish archives and packages
        uses: softprops/action-gh-release@v1
        with:
          files: |
                        ${{ steps.build-crate.outputs.file-path }}
        if: startsWith(github.ref, 'refs/tags/')

Then add the rustler_precompiled dependency:

  {:rustler_precompiled, "~> 0.7"},

Also add Cross.toml in the native/dominantcolors:

[build.env]
passthrough = ["RUSTLER_NIF_VERSION"]

And the .cargo/config.toml:

[target.'cfg(target_os = "macos")']
rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"]


# See https://github.com/rust-lang/rust/issues/59302
[target.x86_64-unknown-linux-musl]
rustflags = ["-C", "target-feature=-crt-static"]

# Provides a small build size, but takes more time to build.
[profile.release]
lto = true

Now we can push the changes to see if the build is green.

Finally after we validated its build success. We can release it.

git tag 0.1.0
git push --tags

Now we have the binaries built in the CI, we can grab them and use them instead.

Consume pre-compiled binaries

In order to get that, we need to do two things:

First we need to use the precompile macro:

  version = Mix.Project.config()[:version]

  use RustlerPrecompiled,
    otp_app: :dominant_colors,
    crate: "dominantcolors",
    base_url: "https://github.com/29decibel/dominant_colors/releases/download/#{version}",
    force_build: System.get_env("DOMINANT_COLORS_BUILD") in ["1", "true"],
    version: version

Then we need to download the checksum:

mix rustler_precompiled.download DominantColors --all

Now we should be able to use the precompiled version locally.

Please note we also provide a DOMINANT_COLORS_BUILD as a way to escape the pre compiled binary. It’s extremely useful for local development.

Publish to hex

Last step, let’s publish it to hex. Before the publish, we will need to make sure the checksum* files are included in the package, otherwise the installation will be failed with checksum not matched error.

  defp package() do
    [
      files: [
        "lib",
        "native/dominantcolors/.cargo",
        "native/dominantcolors/src",
        "native/dominantcolors/Cargo*",
        "checksum-*.exs",
        "mix.exs"
      ],
      ...
    ]
  end

We can verify the final package contents by using mix hex.build --unpack:

❯ mix hex.build --unpack
Building dominant_colors 0.1.1
  Dependencies:
    rustler ~> 0.30.0 (app: rustler)
    rustler_precompiled ~> 0.7 (app: rustler_precompiled)
  App: dominant_colors
  Name: dominant_colors
  Files:
    lib
    lib/dominant_colors.ex
    native/dominantcolors/.cargo
    native/dominantcolors/.cargo/config.toml
    native/dominantcolors/src
    native/dominantcolors/src/lib.rs
    native/dominantcolors/Cargo.lock
    native/dominantcolors/Cargo.toml
    checksum-Elixir.DominantColors.exs
    mix.exs
  Version: 0.1.1
  Build tools: mix
  Description: Extract dominant colors from given image (using kmeans), wrapper on top of rust kmeans_colors.
  Licenses: MIT
  Links:
    GitHub: https://github.com/29decibel/dominant_colors
  Elixir: ~> 1.15
Saved to dominant_colors-0.1.1

Looks great! Now we can publish it.

mix hex.publish

Using it in Livebook

Add dependency:

{:dominant_colors, "~> 0.1.3"}

Then some Elixir code and using Kino render:

image_path = "/Users/mikeli/Downloads/111ebfdc244345c3.png"
{:ok, colors} = DominantColors.dominant_colors(image_path)

styles = fn color ->
   [
    {"justify-content", "center"},
    {"align-items", "center"},
    {"color", "white"},
    {"display", "inline-flex"},
    {"width", "100px"},
    {"height", "30px"},
    {"background-color", color},
    {"margin", "1px"}
  ] |> Enum.map(fn {k, v} -> k <> ": " <> v <> ";" end)
  |> Enum.join(" ")
end

html = 
  Enum.map(colors, fn color ->
    "<div style=\"#{styles.(color)}\">#{color}</div>"
  end)
  |> Enum.join()

Kino.HTML.new(html)

How lovely!

Summary

I love the fact that Elixir can leverage so much in the Rust land with reasonable effort. It opened a whole new world to Elixir and Livebook users.

Next time, when you can’t find a library in the Elixir land, give some Rust crate and rustler a try.

Credits

rustler The one makes all this possible
rustler_precompiled Makes consuming rust based Elixir lib seamless
kmeans-colors Rust library to get dominant colors.
fast_rss Real world example of RSS parser using rustler.
Teenage Engineering EP 133 Just a beautiful object. And colors too.

What’s next

There are lots of other things we can improve for sure. But not a bad start I would say.

optimize the performance for big images
expose more knobs to tune the result
return percentage of each color (assign weight to the color returned)

Orchard Lab

A place to share my thoughts and learnings

🎨 Extract dominant colors in Elixir

Create Elixir library

Create Rust crate inside

Integrate `kmeans-colors`

Publish to hex

Rustler Pre-compilation

Create Github action workflow

Consume pre-compiled binaries

Publish to hex

Using it in Livebook

Summary

Credits

What’s next

Orchard Lab

A place to share my thoughts and learnings

🎨 Extract dominant colors in Elixir

Create Elixir library

Create Rust crate inside

Integrate kmeans-colors

Publish to hex

Rustler Pre-compilation

Create Github action workflow

Consume pre-compiled binaries

Publish to hex

Using it in Livebook

Summary

Credits

What’s next

Integrate `kmeans-colors`