Orchard Lab

Short Blog

Sat, 04 Oct 2025 20:49:28 -0700

It’s been 3 days not reading any news, no social media, using computer just for coding.

Feels great.

While reading my RSS feed today, I realized that you can just write short blog as well. Thanks Seth Godin!

Ship It

Sat, 22 Jun 2024 08:51:07 -0700

I have a problem. A problem that has bothered me for a long time: not finishing the side projects I started.

No matter how big or small the project is, I always manage to give up at the very last minute. Whether the excuse is chasing so-called perfection or finding it’s not useful for myself anymore. Not to mention you could use the fancy term “sunk cost” to justify it.

I am sick of it.

I need to make a change this year.

Instead of letting my mind go wild and constantly running npm init or Xcode -> New, I need to finish the damn thing. I realized that not finishing the project causes more damage to my conscious mind than the “sunk cost.”

An unpopular finished project is fine, even if it’s not popular, even if it doesn’t resonate with other people as it once did with me.

Like training for a marathon, some days you feel awful. You could choose to skip it or have a shorter run. Skipping is very contagious.

Finishing a project needs practice, discipline, accepting imperfection, and prioritizing closure over lingering.

Multiple environments using Fly.io

Sat, 03 Feb 2024 15:01:57 -0800

I’ve been using fly.io since 2021. But I never realize I can deploy multiple environments (staging, production) easily as well. In this post I will share how can you add a staging environment in 5 minutes.

Assuming you already have a fly.toml in your project, something like this:

app = 'example-production'
primary_region = 'lax'

[build]

[env]
  APP_ENV = 'production'
  AWS_REGION = 'us-west-1'

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  cpu_kind = 'shared'
  cpus = 1
  memory_mb = 1024

First let’s create a new config called fly.staging.toml:

cp fly.toml fly.staging.toml

Now let’s modify the fly.staging.toml to reflect what we want in this env:

app = "example-staging" # <--- Making sure you have a different name
primary_region = "lax"

[build]

[http_service]
internal_port = 3000
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 0
processes = ["app"]

[[vm]]
cpu_kind = "shared"
cpus = 1
memory_mb = 1024

[env]
APP_ENV = "staging"  # <--- Here we set the APP_ENV to staging
AWS_REGION = 'us-west-1'

We can’t use fly launch though with this new config. As fly launch only take fly.toml as of today. Instead we would need to create a new app for now.

# making sure the name here is the same with the above fly.staging.toml app name
fly apps create example-staging

Now with the app created. All we need to do is to deploy using this new config:

fly deploy --config ./fly.staging.toml

Yay! You just deployed a staging env using the same repository. You can run fly apps list to check the list of apps now.

Bonus (Auto deployment using GHA)

With this we can have a different Github workflow to deploy to staging when pushes to staging branch:

.github/workflows/fly-staging.yml

name: Fly Deploy Staging
on:
  push:
    branches:
      - staging
jobs:
  deploy:
    name: Deploy app
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: flyctl deploy --remote-only --config ./fly.staging.toml
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

Cheers and happy flying!

🎨 Extract dominant colors in Elixir

Sun, 26 Nov 2023 10:52:27 -0800

This is an article on how to create a Elixir wrapper of Rust crate. If you are more care about using the final library to extract dominant colors, head over to 29decibel/dominant_colors or reference this Livebook gist

I found this very nice Rust library called kmeans-colors which can extract dominant colors from a given image. It based on k means clustering algorithm. It even comes with a binary. Which you can generate color palette like this:

kmeans_colors -i gfx/mountains.jpg --no-file --palette

Command line is great, but I wish I can use this in Elixir and Livebook.

So I go ahead wrote a simple wrapper on top of it using the awesome rustler library. In this article, I am gonna document how you can leverage the great Rust ecosystem and bring them into Elixir.

Create Elixir library

Let’s create a new library using mix new:

mix new dominant_colors

I choose to use dominant_colors rather than something like kmeans_colors_ex so that it’s more general and not bind to the internal algorithm.

After the project created. We need to add {:rustler, "~> 0.30.0"} to the dependency.

Create Rust crate inside

Then we can create a native Rust lib inside our elixir package:

mix rustler.new

Then when it prompted asking for the Elixir module the native module registered to. Type DominantColors, which is the module name mix generated.

This is the name of the Elixir module the NIF module will be registered to.
Module name > DominantColors
This is the name used for the generated Rust crate. The default is most likely fine.
Library name (dominantcolors) >
* creating native/dominantcolors/.cargo/config.toml
* creating native/dominantcolors/README.md
* creating native/dominantcolors/Cargo.toml
* creating native/dominantcolors/src/lib.rs
* creating native/dominantcolors/.gitignore
Ready to go! See /Users/mikeli/projects/dominant_colors/native/dominantcolors/README.md for further instructions.

Now let’s loading the NIF. This is what it becomes

defmodule DominantColors do
  use Rustler,
    otp_app: :dominant_colors,
    crate: :dominantcolors

  # When loading a NIF module, dummy clauses for all NIF function are required.
  # NIF dummies usually just error out when called when the NIF is not loaded, as that should never normally happen.
  def add(_arg1, _arg2), do: :erlang.nif_error(:nif_not_loaded)
end

Let’s try to add a simple test case to make sure the calling to Rust module works:

test "add" do
  assert DominantColors.add(1, 2) == 3
end

OK now we have the basic Elixir calling Rust working through rustler.

❯ mix test
.
Finished in 0.00 seconds (0.00s async, 0.00s sync)
1 test, 0 failures

Randomized with seed 733558

Integrate `kmeans-colors`

Now we can start the real work to integrate the Rust library. https://github.com/okaneco/kmeans-colors

So head over to native/dominantcolors.

Running cargo add kmeans-colors to add the dependency.

cargo add kmeans-colors
cargo add palette
cargo add image

Please notice we also added palette and image crate as well because we are going to use it to manipulate images and colors.

Here we have the first version of it. It’s heavily inspired by the kmeans-colors’s cli.

use kmeans_colors::{get_kmeans, Kmeans};
use palette::{FromColor, IntoColor, Lab, Srgb};

// reference
// https://github.com/okaneco/kmeans-colors/blob/master/src/bin/kmeans_colors/app.rs
#[rustler::nif]
fn dominant_colors(file_name: String) -> Result<Vec<String>, String> {
    // Read file into buffer
    let img = image::open(file_name);
    if img.is_err() {
        return Err("Can not open given file".to_string());
    }
    let img_vec = img.unwrap().to_rgb8().into_raw();

    // Convert RGB [u8] buffer to Lab for k-means
    let lab: Vec<Lab> = img_vec
        .chunks(3)
        .map(|chunk| {
            Srgb::new(
                chunk[0] as f32 / 255.0,
                chunk[1] as f32 / 255.0,
                chunk[2] as f32 / 255.0,
            )
            .into_format()
            .into_color()
        })
        .collect();

    // Perform k-means clustering
    let mut best_result = Kmeans::new();
    let converge = 5.0; // 5 for LAB or 0.0025 for RGB; // threshold for convergence.
    let runs = 1; // default just run once
    let cluster = 8; // number of clusters/colors returned
    let max_iter = 20; // maximum number of iterations
    let verbose = false;

    for i in 0..runs {
        // Run k-means multiple times
        let run_result = get_kmeans(cluster, max_iter, converge, verbose, &lab, 0 + i as u64);
        if run_result.score < best_result.score {
            best_result = run_result;
        }
    }

    // Convert Lab centroids back to Srgb<u8> for output
    let color_codes: Vec<String> = best_result
        .centroids
        .iter()
        .map(|x| Srgb::from_color(*x).into_format::<u8>())
        .map(|color| format!("#{:02x}{:02x}{:02x}", color.red, color.green, color.blue))
        .collect();

    Ok(color_codes)
}

rustler::init!("Elixir.DominantColors", [dominant_colors]);

I won’t go very deep on this, but I assume we will need more tweaks later on or expose more knobs to make it more customizable. But for now it should do the basic work.

Then we can register this function in our Elixir module:

defmodule DominantColors do
  use Rustler,
    otp_app: :dominant_colors,
    crate: :dominantcolors

  def dominant_colors(_arg1), do: :erlang.nif_error(:nif_not_loaded)
end

Let’s write a test to validate it.

test "dominant_colors" do
  file = "./test/fixtures/test3.png"
    assert DominantColors.dominant_colors(file) == {:ok, [
        "#bfbdbb",
        "#2f2e2e",
        "#a3a1a0",
        "#d8653a",
        "#eeedec",
        "#171515",
        "#7e7b7b",
        "#4d4b4b"] }
end

Works great 🎨.

Publish to hex

Now let’s publish it to hex and try to use it in Livebook.

In order to publish the package, we would need to add couple of info to the mix.exs

diff --git a/mix.exs b/mix.exs
index 7dd3f49..800a10c 100644
--- a/mix.exs
+++ b/mix.exs
@@ -7,7 +7,9 @@ defmodule DominantColors.MixProject do
       version: "0.1.0",
       elixir: "~> 1.15",
       start_permanent: Mix.env() == :prod,
-      deps: deps()
+      deps: deps(),
+      description: description(),
+      package: package()
     ]
   end
 
@@ -18,10 +20,23 @@ defmodule DominantColors.MixProject do
     ]
   end
 
+  defp description() do
+    "Extract dominant colors from given image (using kmeans), wrapper on top of rust kmeans_colors."
+  end
+
+  defp package() do
+    [
+      name: "dominant_colors",
+      licenses: ["MIT"],
+      links: %{"GitHub" => "https://github.com/29decibel/dominant_colors"}
+    ]
+  end
+
   # Run "mix help deps" to learn about dependencies.
   defp deps do
     [
-      {:rustler, "~> 0.30.0"}
+      {:rustler, "~> 0.30.0"},
+      {:ex_doc, ">= 0.0.0", only: :dev, runtime: false}
     ]
   end
 end

Making sure you are registered or logged in into hex:

mix hex.user auth

Now we can publish the package. If you got the error of following:

** (UndefinedFunctionError) function Hex.Crypto.decrypt/3 is undefined (module Hex.Crypto is not available)

Then you can try the following:

mix local.hex --force
mix hex.publish

If everything goes right, you should be able to see something like this:

Building dominant_colors 0.1.0
  Dependencies:
    rustler ~> 0.30.0 (app: rustler)
  App: dominant_colors
  Name: dominant_colors
  Files:
    lib
    lib/dominant_colors.ex
    priv
    priv/native
    priv/native/libdominantcolors.so
    .formatter.exs
    mix.exs
    README.md
  Version: 0.1.0
  Build tools: mix
  Description: Extract dominant colors from given image (using kmeans), wrapper on top of rust kmeans_colors.
  Licenses: MIT
  Links:
    GitHub: https://github.com/29decibel/dominant_colors
  Elixir: ~> 1.15
Before publishing, please read the Code of Conduct: https://hex.pm/policies/codeofconduct

Publishing package to public repository hexpm.

Proceed? [Yn] y
Building docs...
Generating docs...
View "html" docs at "doc/index.html"
View "epub" docs at "doc/dominant_colors.epub"
Local password:
Publishing package...
[#########################] 100%
Package published to https://hex.pm/packages/dominant_colors/0.1.0 (722129016b397b09a4a58349ab3f60ad12e4b24e09d1c5a301bd08a79ab18977)
Publishing docs...
[#########################] 100%
Docs published to https://hexdocs.pm/dominant_colors/0.1.0

Our package is now available.

Let’s create a dummy to consume it.

mix new test_colors

Then adding the dependency:

{:dominant_colors, "~> 0.1.0"}

Opps, we got an error:

spawn: Could not cd to native/dominantcolors

== Compilation error in file lib/dominant_colors.ex ==
** (RuntimeError) calling `cargo metadata` failed.

    (rustler 0.30.0) lib/rustler/compiler/config.ex:96: Rustler.Compiler.Config.metadata!/1
    (rustler 0.30.0) lib/rustler/compiler/config.ex:78: Rustler.Compiler.Config.build/1
    (rustler 0.30.0) lib/rustler/compiler.ex:8: Rustler.Compiler.compile_crate/3
    lib/dominant_colors.ex:2: (module)
could not compile dependency :dominant_colors, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile dominant_colors --force", update it with "mix deps.update dominant_colors" or clean it with "mix deps.clean dominant_colors"

Looks like we forgot the add the rust files into the package, that’s why it can’t find certain files in the published package. To fix this, make sure to include the rust files:

  defp package() do
    [
      files: [
        "lib",
        "native/dominantcolors/.cargo",
        "native/dominantcolors/src",
        "native/dominantcolors/Cargo*",
        "mix.exs"
      ],
      ...
    ]
  end

With this, we should be able to consume the package.

Rustler Pre-compilation

This works great if the consumer end have the Rust toolchain ready. However, in the context of Livebook this won’t work by default because Livebook do not have the access to the Rust toolchain like cargo.

Luckily rustler_precompiled is exactly trying to solve this.

The TLDR version of rustler_precompied is that it uses a CI server to pre-compile a bunch of binaries, then tell the Elixir to download those binary directly during installation. Think of it as a multi stage process.

Using CI server to pre compile binary. This is necessary to support multiple architecture machine (X86, ARM, Linux, PC, Mac …)
Save precompiled binary along with version number in Github artifacts.
Pointing the Elixir module to download that (with checksum).

Create Github action workflow

First let’s create a GitHub action workflow. Heavily borrowed from rustler_precompilation_example:

name: Build precompiled NIFs

on:
  push:
    branches:
      - main
    tags:
      - "*"

jobs:
  build_release:
    name: NIF ${{ matrix.nif }} - ${{ matrix.job.target }} (${{ matrix.job.os }})
    runs-on: ${{ matrix.job.os }}
    strategy:
      fail-fast: false
      matrix:
        nif: ["2.16", "2.15"]
        job:
          - {
              target: arm-unknown-linux-gnueabihf,
              os: ubuntu-20.04,
              use-cross: true,
            }
          - {
              target: aarch64-unknown-linux-gnu,
              os: ubuntu-20.04,
              use-cross: true,
            }
          # commentted this out as I am experiencing some compilation issues
          #   - {
          #       target: aarch64-unknown-linux-musl,
          #       os: ubuntu-20.04,
          #       use-cross: true,
          #     }
          - { target: aarch64-apple-darwin, os: macos-11 }
          - {
              target: riscv64gc-unknown-linux-gnu,
              os: ubuntu-20.04,
              use-cross: true,
            }
          - { target: x86_64-apple-darwin, os: macos-11 }
          - { target: x86_64-unknown-linux-gnu, os: ubuntu-20.04 }
          - {
              target: x86_64-unknown-linux-musl,
              os: ubuntu-20.04,
              use-cross: true,
            }
          - { target: x86_64-pc-windows-gnu, os: windows-2019 }
          - { target: x86_64-pc-windows-msvc, os: windows-2019 }

    steps:
      - name: Checkout source code
        uses: actions/checkout@v3

      - name: Extract project version
        shell: bash
        run: |
          # Get the project version from mix.exs
          echo "PROJECT_VERSION=$(sed -n 's/^  @version "\(.*\)"/\1/p' mix.exs | head -n1)" >> $GITHUB_ENV          

      - name: Install Rust toolchain
        uses: dtolnay/rust-toolchain@stable
        with:
          toolchain: stable
          target: ${{ matrix.job.target }}

      - name: Build the project
        id: build-crate
        uses: philss/rustler-precompiled-action@v1.0.1
        with:
          project-name: dominantcolors
          project-version: ${{ env.PROJECT_VERSION }}
          target: ${{ matrix.job.target }}
          nif-version: ${{ matrix.nif }}
          use-cross: ${{ matrix.job.use-cross }}
          project-dir: "native/dominantcolors"

      - name: Artifact upload
        uses: actions/upload-artifact@v3
        with:
          name: ${{ steps.build-crate.outputs.file-name }}
          path: ${{ steps.build-crate.outputs.file-path }}

      - name: Publish archives and packages
        uses: softprops/action-gh-release@v1
        with:
          files: |
                        ${{ steps.build-crate.outputs.file-path }}
        if: startsWith(github.ref, 'refs/tags/')

Then add the rustler_precompiled dependency:

  {:rustler_precompiled, "~> 0.7"},

Also add Cross.toml in the native/dominantcolors:

[build.env]
passthrough = ["RUSTLER_NIF_VERSION"]

And the .cargo/config.toml:

[target.'cfg(target_os = "macos")']
rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"]


# See https://github.com/rust-lang/rust/issues/59302
[target.x86_64-unknown-linux-musl]
rustflags = ["-C", "target-feature=-crt-static"]

# Provides a small build size, but takes more time to build.
[profile.release]
lto = true

Now we can push the changes to see if the build is green.

Finally after we validated its build success. We can release it.

git tag 0.1.0
git push --tags

Now we have the binaries built in the CI, we can grab them and use them instead.

Consume pre-compiled binaries

In order to get that, we need to do two things:

First we need to use the precompile macro:

  version = Mix.Project.config()[:version]

  use RustlerPrecompiled,
    otp_app: :dominant_colors,
    crate: "dominantcolors",
    base_url: "https://github.com/29decibel/dominant_colors/releases/download/#{version}",
    force_build: System.get_env("DOMINANT_COLORS_BUILD") in ["1", "true"],
    version: version

Then we need to download the checksum:

mix rustler_precompiled.download DominantColors --all

Now we should be able to use the precompiled version locally.

Please note we also provide a DOMINANT_COLORS_BUILD as a way to escape the pre compiled binary. It’s extremely useful for local development.

Publish to hex

Last step, let’s publish it to hex. Before the publish, we will need to make sure the checksum* files are included in the package, otherwise the installation will be failed with checksum not matched error.

  defp package() do
    [
      files: [
        "lib",
        "native/dominantcolors/.cargo",
        "native/dominantcolors/src",
        "native/dominantcolors/Cargo*",
        "checksum-*.exs",
        "mix.exs"
      ],
      ...
    ]
  end

We can verify the final package contents by using mix hex.build --unpack:

❯ mix hex.build --unpack
Building dominant_colors 0.1.1
  Dependencies:
    rustler ~> 0.30.0 (app: rustler)
    rustler_precompiled ~> 0.7 (app: rustler_precompiled)
  App: dominant_colors
  Name: dominant_colors
  Files:
    lib
    lib/dominant_colors.ex
    native/dominantcolors/.cargo
    native/dominantcolors/.cargo/config.toml
    native/dominantcolors/src
    native/dominantcolors/src/lib.rs
    native/dominantcolors/Cargo.lock
    native/dominantcolors/Cargo.toml
    checksum-Elixir.DominantColors.exs
    mix.exs
  Version: 0.1.1
  Build tools: mix
  Description: Extract dominant colors from given image (using kmeans), wrapper on top of rust kmeans_colors.
  Licenses: MIT
  Links:
    GitHub: https://github.com/29decibel/dominant_colors
  Elixir: ~> 1.15
Saved to dominant_colors-0.1.1

Looks great! Now we can publish it.

mix hex.publish

Using it in Livebook

Add dependency:

{:dominant_colors, "~> 0.1.3"}

Then some Elixir code and using Kino render:

image_path = "/Users/mikeli/Downloads/111ebfdc244345c3.png"
{:ok, colors} = DominantColors.dominant_colors(image_path)

styles = fn color ->
   [
    {"justify-content", "center"},
    {"align-items", "center"},
    {"color", "white"},
    {"display", "inline-flex"},
    {"width", "100px"},
    {"height", "30px"},
    {"background-color", color},
    {"margin", "1px"}
  ] |> Enum.map(fn {k, v} -> k <> ": " <> v <> ";" end)
  |> Enum.join(" ")
end

html = 
  Enum.map(colors, fn color ->
    "<div style=\"#{styles.(color)}\">#{color}</div>"
  end)
  |> Enum.join()

Kino.HTML.new(html)

How lovely!

Summary

I love the fact that Elixir can leverage so much in the Rust land with reasonable effort. It opened a whole new world to Elixir and Livebook users.

Next time, when you can’t find a library in the Elixir land, give some Rust crate and rustler a try.

Credits

rustler The one makes all this possible
rustler_precompiled Makes consuming rust based Elixir lib seamless
kmeans-colors Rust library to get dominant colors.
fast_rss Real world example of RSS parser using rustler.
Teenage Engineering EP 133 Just a beautiful object. And colors too.

What’s next

There are lots of other things we can improve for sure. But not a bad start I would say.

optimize the performance for big images
expose more knobs to tune the result
return percentage of each color (assign weight to the color returned)

📨 Void Inbox

Sat, 18 Nov 2023 08:45:14 -0800

Over the past weekend, I created a web app called VoidInbox using Phoenix and Elixir. Which can let me receive newsletters or other not so important emails, and read them there. It looks like this:

Why am I building this

With the flourishing of the thousands of newsletters. I’ve always found myself lost in the pile of emails in my inbox. Not only it messes with my “actual” emails, but also it’s hard to read them as well.

Plus I am not comfortable to share my personal email address to random newsletters or email lists.

So I am thinking to build one, a simple enough where I can still confidently subscribe to the newsletters I found useful, where I can read them in one place, but also give me the peace of mind of not letting anyone bombarding my email inboxes.

On top of that, I want to share the process of the creation. I truly believe in open source and self hosting software. But I think we can go even further by sharing the creation process. Hopefully even if people don’t find the software itself is useful, they still can benefit from some of the thinking or ideas within it.

I like building things, I think that’s the ultimately form of finding long lasting happiness. Putting them into the writing form not only helps me to have more clear thoughts, but also enforce me to have some level of rigorous. Bottom line is I might found the thing I wrote useful for my future self.

What’s next

Over the next couple of weeks, I am gonna share the pieces by piece how I build this. It would include but not limited to core, models, UI, deployment etc. Hopefully you can get something useful from it or at least get inspired to create something your own as well.

I love the article Teaching Iteration from Jason Fried. Where I think it’s way more relevant from the software projects perspective, as the software we produced most likely would evolve overtime. By learning how to take something from naive and primitive to useful and beautiful takes time and practice, so I am hoping by doing this to practice the iteration process as well.

Here is the general agenda of what I am gonna to share in the next couple of weeks:

Core. SMTP server to receive email and parse them.
Models. Database models and context to power the app.
UI. Build a simple and minimal UI using LiveView.
Deployment. Deploy it to the cloud provider (fly.io) or using docker compose in any machines.
More features …

Stay tuned to subscribe the RSS feed. At the same time, give it a shot to the demo version of it on https://void-inbox.fly.dev/ and let me know what you think.

Or if you can’t wait to get your hands dirty on the source code, you can head over to 29decibel/void_inbox giving it a spin as well.

See you in the next post.

Fall in love with the questions

Fri, 06 Oct 2023 06:16:32 -0700

I came across a great book called The Socratic Method by Ward Farnsworth, which was recommended by Rich Hickey in his talk Design in Practice.

It’s an amazing book, easily the best one I’ve read this year. The ideas in the book have helped me to connect many thoughts I had before on “questions”.

The Socratic method is a style of thought and a method of inquiry that involves asking questions to stimulate critical thinking and to uncover underlying assumptions. It’s not about definitively answering hard questions, but rather about asking and pursuing them in order to arrive at wisdom.

One change I made every morning this year is to change TODO items to questions. Instead of writing “Fix the issue of slow builds”, I write “Why the our build takes ages to finish?”

On the surface, it might look very subtle or maybe even jokingly funny (obviously we can’t change every TODO into questions as well like booking a dental appointment for my kids). But I found myself way more engaged to solve the problem than merely following the directions.

More importantly, it has me naturally asking more and more questions. So you could easily see the trajectory of the problem solving process. By the end of the day, you just somehow feel you’ve learned more than merely solving that one problem. Those questions linger in your mind way longer than the TODO list.

One known common technique we use in postmortems is Five Whys. Sometimes it feels intense (interrogative), but most of the time, everyone ended up with a more clear picture of the overall situation and concrete ways to prevent it in the future.

At the same time, asking questions couldn’t be more relevant in the age of LLMs/ChatGPT. I once shared with my colleagues that I think the era of “fearing to ask stupid questions is gone”. Thanks to ChatGPT, anyone can ask any questions anytime without worrying the embarrassment.

I certainly found myself asking many more questions this year thanks to ChatGPT. Partly because lots of questions I have are so customized or context specific which was very hard to search or even find the right audience to ask. So I just gave up asking it all together. But now I can just briefly layout the situation and the challenge, let the LLM handle the rest of clarification or refinement of the questions.

Yes we might ask many bad questions, but aren’t they better than no questions?

Every time you ask and answer good questions, your understanding gets a bit deeper. You better understand the other side and the weaknesses on your side. You see more complexity.

A question puts pressure on whoever receives it. If you ask questions of yourself, you are the recipient of the pressure. That’s good. Stating an opinion is roughly the opposite. It releases pressure. Pressure is uncomfortable, so most people think and talk in opinions.

The Socratic Method

One quote I really like is “Fall in love with problem, not the solution.” (Oh, there’s even a book now)

So let’s “Fall in love with the questions, not the answers.”

Bun + Prisma

Sun, 17 Sep 2023 09:00:30 -0700

I just can’t stop having fun using bun

I have an old postgresql database backup. I need to quickly inspect the tables/models in the database and then extract data from it.

Spin up postgres

After having the backup sql ready output-backup.sql. I created a docker-compose.yml file to quickly spin up a postgres instance.

version: '3.1'

services:
  postgres:
    image: postgres:latest
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: example_dev
    ports:
      - "54321:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Now let’s re-dump the data back into the local database.

psql -U postgres -h localhost -p 54321 -d example_dev -f output-backup.sql

Okay the fun parts begin.

Bun + Prisma Fastlane

Set up a new package using bun:

mkdir data-explorer
cd data-explorer && bun init -y

Now let’s add prisma:

bun add prisma
bunx prisma init --datasource-provider postgresql

Remember to change the DATABASE_URL to our local one in .env:

DATABASE_URL="postgresql://postgres:postgres@localhost:54321/example_dev?schema=public"

Inspect the database now:

bunx prisma db pull

Generate prisma client:

bunx prisma generate

Hooray!

Now we can play with the database:

import { PrismaClient } from "@prisma/client";
const prisma = new PrismaClient();

const allUsers = await prisma.users.findMany();
console.log(allUsers.length);

See it now:

bun run index.ts

The beauty of Prisma is that the generated schema.prisma is like a X-Ray to the database. Not only we can see the tables and relations, but also we can manupulate it in a type safe way (with autocompletion of course).

Such a joyful experience.

Summary

I still can’t believe how easy it is to run a TypeScript project using bun with massive npm modules without losing my mind.

References

https://bun.sh/guides/ecosystem/prisma https://www.prisma.io/docs/getting-started/setup-prisma/add-to-existing-project/relational-databases/introspection-typescript-postgresql

Playground

Sat, 09 Sep 2023 09:32:46 -0700

I love tmp folder driven development, it gives me the freedom to experiment without worrying about polluting things.

tmp folder driven development:

Start with tmp folder and messing around libraries or scripts before put it into the actual project.

Recently I’ve had a great time using bun:

Develop, test, run, and bundle JavaScript & TypeScript projects—all with Bun. Bun is an all-in-one JavaScript runtime & toolkit designed for speed, complete with a bundler, test runner, and Node.js-compatible package manager.

All that means to me is that I can write straight-up TypeScript and using any npm packages without messing around with endless configurations and build tools.

Idea

While I am building a desktop app, one feature I want to have is to get the cover image of the pdf file.

So, instead of immediately adding npm packages directly to my project. I want to quickly try it out first.

Playground

Playground is such an interesting and powerful concept if you think about it. It’s safe, sandbox-ed.

To set up a playground for my experiment. I did the following (thanks to bun):

cd /tmp
mkdir pdf-cover
bun init -y

That’s it. Now I can just start writing TypeScript and start adding libraries.

Assumption

The assumption is to read the first page of the pdf and convert it into an image. As that’s where most of the pdf (especially books) place their cover image.

Experiment

To extract the first page of the pdf, we can use pdf-lib.

bun add pdf-lib

Here is the code:

const pdfFilePath = "example.pdf";
// Read the existing PDF into a buffer

const existingPdfBytes: Buffer = fs.readFileSync(pdfFilePath);

// Load the PDF with pdf-lib
const pdfDoc = await PDFDocument.load(existingPdfBytes);

// Create a new PDF document
const newPdfDoc = await PDFDocument.create();

// Copy the cover page from the existing PDF into the new PDF
const [coverPage] = await newPdfDoc.copyPages(pdfDoc, [0]);
newPdfDoc.addPage(coverPage);

// Serialize the PDF to bytes and write it to the file system
const newPdfBytes: Uint8Array = await newPdfDoc.save();
const coverPdfPath: string = "cover_page.pdf";
fs.writeFileSync(coverPdfPath, newPdfBytes);

Running our script is dead simple:

bun run index.ts
# Because it calls graphicsmagick underneath
brew install graphicsmagick

Now we get the cover page pdf. Next let’s try to convert it into an image.

To do that. We add pdf2pic package.

bun add pdf2pic

pdf2pic can convert existing pdf to an image given some options.

const coverOutputPathParts = parseFilePath(coverOutputPath);
 const options = {
   density: 200,
   saveFilename: "cover-image",
   savePath: "./",
   format: ".png",
   width: 600,
   height: 600
 };
 const convert = fromPath("cover_page.pdf", options);
 const pageToConvertAsImage = 1;

 await convert(pageToConvertAsImage, { responseType: "image" });

Then let’s run our script again using bun run index.ts. Cool, now we get the image we want.

However, the image size is not well-proportioned as the original pdf dimensions. So we need to figure out the right dimension of the original pdf.

We can then go ahead write a function to figure that out:

async function getCoverPageDimensions(pdfFile: string): Promise<{
  width: number;
  height: number;
}> {
  // Read the existing PDF into a buffer
  const existingPdfBytes: Buffer = fs.readFileSync(pdfFile);

  // Load the PDF with pdf-lib
  const pdfDoc = await PDFDocument.load(existingPdfBytes);

  // Get the cover page
  const coverPage = pdfDoc.getPages()[0];

  // Get the dimensions
  return coverPage.getSize();
}

Modularize

Now we have the proof of concept, without any of those fighting with modules, TypeScript setup, build configurations, ts-node.

In order to consume this, let’s turn it into a module (a file with exported functions) instead. So that it can act as a well isolated unit.

Let’s define the interface we want:

export async function extractAndConvertCoverPage(
  pdfFilePath: string,
  coverOutputPath: string
) {}

Also as you notice from the previous code, it’s quite messy and inefficiencies. Which is totally fine, because that’s the point of playground. We want to get to a working version first and validate our ideas, then into the optimization phase.

The beauty of the interface is that we can hide all the inefficiencies behind the scene as well. So we can improve it without impacting the callers.

After some optimization here is the final code:

import fs from "fs";
import { PDFDocument } from "pdf-lib";
import { fromBuffer } from "pdf2pic";
import path from "path";

const OUTPUT_IMAGE_DPI = 200;

export async function extractPDFCoverImage(
  pdfFilePath: string,
  coverOutputPath: string
) {
  try {
    // Read the existing PDF into a buffer
    const existingPdfBytes: Buffer = fs.readFileSync(pdfFilePath);

    // Load the PDF with pdf-lib
    const pdfDoc = await PDFDocument.load(existingPdfBytes);
    const pdfDimensions = await getCoverPageDimensions(pdfDoc);

    // we need a new pdf doc to get buffer which will be used by pdf2pic
    const newPdfDoc = await PDFDocument.create();

    // Copy the cover page from the existing PDF into the new PDF
    const [coverPage] = await newPdfDoc.copyPages(pdfDoc, [0]);
    newPdfDoc.addPage(coverPage);

    // Serialize the PDF to bytes and write it to the file system
    const newPdfBytes: Uint8Array = await newPdfDoc.save();

    const newPdfBuffer = Buffer.from(newPdfBytes);

    // now we can start converting the pdf to image
    const coverOutputPathParts = parseFilePath(coverOutputPath);
    const options = {
      density: OUTPUT_IMAGE_DPI,
      ...coverOutputPathParts,
      ...pdfDimensions,
    };
    const convert = fromBuffer(newPdfBuffer, options);
    const pageToConvertAsImage = 1;

    await convert(pageToConvertAsImage, { responseType: "image" });
  } catch (err) {
    console.error("Error:", err);
  }
}

function parseFilePath(absolutePath: string) {
  const ext = path.extname(absolutePath).substring(1); // Get extension without the dot
  const filename = path.basename(absolutePath, `.${ext}`); // Get filename without extension
  const directory = path.dirname(absolutePath); // Get directory path

  return {
    saveFilename: filename,
    savePath: directory,
    format: ext,
  };
}

async function getCoverPageDimensions(pdfDoc: PDFDocument): Promise<{
  width: number;
  height: number;
}> {
  const coverPagePdf = pdfDoc.getPages()[0];
  // Get the dimensions
  return coverPagePdf.getSize();
}

Gist link

Learnings

All-in-one tools are great, fast tools are great. Not because they are more efficient, which of course they did. But more importantly, they don’t get in the way between you and your ideas.

Ideas are fragile.

Don’t get lost before you even start experimenting and cultivating your ideas.

Let’s do more tmp folder driven development, let’s do more playgrounds, let’s do more bun 🚀!

How do you feel

Sat, 02 Sep 2023 09:49:18 -0700

With every single sophisticated tool nowadays, we are constantly measured by numbers: how well we perform, how well we accomplish certain tasks, how effective we were in our training.

However, we still rarely measure how we actually feel.

The first time I encountered this question was actually from my Suunto watch. By the end of the running workout, on top of showing the running summary, it will ask, “How was it? (Excellent, Good, OK, Poor ..)”.

Suddenly, I realized that what we feel matters. Because there are just so many things stats and completion rates cannot capture. And how we feel matters in the longevity of the training. Sometimes, even if the stats show perfect numbers, but if we constantly feel terrible, we might risk ourselves getting injured or burnt out.

Asking “How do you feel?” can create a sense of being cared for, which may enhance the psychological benefits of exercise and contribute to a more positive overall experience.

At BetterUp, I love the conscious checking part of our sprint retro meeting, which I never had before. In the past, all we cared about was how many tickets we closed, how well we could estimate the next sprint, but no one cared about how you feel.

To be clear, I don’t think asking “how do you feel” is the ultimate cure for team productivity, but I can say, I feel much more motivated and recharged after just casually chatting about our feelings, or putting those in my journal.

The same can apply to the products we use. Sometimes we can’t articulate why we prefer certain tools over others, but they just make us feel good (Apple WWDC 2013 - What do we want people to feel?).

Next time, try to ask more of yourself (or others), “How do I/you feel?”

So the next time you find yourself overwhelmed by metrics and numbers, take a moment to step back and listen to your feelings. After all, a little emotional self-check can make the journey, whether it’s a sprint retro or a morning jog, all the more meaningful.

Tools

Sat, 26 Aug 2023 09:46:48 -0700

Great tools enable us to focus and unleash our creativity. They either allow us to do things we couldn’t do before or provide us with new perspectives we hadn’t considered.

Here are a couple of tools that have stayed with me for a long time, and I am grateful to have them. I am a macOS user, so some of these tools are only available on this platform. However, you can find alternatives on other platforms as well.

Alfred

https://www.alfredapp.com

Alfred is an award-winning app for macOS which boosts your efficiency with hotkeys, keywords, text expansion and more. Search your Mac and the web, and be more productive with custom actions to control your Mac.

To me, Alfred is an integral part of macOS; it’s the first thing I install on any macOS system.

It’s a launcher, but with an extremely customizable workflow. I love its simplicity; everything is displayed in a list format. Each item on the list contains just three elements: title, description, and icon.

While at Amazon, I even created an Alfred workflow for internal use, allowing developers to search for packages, pipelines, wikis, and contacts. Thousands of developers at Amazon used it.

You can easily create workflows integrating with your company’s unique tools or workflows as well using any programming language.

Raycast is the new kid in this space, offering more sophisticated and rich user interactions. I tried it a couple of times but eventually returned to Alfred for its simplicity.

iTerm2

https://iterm2.com

iTerm2 is a replacement for Terminal and the successor to iTerm. It works on Macs with macOS 10.14 or newer. iTerm2 brings the terminal into the modern age with features you never knew you always wanted.

Solid terminal with very few surprises. It’s just so fast.

Warp is the new kid in this space, but I found it to be “too helpful” at times.

Tmux

https://github.com/tmux/tmux

tmux is a terminal multiplexer. It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal.

We usually work in a “workspace” manner, meaning we run multiple tasks for one project—logs, frontend, backend, monitoring.

Tmux is like the express lane that enables you to quickly jump back into (restore) the context you left. You can start a development session with multiple tasks running—using Vim, tailing logs, and configuring the frontend—in a single session.

The beauty is the next day, you can instantly back to the entire context where you left the day before.

iA Writer

https://ia.net/writer

iA Writer gives you the essence of pure writing by providing an unobtrusive interface that lets you concentrate on your text. Despite its minimalistic approach, the application offers a variety of distinctive features that are specifically designed for writing-focused tasks. With iA Writer, you can gain unparalleled insight into your writing structure, detect superfluous words and clichés, and navigate through your documents and notes with ease.

There is no other writing tool in the world better than iA Writer for me. It’s the one and only tool I use when writing long-form content (including this one of course) or when extreme focus is needed.

It’s fast, stylish, and unobtrusive. Something about this tool makes you feel like you are connecting with the words, and nothing else matters.

If you don’t know what I am talking about, you should try it.

DayOne

https://dayoneapp.com

I don’t write journal everyday. But I still find the joy whenever I re-read the memories I’ve captured. It’s local-first software, simple and elegant.

Even if I use paper notebooks for many journals, I still frequently use DayOne, especially when I’ve had great times with the kids, so I can include pictures or even voice recordings.

Syncthing

https://syncthing.net

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers in real time, safely protected from prying eyes. Your data is your data alone and you deserve to choose where it is stored, whether it is shared with some third party, and how it’s transmitted over the internet.

This tool is the unsung hero in the background. It’s a P2P, reliable, fast syncing tool.

I use it to synchronize folders among multiple macOS devices. I normally setup a special folder that allows me to share things between my work and personal laptops. I’ve never encountered a single issue. It even worked flawlessly when I traveled back to China and tried to sync things from my Mac mini back in US.

Logseq

https://logseq.com

Logseq is another local-first outliner-styled note-taking software. It took me a while to embrace its outlining nature. After using it for a year, I’ve grown to love its outline aspects. This limitation genuinely forces me to think more clearly and coherently.

You can link notes or ideas from each other, and even make PDF notes right inside Logseq. Best of all, all data is stored locally in your selected folder in Markdown (or Org) formats.

Tools are a personal thing, things works for me might not get the love from others, which is fine. Keep exploring and creating.

Minimal setup time for your new project

Fri, 25 Aug 2023 15:29:43 -0700

I remember the early days of Rails when people often criticized the Java world, or the J2EE world to be more specific, for being too complex and over-engineered, even for simple projects.

Slowly, the whole industry is now shifting more towards bootstrapping, being lean, and scrappy.

Any time people try to set up a solid foundation, they are accused of wasting time; any time people try to create things with scalability in mind, they’re labeled as over-engineering.

The truth is, a solid foundation is not over-engineering; its constraints help you focus on the problem, which can actually increase your creativity. A good foundation is about eliminating unnecessary options; it’s about focusing on doing things in a limited number of ways—sometimes, the only way.

Returning to the Rails example, yes, it’s opinionated, but there is a well-established pattern for doing certain things. Controllers, actions, models, and templates are all the tools we have. Try to only use these tools and start creating.

Another example is React. Compared to other frameworks, its success largely stems from its limitations—the very few ways of putting pixels onto the screen.

A good, solid setup at the beginning is not a waste of time; it saves time both in the short term and long term. In the short term, it boosts your confidence and unleashes your creativity; in the long term, it ensures that your idea can still work.

When starting a new project, try to set aside time for minimal foundational setup. Do not gloss over this important step.

It can take you further.

The Future Computers

Mon, 05 Jun 2023 18:02:40 -0700

Such a great read on Jason Fried’s Two visions of the future.

Though, I am not sure how I feel about either of them.

Sometimes I wish there were more easy to use UIs to get things done. Sometimes I wish all I needed to deal with was text, instead of learning all kinds of fancy UIs from your designers.

The best computers are the ones that disappear or blend into your life. This is even often mentioned in Apple’s own marketing.

I appreciate the silence of Apple computers, from the early Mac mini to today’s Apple silicon computers. From this sense, it’s fantastic to be able to work without thinking about computers.

On the other hand, this VR thing straps a whole bunch of UIs in front of my eyes.

I think VR would be great for meditation, for games, for emerging experiences. Which really is about as little UI as possible, but rather trying to create whole different environments.

But staring at large rectangular “windows” by strapping on a device? Maybe not. At least for me.

Oh, and for now, I would be leaning towards a more headless (UI-less) future.

April 15 2023

Sat, 15 Apr 2023 09:10:11 -0700

We [at Apple] feel that, for some crazy reason, we’re in the right place at the right time to put something back. And what I mean by that is, most of us didn’t make the clothes we’re wearing, and we didn’t cook or grow the food that we eat, and we’re speaking a language that was developed by other people, and we use a mathematics that was developed by other people. We are constantly taking.

And the ability to put something back into the pool of human experience is extremely neat. I think that everyone knows that in the next ten years we have the chance to really do that. And we [will] look back—and while we’re doing it, it’s pretty fun, too—we will look back and say, “God, we were a part of that!”

We started with nothing. So whenever you start with nothing, you can always shoot for the moon. You have nothing to lose. And the thing that happens is—when you sort of get something, it’s very easy to go into cover-your-ass mode, and then you become conservative and vote for Ronnie. So what we’re trying to do is to realize the very amazing time that we’re in and not go into that mode.

Steve Jobs Archive

April 10 2023

Mon, 10 Apr 2023 08:39:11 -0700

Art is about the maker

Its aim: to be an expression of who you are.

You are creating the work that best represents you.

Wanting to outperform another artist or make a work better than theirs rarely results in true greatness. Nor is it a mindset that has a healthy impact on the rest of our lives. As Theodore Roosevelt pointed out, comparison is the thief of joy. Besides, why would we want to create with the purpose of diminishing someone else?

Great art is an invitation, calling to creators everywhere to strive for still higher and deeper levels.

Rick Rubin

The Creative Act: A Way Of Being

April 08 2023

Sat, 08 Apr 2023 10:35:28 -0700

If we can tune in to the idea

of making things and sharing them

without being attached to the outcome,

the work is more likely

to arrive in its truest form.

Rick Rubin

The Creative Act: A Way Of Being

Griffith Observatory, LA

Sat, 08 Apr 2023 10:30:03 -0700

10/30/2022

Weir Canyon Trail

Sat, 08 Apr 2023 10:30:03 -0700

3/26/2023

First article using Bamboo 🎋

Sun, 15 Jan 2023 16:13:08 -0800

I am really tired of the unnecessary ceremonies of writing a new article using my beloved static site generator:

open terminal
find the folder
create a new file with the title
open editor to log down my thoughts
git add, git commit and git push

Instead of doing all of these, how about something which you can just open, write and publish (while still using the greatest static generator).

Yeah, that’s Bamboo 🎋, a little macOS native app I am working on so I can write more in 2023 ✍️.

Next Journey

Fri, 02 Dec 2022 17:10:20 -0800

I promised to share what will be my next journey. But before that, I want to share 3 personal stories.

Story 1

In 2015, while I was living in Foster City Bay Area, I met my neighbor Robin. At that time, I started running on and off, but unfortunately in the every possible wrong way (this simple sport seems not simple at all). Turns out he’s a seasoned ultra and sub-3 marathon runner, most importantly, he’s patient and willing to guide me through.

He invited me to the BURN running community, started coaching me on how to run the first marathon (The longest distance I ran before that was 5k). He made plans for my 3 months training, check-in after every hard workout, helped me through the injured times and how to recover from it.

Eventually, in the December CIM (California International Marathon) race, he paced me the whole race, and I finished in 3 hours 17 minutes. I can’t even possibly think of finishing 26.2 miles for the first year of running, not even to mention under 3:30!

Story 2

At the beginning of this year. My son Aaron saw I had a brand-new notebook (LEUCHTTURM 1917) sitting on my bookshelf. He asked me whether he can have it. I thought this would be a great way to nudge him to start journaling, also an opportunity for me to journal more often (which I only do one or two times per week). So I created a little challenge with him.

The rule is if he can journal 30 days in a streak, then the notebook would be his; at the same time, I will start journaling with him as well, if I skip any day, the notebook will also belong to him. So while giving me some motivation, it also makes the challenge a little more fun.

Fast-forward to now, Aaron finished his 30 days streak (even on his sick days), he is still journaling now and then when finding interesting things or back from a family trip, he got his notebook of course. For me, I journaled every single day since that. Its Aaron helped me “installed” this micro habit which I don’t even need to think about it before bed now.

Story 3

I’ve always wanted to start blogging again (last time was still in college). But always put it off for all kinds of reasons. In James Clear’s book Atomic Habits, he shared that he’s making bets with his friends to encourage himself to build the habits. So I thought to give it a try. Then I sent this message to my friend Jason on Telegram on Jan 25, 2022.

I will write 5 articles by end of Feb 10, and published it. Otherwise I will give Jason Lin 1000 dollars.

Yes, I finished 5 articles before Feb 10 (Cause this guy published the message to LinkedIn, which turns it into a thing not about money but pride).

But I still vividly remember, after the 3rd article, I felt like running out of steam. I don’t think I would have finished the last 2 posts if there weren’t someone out there expecting the writings from me (and the 1000 dollars, and the post).

Connecting the dots

Maybe these stories seem random to you, but to me, they are all connected, showing the importance of human relations, the importance of even simple message check-ins with real human beings once in a while.

As people working in software and technology, all the time, often over-estimate the power of machines and under-estimate the power of human beings. At least that’s something blinded me for such a long time.

Forgot about all the apps, websites, fancy tools, it’s only taking another human being to transform yourself, even if (especially) if it’s a kid.

I am happy to start my next journey at BetterUp, which I believe, could help more people transform themselves through other human beings. No matter if BetterUp can success or not, but at least, we can redirect some attention to human relationships, not just technologies.

Journey at Amazon

Thu, 01 Dec 2022 15:05:12 -0800

I’ve never imagined that I could stay at one company for almost 8 years, let alone Amazon.

But here I am, last week I finished my longest career tenure at Amazon. It was a grateful, intensive yet fulfilling journey.

Over the years, I’ve realized that how much time you spent on one place is not about that company, but more about people around you. I felt so lucky to have a great(best) manager, a stable and caring team and lots of super smart and talented people there. We tackle one problem after another, creating new initiatives, cancel projects, start new ones. All those ups and downs with the team are just a valuable experience for me. Those people constructed my whole working and growth experiences, those experiences and opportunities shaped who I am. It gave me new perspectives on seeing the path ahead of me.

I am so grateful for working with my hiring manager (Krishna G) for about 6 years. While I was still hesitating joining the team back in 2014, he said to me over the phone: “You will not be bored here.”. Ok ok, I cheated a bit, he also said to me “You will not be doing on-call”. But anyway, it turns out both of what he said to me were true, I got the opportunity to do dedicated front-end work and improve a lot of our customer and developer experiences, and I had a intensive and yet very enjoyable fulfilling experiences solving all kinds of problems. He’s always trying to motivate me to find the next thing to create/improve to make our customers happier while leveraging my own interests, most importantly, giving me the freedom to do so. I once shared in my yearly review that “Krishna is like my charger, who can always find ways to boost my energy level to 100%."

I always tell me friends that don’t expect learning secret technologies at Amazon (I am sure there are of course), cause that’s not the point. The secret of Amazon is all about learning and exercising the LPs (Leadership Principles). I am grateful that I got the chance to really understand and breathe the meaning of them in the day-to-day work. All the technical choices, product initiatives, ideas, innovations start with LPs. By arming with those north-star values plus the relentless emphasis on writing (which is really just critical-thinking), making Amazon a unique place to be.

Like all other journeys, you will encounter a forking point someday. It’s not about how anything wrong with the current trail, but the fork leading to the new path just seems so much more attractive and aligning with my believes and core values better. Which I will share more in the next post.

Championships, coaching and zen

Thu, 10 Nov 2022 10:28:43 -0800

There are books which its title can’t justify the greatness of its content.

The legendary coach Phil Jackson’s book Eleven Rings - The Soul of Success is definitely one of those.

I’ve known about Phil Jackson since middle school while watching NBA playoffs of Bulls and Lakers, he had coached the best basketball players like Micheal Jordan and Kobe Bryant, lead Bulls and Lakers to win 11 NBA championships across his entire coaching career. And yes I know at that time people called him Zen Master. Of course, I had zero idea or interest on what’s that even mean when I was a teenager back then.

I’ve always been put off by those kinds of books with “success” in the title, at the same time though, I’ve also attracted by the books with “soul” in the title. Mostly because I’ve been holding the idea of in-directness, or achieving things in the natural way too tight. Seeking success without meaningfulness to me seems unnatural and uncomfortable. But eventually I decided to pick up this book, it turns out that every bit of it is as natural as possible on leaderships, coaching and self actualization.

As a software engineer, I’ve been more and more realized that the importance of coaching, the relationships played to achieve your self actualization. During my past 15 years of career, I’ve had the most significant satisfaction and growth in the past 8 years at Amazon. I would attribute that to my manager, my mentor and friend Krishna G without a blink. Great manager or mentor are the ones helping you to find your own purpose and meaning to reach your actualization. Looking back, there are so many overlaps between Phil Jackson and Krishna in terms of coaching philosophy, making someone to become better themselves while contributing to the greater goals of the team. I feels extremely lucky being able to find the harmony working and delivering with a manager/teacher like him.

Maybe that’s one of the reason why I find this book resonate with me such given my experiences with Krishna.

We all often put too much attention on the heroic individuals, the super stars, the one man changes everything.

After reading the book, I had much more admiration and respects on Phil Jackson’s ability to accelerate Micheal Jordan and Kobe Bryant’s process of becoming a leader, enabling them to get the bigger achievements than before. Most importantly, to expedite the process of those super star’s actualization. Somehow, it made me feels the super stars are “less magic” than before.

I’ve learned so many from the book about finding the internal drive of working, about managing angers, about leadership, about team building, about finding your inner peace, about how to making choices, about achieving your self actualization. It’s a combination of the last second shot excitement and spiritual Zen-ness philosophies.

It’s philosophical while down to the earth.

I feel lucky that I opened the book and happy to find the reason why some of my experiences makes me happy or sad. Above all, it makes me cherish more about the people I am working with, the current moment.

Beginners Mind

Thu, 06 Oct 2022 16:18:02 -0700

Sometimes, not knowing something exists is a blessing.

How many times have you got discouraged once you know your “awesome idea” already being implemented by others, maybe also better in every other ways than yours as well.

When I was a junior software engineer, I always hope I could’ve known more. Hopping that I could’ve known there are already solutions to the problems in my head in all kinds of forms. So that I don’t need to re-invent the wheel.

After doing software development for a while (little more than 15 years), also being a customer of using all kinds of software. I realized one thing: No matter how simple and ubiquitous other software behaves and looks. Turns out, the thing you build will almost always slightly different than others.

My point is, people choose using different tools for lots of reasons, sometimes for certain functionalities, sometimes for certain UX, sometimes even just for the joyfulness the tool sparks.

Every single tool has a different taste in it. Because the human who made them are different.

Never stop creating. Don’t be afraid of re-inventing the wheel, especially in the software domain.

Best part is even if you somehow produced something exactly as someone else already built (which is very unlikely), the process of turning something from 0 to 1 will well worth it.

The Missing Helper for Your Static Blogs - Part 1

Thu, 06 Oct 2022 15:36:11 -0700

Notes: This is a documentation-driven-development series I am currently experimenting. So most likely the code doesn’t exists yet before publishing this article. What I am hoping is that even if the project eventually can not be finished, the idea could still live in somewhere.🤞🏼

Static site generators are a blessing. Since Jekyll and Github pages, the world had been moving towards static sites in big strides. To say the least, personal blogs are among the majority of them.

Static site generators are great in so many ways, the best parts probably is the 1. endless customizability and 2. content first.

However, there comes a cost. Thinking about the process you writing a blog post:

Open iTerm
cd ~/sites/orchardlabdev-site
hugo new content/posts/i-have-something-to-share.md
code .
Locate the new file and start writing.

Not to mention the specialized tools we are using here, to start writing something, we would need to do 5 steps. Of course you can have some shortcuts to streamline this process. But then in the writing mode, if you want to put a picture, you would need to drag it drop to a certain folder then grab the name and link it correctly in the markdown file. So much frictions to write!

So I am wondering, can we have a little helper doing some of these (if not all of them) for us? Something can make us writes more, think more, instead of fiddling the tools and facing 10 options. I would love to have something like this:

open a link
start writing
Drag and drop will automatically put the images in the right folder and insert the image link as well
After all done I can publish (which commits with message provided and then push to remote).

Can we build something like this? So that’s my plan.

Shape of the tool

I would like the tool to be a small binary. Which I can fire up in the static site folder.
It can automatically detect the static site generator according to the folder structure (Hugo or Jekyll).
The main editing interface would be a web UI which we can:
- Write
- Edit
- Publish
(Optional) Basic authentication of the web UI. This is not required at all at the beginning but would be nice later on if we want to deploy it to the cloud.

Choosing the tools

For the binary I would like to use Rust, cause that’s a good one if you want your tool to show up on the HackerNews front page :-). Kidding aside, I think Rust is great at building cross-platform binary tools, and it’s fast too.

For the Web UI, I will use Svelte.

High level design

File based. It’s all file based, meaning the file on the disk is the single source of truth. There is no database evolved at all.
Rust will read the local folder structure and convert them into domain models.
Key domain models here would be posts which correspond to the local markdown files
Each post will have the common metadata cross different static site generator tools
- title
- author
- published at
- content
- layout (this can be configured / changed)
Folders contains the markdown files will be called collection
New posts will automatically goes into the collection (if there is only one), otherwise user can choose where the default writing goes to
Rust exposed RESTful API for our Svelte Web UI to use
- List collections
- List posts among each collection
- Create new post (markdown file)
- Save/update new post (markdown file)
- Delete a post (markdown file)
- Publish (which will trigger local git commit)

Since I am currently using Hugo, so I would like to support Hugo first and then increase the scope of the tool to others like Jekyll or 11ty etc.

What’s next

Next we will be start building the core logics using Rust first. See you next post.

Software Sometimes Is Annoying

Sat, 16 Apr 2022 10:25:35 -0700

This is the start of the story. Unfortunately, this is the end of the story too.

About two months ago, this the above screen popped up in my Tesla. I thought, well, look how smart this car is. It can even tell me the tire’s wear conditions without using coins or special tools to measure. Software is amazing!

I took the car to the dealer, as expected, they replaced two brand new tires on the rear side, since the front two tires are still pretty decent, so they kept it there (to save me some money). Awesome, brand new tires, peace of mind.

The only problem is, the warning still stays there. Because the rear side is too new, so the difference between the front and the rear is still big enough than the software’s configuration. And there is no local reset or override option in the car’s software system itself, this is controlled by the “centralized” configuration in the Tesla software. The service person explained to me that the other Tesla team was well aware of this issue and working a fix in the future software updates, but they (dealers) have no way to fix it.

The upside is that this could be fixed automatically over the wire through a software updates. The downside is it might takes a long time or forever.

As a software engineer, a change might not be that strait -forward than it looks. Yes, it’s a configuration change, but I guess we can’t just change the configuration globally (again I am not a car person, so maybe what I am going to say here is totally rubbish). There will be discussions around should we allow local overrides or not, how much buffer should we allow, how do we expose this local overrides, shall we only allow technician to do it or any car owner to do it. Once we finalized the design for the change, we would allocate resources to design the UI and put the work in the pipeline. Anything more important happened could again de-prioritize this and move it into the backlog.

Everyone has a different opinion regarding to the importance of a software fix. From customer’s perspective, any feature we rely on is worth the fix. But what bothers me the most is the “false positive” signal it shows (all the time).

Because of this always on warning on the dash, after couple of weeks of driving, it numb my brain to pay any attention to any warnings anymore. Cause I will by default treat anything showing up there as “false positive”. Depending on the person, it might make some people more ignorance or more stressful.

I can’t remember even once in the past that I sent my car to the dealer to clear warnings in the dash ended up they are saying: “Sorry our software teams are working on it, we can’t do anything about it now.” As a software engineer, this makes me sad. I can feel even a simple decision we made might makes some users miserable using it.

Software in a way empowers people to have more customizations than the analog world, but at the same time, it also makes us feels helpless by using it. Why is that?

I think there are so many factors involved. There is no single one or two things 100% determined the way software has to be. I will just share some of my incomplete thinkings around this.

I think one of the reason is the centralization, software becomes more and more centralized, there is only one team working on this significant feature which applies to every car running on the road. Every company eventually becomes a software company. People design car softwares in California might rarely think of people driving it everyday in the freezing places. They are designed centralized while distributed globally. Why it has to be centralized and uniformed? Because it’s cheaper to do so.

Another reason I think is the lacking of standard or componentized structure. Replacing a compatible wheel or tire is way more strait-forward than replacing a piece of software component. Because most of the time, tangible things are an open book while the software is a black-box. From this perspective, I think WebAssembly might help in the future software design, especially for things with interchangeable components. Yes, there is for sure no silver bullet.

I am a huge fan of Dynamicland. One thing I interpreted is that it exposes the black-box software to the outside. And enable people to change, transform and collaborate on top of it. At the same time, software enables the dynamism and make the regular object like a living thing. I don’t know if and how this could applies to the car software, but I would be more than happy if it can enable me to get rid of that annoying warnings.

Thick-client software is coming back ?

Mon, 11 Apr 2022 17:33:59 -0700

We are living in a world of saturated SaaS products. There is always a service for that. Every single tool we’ve used to install on our machine now have a version in the cloud. From notes taking (Notion) to photo editing (Adobe) to designing tools (Figma) to even coding (VS code in the cloud). I think it’s fair to say, SaaS services pushes the instant gratification of getting things done to the limit. By the end of the day, all you need is a browser and a URL.

Of course this is not the complete story. Along with this convenience, you also have to accept couple of sacrifices as well. The always on connection, losing (at least part of) privacy, and dependency of the running service are among the most notable ones.

I can see there is an interesting come back of the original thick client software coming back. But this time is different than the prior web 2.0 days. Before web 2.0 we are doing thick client because of low performance of the browser or the lacking of interactivity of the web.

This time however, thick client or another cool term Local-first software is mainly caused by privacy concerns, decentralization of the future. It’s not all about performance, although that matters as well. But mainly we want something to be firstly working on my local machine, then we want that cloud-enabled ability to collaborate as well.

Please remember local-first doesn’t mean it’s built using native technology, although that’s the natural choice because local means close the native platform it provides. We can still leveraging web tools or browser to develop local-first software as well. As a web developer, I am particularly interested in building offline first software using web-assembly recently. It enables us to reach most of the people while not losing too much on the native performance.

Yes, there will be tons of people are willing to make all of these three sacrifices given it’s so compelling to just pay and forget about it. Software at the end of day, maybe to most of the people, is just a tool to get things done, to make some zeros to ones or vice versa. Also local first software might not applying to everything. For example, it doesn’t make too much sense to make a food ordering service offline first. But I think local-first software will shine the most on creativity and some productivity or privacy focused areas. Yes, this is what I will bet on and personally want it to happen.

Deployment is part of developer experiences

Wed, 30 Mar 2022 00:00:00 +0000

Story 1

Back in 2005 when I was still in college. Friends found me a part time job developing a desktop application for a barber shop. .NET and C# pretty much on the headlines news everyday, it’s also a modern way of writing desktop application at that time. So I choose C# to development the application.

Microsoft makes the development process as easy as drag and drop controls and then hooking your database business code. I had so much fun of doing the development.

By the time of the deployment though, it gave me a hard time. Because every machine needs to have the correct version of .NET runtime installed, and a SQL server instance running at least somewhere in the network as well. It’s been a nightmare for me to have it working on the client’s machine.

Story 2

Fast forward to 2011. I was super excited about Ruby and Ruby on Rails, I got my first Mac. Another time, I got a contract work to develop an expense system for a big company in China.

Of course I am gonna use Ruby on Rails, cause there is nothing at that time beat the developer experience of that. While I am having all the fun and learnings having the web application built. I realized that the client can only allow windows operating system running at their company. Anyone who familiar with the early days of getting Ruby on Rails running on a windows machine should know what I am talking about. It took weeks to finally get it up running on that machine, not only that, every new deployment is like another big unknown for me. Because of the huge trouble of that, they finally canceled the entire project and looking for someone else who can build them a ASP.net application.

Deployment focused

Over the years, I’ve been very much appreciating the programming languages with the mind of easy of deployment. Languages like Rust or Go just makes the running your software on other people’s machine dead easy.

Customer mindset

As a user, I would be way much happier if it’s a single binary rather than 200 files scattered all over the place. That’s why I am huge fan of tools like syncthing - files synching tool and ran - local http server. I think that’s also a reason I favor Mac over Windows because of the single .app application file.

SPAs - self contained program

This might be sounds like a stretch argument, but I think Single-page application to some extend is an easier (might not be better, but that’s totally different discussion) way to distribute client side UI than traditional server side rendering apps.

Couple years ago, we want to migrate a huge production running web application from an old JSP (customized Java server) to a modern Spring based to mitigate some security risks. At that time the UI is written in a SPA framework, the Java server vended APIs and using JSP for the layout only. No matter how much I dislike client side rendering or the specific SPA framework they chosen at that time (spoiler: not React), I really appreciate the self-contained nature of SPAs. The migration of the backend is super smooth.

Long lasting

I’ve written quite some side projects over the years, interesting thing is the one wrote using Go or pure client side JavaScript are still running perfectly fine, but the ones written in Ruby or Node can’t even pass the bundle or npm install phase.

To end this already long article, I guess all I am trying to say is treating software deployment as part of the crafting. Cause real artists ship.

Don't just follow

Mon, 28 Mar 2022 00:00:00 +0000

Someone sent me a CR the other day. And the code went something like this:

function validateInputLanguage(input: InputOptions): void {
  // checking input is valid
  if (input.language) {
    // ....
    // we are good
    return;
  } else {
    input.language = defaultLang;
  }
}

The code here doesn’t matter that much though. I read the CR, put the following comments:

I would recommend to not mutate passed input, instead of returning the validation result plus reasons in the return value (instead of void).

Then the engineer relied to me this in the Slack:

Hi Mike, I saw your comments. I just follow what the original code patterns, the other places did the same kind of validations as well.

Every time people replied something like this, they basically shut down a door to ask why.

Yes, I understand there could be other places like this, but what if we start asking questions instead of just trying to get the job done?

Why did we make such pattern in the first place?
Does this pattern makes sense?
Is there any other better ways to achieve the same thing?
Is that risky to change the pattern now?
How we can reduce the risks to break things but still can make the code slightly better than before?
If now is not the right time, when do you think that time will be?

Come on, we all have things to do, why bother asking so many questions?

Every code change ultimately is an exercise for our brain, or a hidden conversation to the original author. Even though we could still ended up making the same code change, but with these questions in mind will help you making a bigger impact in the long term.

Pattern matching in TypeScript using ts-pattern

Tue, 22 Mar 2022 00:00:00 +0000

While I mentioned how great to do pattern match in Elixir. It’s surprisingly good to do this in TypeScript thanks to the ts-pattern.

Here is what I did to grab the content of different RSS feeds using Elixir:

def get_item_content(%{ "content" => %{ "value" => value } }) 
    when is_binary(value), do: value
def get_item_content(%{ "content" => content}) 
    when is_binary(content), do: content
def get_item_content(%{ "description" => %{ "value" => value } }) 
    when is_binary(value), do: value
def get_item_content(%{ "description" => description }) 
    when is_binary(description), do: description
def get_item_content(%{ "content_html" => content_html }), do: content_html
def get_item_content(_), do: "No Content"

And this is the ts-pattern version:

type RSSItemShape =
  | { content: string }
  | { content: { value: string } }
  | { description: { value: string } }
  | { description: string }
  | { content_html: string };
  
function extractContent(item: RSSItemShape): string | null {
  return match(item)
    .with({ content: __.string }, (r) => r.content)
    .with({ content: { value: __.string } }, (res) => res.content.value)
    .with({ description: { value: __.string } }, (res) => res.description.value)
    .with({ description: __.string }, (r) => r.description)
    .with({ content_html: __.string }, (r) => r.content_html)
    .exhaustive();
}

First of all I should say this might looks a little like comparing Apple to Orange. Because Elixir’s pattern matching is a language feature, while the pattern matching of TypeScript is to leveraging the ts-pattern library. The performance of Elixir’s version is most likely better than ts-pattern until someday this was built into V8 node engine, I don’t need to worry the Elixir’s pattern matching won’t work in the future unless syntax changes. But I would need to keep an eye on the version upgrades of ts-pattern which might introduce non-backward compatible changes.

However, today I am just gonna compare it from the developer’s experience perspective, how would it helps us to achieve our goals, express our intention more efficiently.

The first thing I noticed is that both the Elixir version and ts-pattern version are pretty concise. Yes Elixir has more flexibility that you can do the pattern matching on function level or block level. But I think ts-pattern did a great job in terms of readability. Unlike Elixir can absorb this into the language design, ts-pattern has to be creative within the JavaScript language syntax boundary. But ts-pattern already did a great job by boiling it down to the minimal key words: match, with, otherwise and exhaustive.

Composable. The one lovely thing about ts-pattern is the composable due to its method calls. Composing multiple pattern matching can be quite handy and strait-forward in ts-pattern.

Last thing of course is the type safety ts-pattern can give you. It won’t compile if you are missing cases when using the exhaustive keyword. This could prevent some errors creeping into the runtime. Be careful when passing any object into the match, as ts-pattern has no way to know if you covered all the cases or not. So it will only throw exception during the runtime.

// matching any object
function extractContent(result: any): string | null {
  return match(result)
    .with({ content: __.string }, (r) => r.content)
    .with({ content: { value: __.string } }, (res) => res.content.value)
    .with({ description: { value: __.string } }, (res) => res.description.value)
    .with({ description: __.string }, (r) => r.description)
    .with({ content_html: __.string }, (r) => r.content_html)
    .exhaustive(); // This will not prevent compile time check as we are matching any object
}

Overall I am super happy with ts-pattern, hopefully it can make into JavaScript feature someday, as TypeScript essentially constrained by the target language poses.

SPAs nudges us to caring more UX

Fri, 18 Mar 2022 00:00:00 +0000

I still vividly remember the days when BackboneJS came out, it might not the first Single Page Application framework, but probably the most notable one at that time. Before that, I was doing lots of Rails development for building web apps. BackboneJS for sure gave me a feel of fresh air when developing web UIs. Everything related to UI just became so lightweight to develop. Yes we would need to start thinking extra things which were covered by the browser in the past, things like manually handle form submission, URL redirects using JavaScript which is built-in into the browser already.

But there is the upside, like everything else. It’s the tendency to create more customer experience focused product, it nudges you to think more about the experience, instead of your beautiful data. I am not saying SPA will inherently have a better UX than MPA, that’s also what so many people are criticizing SPA for, saying they ruined the web, which I can’t disagree more. Just look at the advancement we made over HTML5, JavaScript and CSS, it’s hard to imagine if we are still rendering each web page statically on the server side using JSP, are we gonna get any of these? It’s exactly because we are keep pushing the boundaries using rich client side computing, creating madness things on the web pushes these web standards forward. Somewhat like the game industry did to the computer hardwares.

But maybe we are doing way too much on the client side using JavaScript? There is a trend in the recent years that to move most of the things if not everything back to server. At first I was one of them cheering for this as well, because of the insane complexity of the front end space. Until one day I was trying to do a tab switch using web socket to change the state on the server side. Wait a second, there is something wrong here. As a user, I am suddenly scared of knowing that everything I do on my screen were processed and scrutinized on the server.

Not because it’s posting the privacy issue by piggy back to servers every single click (it does so too), the problem I am concerned is we setup ourself again to be further away with the users. I’ve been doing a lots of front end only development and mixing both, at last I found that it’s extremely hard to build something really great by thinking everything in the backend context. Dealing with data vs dealing with human interaction is fundamentally different. I admire people can do both well at the same time.

SPAs aren’t a mistake. It’s something in the history for the first time reminds us: your code doesn’t matter if it can’t express nicely in a human friendly form.

We as human like to go to extremes, because it’s so much easier to do so. It’s hard to calibrate, to pick, to combine, to make the best of both worlds. Though I think we are gradually reaching there. Oh, also don’t forget, doing everything on the browser sometimes can be quite beautiful.

RSS feeds parser and core models

Sat, 12 Mar 2022 00:00:00 +0000

Parser

There are so many RSS parser out there. I’ve tried lots of them, including JavaScript, Elixir and Rust versions. So far the most reliable one I’ve tested is the Rust version from rust-syndication. There is even a Elixir fast_rss wrapped on top of the rss.

The wrapper is pretty lightweight rustler makes the whole interop with Rust with Elixir so much easier and straightforward.

I could’ve just use fast_rss, but I also want to parse Atom feeds as well. So I ended up took the approach of fast_rss and build another wrapper around atom feed parser on top of Rust’s atom.

This is what the Rust code wrapper looks like.

Core models

Once we have the RSS and Atom feed parser, along with JSON feed. We would ended up with a map data structure of the feeds. Since there are so many stuff in the RSS feeds, all I want for now is just part (title, date, content, author) of that. So I would like to be working from the backwards. Start from what the UI might looks like, then define the core models to support this UI.

I took a screenshot of an awesome app called NetNewsWire (You should try it if you haven’t).

By looking at the UI, here you can find the minimal information we would need to extract from the RSS feed.

defmodule Item do
  defstruct id: nil, title: nil, date: nil, content: nil, link: nil, authors: []
end

defmodule Feed do
  defstruct link: nil, title: nil, favicon: nil, items: []
end

These two models by no means are the final version, but it’s a good start for us to start to create our minimal RSS reader UI. Once we have the model in mind, the fun part of leveraging the power of Elixir begins.

Pattern matching to extract

Okay, so now we have both the core models and the parsed RSS/Atom feeds in Elixir’s map format. Time to normalize/extract the key contents into our defined core models above. I think that’s where the Elixir’s pattern matching really shines.

For example, we would like to extract the content of the feed item. For different rss versions, the place they put the content will be different. Here are a couple of examples and its Elixir extraction function:

The entire content field extracting including error case handling (nothing can be found) can boils down to this:

def get_item_content(%{
      "content" => %{
        "value" => value
      }
    })
    when is_binary(value),
    do: value

def get_item_content(%{"content" => content}) when is_binary(content), 
    do: content

def get_item_content(%{
      "description" => %{
        "value" => value
      }
    })
    when is_binary(value),
    do: value

def get_item_content(%{"description" => description}) when is_binary(description),
  do: description

def get_item_content(%{"content_html" => content_html}), do: content_html

def get_item_content(_), do: "No Content"

Note: is_binary here is basically trying to check if the given value is a string. You can read more here at Elixir’s doc: https://elixir-lang.org/getting-started/binaries-strings-and-char-lists.html

I don’t know about you. But I found extremely satisfying writing pattern matching comparing to nested if/else or guard statement. I think mostly because how human’s brain works, we are exceptionally good at pattern matching. Instead of one step at a time checking x then y, we describe what the stuff we want out of the given shape of data.

Here you can find the all the data extraction related code. Please notice this project is still being actively developed.

Now we actually have a minimum working version of given an feed url, parsing and extracting them into a the desired data model we want. Next, we will see how can leverage the part we already have so far to make something useful.

Create a RSS reader and beyond in 2022

Fri, 11 Mar 2022 00:00:00 +0000

I’ve been using RSS reader for almost more than 15 years. Right start from graduated from college. It’s amazed to see all kinds of different forms product on top of this open (while also chaotic) format.

There are native apps, AI based, open source web apps. We’ve never been at a time lacking of RSS readers. While bother creating another one?

First of all, I love working on open data. Same reason why I love the open web so much. You don’t need to get permission in order to scrape a webpage (unless you commercialize the product which is a different topic), you don’t need permission in order to write a MonkeyScript to make the web page looks like what you wanted. It’s the librating and freedom this platform enabled attracts me so much.

Second, I always want a little bit different here and there for the RSS reader I wanted. For example, I want a different theme or font in order to focus reading a long article, I want to highlight some parts or write something about it, sometimes I also want the reader just generate a PDF offline for me without open the webpage, I also want to read the full article of some feeds while they somehow configured it just to read the first paragraph.

Last but not least, I am hoping to create something beyond just a RSS reader, something a little bit bigger scope for my online readings, like read it later.

Ultimately, it’s the experience of building such thing matters the most.

Use Svelte in Phoenix the simple way

Tue, 08 Mar 2022 00:00:00 +0000

TLDR

Gist link for the changes to make this happen

Why

While there were few options to integrate Svelte into existing Phoenix app. But I still don’t find any of them to be particular simple. What I really want is something that I can understand, as minimal as possible, no heavy Node tooling involved. After play with different options, I finally found a sweet spot for me to use Svelte while not overwhelmed by the additional toolings. In fact, it’s just couple lines of configuration with two files both less than 20 lines of code.

Before diving into my solution to this. Let’s first understand how Phoenix integrate with JavaScript/CSS assets by default.

Turn Svelte components into JS

Since 1.6, Phoenix uses esbuild to build your JavaScripts. The esbuild is a standalone Go program thus no Node, NPM invovled. That’s why in the assets folder, there is neither package.json nor node_modules . The esbuild will slurp the entry app.js then output the results into priv/static/assets.

So the first thought I have is how can I not step away from the default path, still using esbuild for Svelte. The good news is that there is a esbuild-svelte plugin, the bad news is that I have to invoke the plugin asynchronously, in other words, by writing a JS build file.

Install packages:

cd assets && npm install esbuild esbuild-svelte svelte svelte-preprocess --save-dev

Luckly, esbuild-svelte already handled the heavy lifting part. So the build.js is actually quite simple:

// build.mjs
import esbuild from "esbuild";
import esbuildSvelte from "esbuild-svelte";
import sveltePreprocess from "svelte-preprocess";
import { readdir } from "fs/promises";

async function allSvelteComponents() {
  const baseDir = "./js/app/";
  const all = await readdir(baseDir);
  return all.filter((f) => f.endsWith(".svelte")).map((f) => `${baseDir}${f}`);
}

(async function () {
  const components = await allSvelteComponents();
  const isProd = process.env.NODE_ENV == "production";
  esbuild
    .build({
      entryPoints: components,
      format: "esm",
      bundle: true,
      minify: isProd,
      watch: !isProd,
      outdir: "../priv/static/assets/svelte",
      plugins: [
        esbuildSvelte({
          preprocess: sveltePreprocess(),
        }),
      ],
    })
    .catch(() => process.exit(1));
})();

Using .mjs as extension here to indicate this is a ES module. So that I don’t need to change the package.json’s type globally.

Now if you manually do node ./assets/build.mjs you should be able to see the built components JS in the priv/static/assets/svelte folder.

Let’s add this build step into both dev and deployment time:

# mix.exs
 defp aliases do
    [
      "assets.deploy": [
        ...
        # build svelte components
        "cmd --cd assets NODE_ENV=production node ./build.mjs"
      ]
    ]
  end

# dev.exs
watchers: [
    ...
    # build and watch svelte components
    node: ["build.mjs", "--watch", cd: Path.expand("../assets", __DIR__)]
  ]

This is it.

Consuming it through `<script type="module">`

<script type="module"> is such a wonderful thing. It finally makes consuming JavaScript so much easier and nicer.

Now all you need to do to render a Svelte component is put this in your .heex template:

<div id="hello"></div>
<script type="module">
    import Hello from "<%= Routes.static_path(@conn, "/assets/svelte/Hello.js") %>"
    new Hello({
        target: document.getElementById("#hello"),
        props: { name: "Mike" }
    })

This is it.

Render it the Elixir way

To push it a bit further. You can write such a helper to make the above rendering even simpler.

defmodule EasySvelteExampleWeb.SvelteView do
  use EasySvelteExample, :view

  # just for default values
  def render_component(conn, name, props \\ [])

  def render_component(conn, name, props) when is_list(props) do
    render_component(conn, name, Enum.into(props, %{}))
  end

  def render_component(conn, name, props) when is_map(props) do
    component_id = "svelte-ele-#{name}-#{random_id()}"
    component_import_path = Routes.static_path(conn, "/assets/svelte/#{name}.js")
    props_json = props |> Jason.encode!()
    component_class_name = name |> String.split("-") |> Enum.map_join(&String.capitalize(&1))

    [
      content_tag(:div, "", id: component_id),
      content_tag :script, type: "module" do
        """
        import #{component_class_name} from "#{component_import_path}"
        new #{component_class_name}({
            target: document.getElementById("#{component_id}"),
            props: #{props_json}
        })
        """
        |> raw
      end
    ]
  end

  defp random_id do
    for _ <- 1..10, into: "", do: <<Enum.random('0123456789abcdef')>>
  end
end

Now to render a svelte component in .heex:

<%= SvelteView.render_component("Hello", name: "Mike") %>

Here you can find the gist link of all the changes needed for this.

Re-watch Rethinking reactivity

Fri, 11 Feb 2022 00:00:00 +0000

https://www.youtube.com/watch?v=AdNJ3fydeao

Wether or not you are a front end developer or even a software engineer. This talk worth your time either from a inspirational perspective on how can we create more accessible things, or from a marketing perspective, I’ve never saw a better “marketing talk” for a front end framework, watching this talk constantly reminds me of Apple or Volvo’s advertisement.

Reactivity, the easy way

Svelte is more like a language other than a framework. It’s like a modern dialect of the web, combined all three of HTML, CSS and JavaScript in one single file. The most interesting part is the compiler behind it, which tracks the “changes” of your variables. The magics lies into the assignment tracking, for example when you do foo = 42, anywhere uses foo will be evaluated thus you get the “automatic” reactivity of the interface without needing to do anything manually.

Reactivity in JavaScript land is not new, RxJS has been there for a long time. But thinking in terms of streams is quiet some learning curve or adaption for programers, while assignment is almost the first day we learned from programming 101.

Compiler

The other key part of the Svelte is the compiler. There is a trend that we are doing more and more in the back scene instead of shipping large frameworks. The compiler enables us to still write in the way that we used to, but free from the limitation of given languages, cause the compiler will help “translate” them into “low level” bits.

It enables us to think more in the human friendly way, instead of forcing us to adapting our thinking approach in the machine way.

Ultimately, it’s a decoupling and abstraction. So the compiler could improve our web app performance without us changing our existing code.

Another advantage or the compiler is to produce more optimized bundle depending on the environment. Whether it would be JavaScript script on the client side, or plain string on the server side.

All of these are the great parts, but what touched me the most is the non-technical parts.

Accessible tool

I am always fascinated about the non-technical parts for certain tools or technology, in other words, the beliefs of the creator behind the scene. One reason might be the “fear” that I have over the loved technology might change, because even though the technology or the implementation changes, the thinking or main idea behind it rarely do.

Another reason is I think ideas simply lasts longer than specific technology. Just like Dynabook did to iPads, Smalltalk did to the OOP (though Alan Kay means quite differently things) and a lot other ideas influenced and shaped Notion.

If I have to summarize what’s fundamentally different Svelte is different than other front end frameworks. It’s accessibility. Here in the context I didn’t mean the Web Accessibility though it’s super well considered and covered in Svelte already.

Here I am more talking about Svelte is more accessible to use in order to create web interactive UI. HTML is no doubt is the most accessible pillar of three web elements(HTML, CSS, JavaScript). By bringing back the focus to the HTML markup while adding interactivity is a tremendously mental reliving comparing to popular front end frameworks like React which is almost exclusively using JavaScript to create everything.

Rich mentioned his inspiration came from his wife using spreadsheets to calculate, analysis complicated business problems.

I believe tools are fundamentally intend to be used by more and more people, making it less intimidating and more accessible is the key to empower regular people. Apple succeeded not because they looks shinny and beautiful, but because they demystified “professional” tools and enables you and me to create as well.

Which reaches Rich’s last words of the talk, which I think it’s the essence of Svelte: In a strange way, it brought us all a little closer together.

What I've been missing by using VS Code

Tue, 08 Feb 2022 00:00:00 +0000

VS code is awesome, sometimes too good. I used to writing pretty much everything from work to side projects, from Ruby to JavaScript exclusively using Vim. But I noticed that I’ve slowly transited using VS Code 90% of the time for the recent years.

It’s been great, it helped me caught lots of bugs because of the smart auto completion, it prevent me from writing so many typos because of the built in spell checking, plus TypeScript almost take over the entire front end development, you never need to worry about code formats, because every save will automatically convert your code into uniformed style through “prettier”. The code we wrote feels much more “safe” and “beautiful”. In other words, the code we wrote at least from visual perspective looks more like machine produced code.

The other day, because of VS Code crashes on my machine and refuse to re-opening. I opened the Elixir project using my old friend Vim. Prior to using Vim, I thought I would feel uneasy, cause all the assistant now is gone. But I immediately feels something so familiar before, I feels like I am start “thinking” again. I feels like I am in control.

The slogan of Copilot being just a “copilot” or assistant is an illusion in practice. Once you’ve been used to the assistant, you will never be able to truly in control. Every single step of design, every details, variables will be influenced by your assistant. On the positive side, you probably will never be able to coding “by yourself”, on the negative side, you would never truly feels the flow of coding. It’s like riding using with the learner’s wheels on, yes it’s riding, but no, you didn’t know what’s riding really feels.

Programming guru Charles Petzold once wrote about this topic in his essay Does Visual Studio Rot the Mind? back in 2005. One thing resinated me lot is the “IntelliSense” changes how we think:

But the implication here is staggering. To get IntelliSense to work right, not only must you code in a bottom-up structure, but within each method or property, you must also write you code linearly from beginning to end — just as if you were using that old DOS line editor, EDLIN. You must define all variables before you use them. No more skipping around in your code.

These intelligent tools are not neutral, they will find every opportunity to change the way how we code, or even worse, how we think.

We as programmers spent lots of time typing code facing the screen. Because we need to deliver that feature, because we need to cut the latency to half, because we need to increase the conversion to 40%. But there is another side of coding as well.

Creator of Redis once said:

Code is like a poem; it’s not just something we write to reach some practical result.

If you can, open Vim or other plain text editor someday, write some code. Hopefully you will find what you’ve been missing all this long.

How can we use computers for the "good"?

Mon, 07 Feb 2022 00:00:00 +0000

When I say “good” here, I am more talking about the non-utilitarian part. I am not talking about using computer to pay your bills, making an important phone call, or using maps for directions. I am more talking about “the others”.

I was touring a school with my son the other day. While we were in the computer class room. I was attracted by all the printed art made by computer programs hanging on the wall.

Some of them are fine drawing pixel art, some of them are simple shape combinations to form cute animal, some of them are abstractions. This reminds me the very original vision of the computer when I was a kid. You can use computer to make all sorts of things, to express yourself.

Fast forward to today, when people think about computer, it’s social medias, it’s Youtube, it’s browsing exotic pictures, it’s an entertainment device. Part of the reason is because of the internet, part of the reason is the mobile devices. These two things together created multiple portals stay with you all the time. You can escape anytime to a different world, different place all the time. But we almost forget how to create, how to make something simple or beautiful, something fulfilling our creation mind. We were more attached to the outer world, but less with our inner world.

You probably would argue that computers are just tools, it depends on how you use them. But the problem is today’s computers / operating systems are not neutral devices. They almost always emphasize one thing over another, they almost always trying to change how you think about it. If you pay attention to the default applications installed on the Mac OS, you would easily spot the trends. There are now pushing you to sign in iClouds, News, TV apps installed by default, and you can’t remove it wether you use it or not, it sits there the first time you start your computer setup. You can hide it, but please never underestimate the power of defaults. You can also learn more from the awesome book Nudge, it talks extensively on how defaults can make a huge difference to your behavior and life.

Computers are great on many things, but I still very much love the part that it enables us to create something we can’t create before.

Follow people vs follow technology

Wed, 26 Jan 2022 00:00:00 +0000

I’ve been a software developer for about 15 years. Along the way, I’ve been writing software using different tools and technologies (mixing languages and frameworks in my head): .NET, C#, Ruby (On Rails), JavaScript, Scala, Clojure, Go, Swift, TypeScript, Elixir.

It’s hard to describe in one or two sentences what I learned along the way from all these languages and frameworks. I sometimes even felt offended by asking listing the languages, frameworks you’ve been using in the past. Because I think that missed the point.

One day I suddenly realized that maybe, asking who had the most impacted to your engineering career is more meaningful than asking what tools or technologies you’ve been used. Maybe even more important than what you’ve created along the way. Because by creating what, most of the time, is not under our control.

Things that resinated with me, or what actually changed my mind or brain, are always people, or now people always uses another fancy word – community.

I truly understand what design pattern means by learning from Terry Lee back in 2007, I was ignited by DHH’s demo of RubyOnRails in 2008; I was then in the rabbit hole of Ruby programming and Vim by watched all of Gary Bernhardt Destroy All Software. Then all of the sudden, Rich Hickey and David Nolen came from no where and washed my brain about programming completely. While doing Clojure and ClojureScript programming feels quite satisfying and I created some useless tools along the way. I was also greatly influenced by Mike Bostock and started thinking the computation medium through building data visualizations at the company.

At the same time, I somehow feels like my brain is still stuck with Ruby, the simplicity (weirdness) of the syntax just dominant my mind for constructing any programs. So then I re-discovered Elixir. Of course, I’ve watched lots of talks by the creator José Valim behind it. Whenever I am coding with Elixir, I feels like I am playing an art piece José made. The words (highlights are in the comments) of him also got me deeply.

That said, I don’t know from where you got the impression Elixir is mutable. Or got the impression we are trying to make “a better functional language”. Yes, it is a functional language, but we never claim it to be a better one (as it is a shallow goal). – josevalim original

Most recently I started following Rich Harries closely. Especially his talk about Rethinking Reactivity. (I will write more about my takeaways on this talk later on). But in nutshell, he encourage us to see the web development in a much simpler and accessible way.

Ultimately, the tools or languages we are using is made by humans. There are world views implanted into the tools or frameworks, therefore it will eventually “get” us as well. By adopting a tool is more like adopting a new perspective of seeing things.

Svelte vs LiveView to my brain

Tue, 25 Jan 2022 00:00:00 +0000

I have been building a web app product using Phoenix LiveView recently, it’s been very pleasant experiences for me. Since I was a long time Ruby (On Rails) developer since 2011 and then got my brain washed on functional programming (Clojure, ClojureScript) in 2014, so Elixir turns out a perfect fit for me (especially my brain). It has the dynamic language feeling but also comes with compiler helps to avoid silly bugs; it has powerful pattern matching; the syntax of the language is simple and concise. I often looking at the code I wrote and felt extremely satisfying, almost to the extent of seeing the code I wrote like an “art” piece (thanks to Jose on the language design). There are lots of things Elixir changed my brain in the good way that I will write about in a later post. In this one, I would just focus on the LiveView part.

If you haven’t heard LiveView yet, you can read more on here. Essentially, I think LiveView is trying to leverage three things: WebSocket, OTP and JS diffing/patching. Of course this is a dramatically simplified pillars of LiveView, but I think these are the keys if you want to understand what’s going behind the scene.

The first part and arguably the most important part is the web socket, it enables the long live communication to the server, so that any client side events can be “think” as they are happened directly in your Elixir code as well. Let this idea sink a bit for you before moving on. Because I think this is the fundamental difference not only technically but also can change the way you think about the programs.

For example, we want to switch a toggle button to change a user profile configuration.

Here is the traditional flow of simple client/server updates:

user click a button
send a fetch request to the server
server set up an endpoint taking the request from client
server react to that event by doing some database query/updates
server send the serialized response to client
client received the response and do things

LiveView’s flow would be like:

user click button
server side code got the “callback” event, doing database query/updates
describe the new shape of the UI
[user saw the new UI]

Maybe the number of steps in the list can not justify the mental burden it applies to our brain. But here are couple things I would have to think of in the traditional client/server flow which I don’t need to in LiveView:

How should I let the client side send the request?
What endpoint I should use, shall I design a common endpoint so that it can take other updates later on on other places as well or just specific to this one?
If I designed it to be more general, how could I prevent it from updating other things I don’t want it to?
Authentication / CORs …

Please notice that there are lots of other nuances in between that I omitted, I also deliberately uses “callback” event here because that’s a more familiar term for client side engineers. But my point is by looking at this new flow, you will realize there is a huge mind shifting difference: LiveView enables you to think closer to the data source.

With LiveView, sometimes you almost feels like you are coding everything in one machine without the internet. Because it eliminated the events listening, the network call, the endpoint handles it, the data contracts design between client and server, the serialization of the data passing back and forth. By saving all these mental bandwidth, it enables you to think more closely to the data source.

It’s such a blessing to be not thinking all the data flows, endpoints in order to flip a switch. But there is a catch, not technically (though there indeed is, but not the major thing I want to focus on this article), but mentally about how you think about the your UI or UX.

I found out comparing to traditional client side UI frameworks like React or Svelte, the main impact it has over my mind is how I approach the UI/UX. With the power of seamlessly communication between client and server side, a lot of times I would easily falls into eagerly manipulating the data bits, being more obsessed with the data, sometimes by sacrificing the user experiences.

I think this is not an issue just applying to myself, people doing both design and coding were often impacted by this as well.

I found myself ended up asking different kind of questions using a client side framework vs LiveView. Because client side JavaScript, obviously, enables you to think more closely the the human being, cause it’s naturally, physically closer to the user. While your elixir function is ultimately lived inside a different process in a distant warm data-center.

When I tried to use Svelte (or any other client side framework, I will have another post on why I love Svelte so much) to replace some of the UI, I found out that it enables me to “sit” more closely with the user, thus it focus me to think more from the actual user perspective on how they use the product, how we should present the data using what transition or animation so that it’s more “accessible” to them.

Productivity is a double edge sword. Being able to do things extremely fast can enables us delivering features we can’t do yesterday, but sometimes it comes with an invisible cost or benefits to our brain, to the ways we are thinking about a problem.

I would admit that I’ve both overused client side frameworks and LiveView in the past. By overusing them, I would quickly found out the cons of the tools in a more tangible way.

I sill love LiveView (and Elixir) and Svelte. Using these tools just makes me happy.

Orchard Lab

Short Blog

Ship It

Multiple environments using Fly.io

Bonus (Auto deployment using GHA)

🎨 Extract dominant colors in Elixir

Create Elixir library

Create Rust crate inside

Integrate kmeans-colors

Publish to hex

Rustler Pre-compilation

Create Github action workflow

Consume pre-compiled binaries

Publish to hex

Using it in Livebook

Summary

Credits

What’s next

📨 Void Inbox

Why am I building this

What’s next

Fall in love with the questions

Bun + Prisma

Spin up postgres

Bun + Prisma Fastlane

Summary

References

Playground

Idea

Playground

Assumption

Experiment

Modularize

Learnings

How do you feel

Tools

Alfred

iTerm2

Tmux

iA Writer

DayOne

Syncthing

Logseq

Minimal setup time for your new project

The Future Computers

April 15 2023

April 10 2023

April 08 2023

Griffith Observatory, LA

Weir Canyon Trail

First article using Bamboo 🎋

Next Journey

Story 1

Story 2

Story 3

Connecting the dots

Journey at Amazon

Championships, coaching and zen

Beginners Mind

The Missing Helper for Your Static Blogs - Part 1

Shape of the tool

Choosing the tools

High level design

What’s next

Software Sometimes Is Annoying

Thick-client software is coming back ?

Deployment is part of developer experiences

Story 1

Story 2

Deployment focused

Customer mindset

SPAs - self contained program

Long lasting

Don't just follow

Pattern matching in TypeScript using ts-pattern

SPAs nudges us to caring more UX

RSS feeds parser and core models

Parser

Core models

Pattern matching to extract

Integrate `kmeans-colors`

Consuming it through `<script type="module">`