a formula a day

jaxvis animates a JAX function by walking its jaxpr, so every frame is a real intermediate value and not a drawing of one. Annotate the function, run it, get the clip.

One formula a day, published when it turns out well.

If you want to see more randomly generated things — shapes, flows and clouds made for no reason at all — they are all on this page.

one dimension · softmax · came from


2026-10-17 · permalink
One dimension

Every clip starts flat at zero, rises into $y = x$, and from there each bend is one more primitive. The caption names it.

The window is fixed, so a curve that leaves it is pinned dimly to the edge instead of disappearing.

import jax.numpy as jnp
import jaxvis
import fad

fad.day(__file__)

x = jaxvis.rail(-6.0, 6.0, 500)

The squash

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

Negate, exponentiate, add one, divide. The straight sloping line is $-x$. The exponential blows off the top; the division folds it back into a window one unit tall.

@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-0.35, 1.35), palette="vapor"))
def sigmoid(p):
    return 1.0 / (1.0 + jnp.exp(-p))

The symmetric squash

$$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}$$

The same squash, centred on zero: one exponential running each way, and their difference over their sum.

@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-1.35, 1.35), palette="bloom"))
def tanh(p):
    up, down = jnp.exp(p), jnp.exp(-p)
    return (up - down) / (up + down)

The soft hinge

$$\operatorname{softplus}(x) = \log(1 + e^{x})$$

The same exponential. The log turns a wall into a hinge.

@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-1.0, 6.5), palette="ultra",
                               fill=0.08))
def softplus(p):
    return jnp.log1p(jnp.exp(p))

GELU

$$\operatorname{gelu}(x) \approx \tfrac{1}{2}x\left(1 + \tanh!\left[\sqrt{\tfrac{2}{\pi}}\left(x + 0.044715x^{3}\right)\right]\right)$$

A sigmoid-shaped gate multiplied back onto $x$. The dip left of zero, the part ReLU does not have, arrives in the last multiply.

@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-1.5, 6.5), palette="bloom"))
def gelu(p):
    inner = 0.7978845608 * (p + 0.044715 * p ** 3)
    return 0.5 * p * (1.0 + jnp.tanh(inner))

A square out of sines

$$f(t) = \sin t + \tfrac{1}{3}\sin 3t + \tfrac{1}{5}\sin 5t + \tfrac{1}{7}\sin 7t$$

Each harmonic arrives at full height, gets divided down, settles into the corners. A self-rescaling window would hide every division.

s = jaxvis.rail(-jnp.pi, jnp.pi, 500)


@jaxvis.draw(s, **jaxvis.curve(s, ylim=(-1.45, 1.45), palette="vapor"))
def square_wave(p):
    y = jnp.sin(p)
    y = y + jnp.sin(3.0 * p) / 3.0
    y = y + jnp.sin(5.0 * p) / 5.0
    return y + jnp.sin(7.0 * p) / 7.0

The bell

$$y = e^{-x^{2}/2}$$

Square, negate, halve, exponentiate. The parabola falls off the bottom and comes back bounded.

@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-0.3, 1.25), palette="ember",
                               fill=0.10))
def bell(p):
    return jnp.exp(-(p ** 2) / 2.0)

Ringing down

$$y = e^{-t/3}\sin 4t$$

The envelope is never drawn. It is one multiply, and you see it as the curve being pulled to the axis from the right.

r = jaxvis.rail(0.0, 12.0, 500)


@jaxvis.draw(r, **jaxvis.curve(r, ylim=(-1.15, 1.15), palette="ultra"))
def ring_down(p):
    return jnp.exp(-p / 3.0) * jnp.sin(4.0 * p)

2026-10-20 · permalink
Softmax, and the subtraction nobody mentions

Twenty-four scores go in, twenty-four probabilities come out. The textbook writes two operations. The jaxpr shows four.

import jax.numpy as jnp
import jaxvis
import fad

fad.day(__file__)

x = jaxvis.rail(-6.0, 6.0, 24)

The formula as written

$$\operatorname{softmax}(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}}$$

Exponentiate, then divide by the total. Watch the sum arrive as a flat line: one number, broadcast back across all twenty-four.

@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-0.15, 1.15), palette="vapor"))
def softmax(p):
    e = jnp.exp(p)
    return e / jnp.sum(e)

The formula as implemented

$$\operatorname{softmax}(x)_i = \frac{e^{x_i - \max x}}{\sum_j e^{x_j - \max x}}$$

Subtracting the largest score changes nothing — it cancels, top and bottom. What it buys is that the biggest exponent is now $e^0 = 1$, so nothing can overflow. Two extra steps in the jaxpr, and the curve that comes out is identical.

@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-0.15, 1.15), palette="bloom"))
def stable_softmax(p):
    e = jnp.exp(p - jnp.max(p))
    return e / jnp.sum(e)

2026-10-18 · permalink
Where it came from

Colour means which input, not value. Each input is a $9\times9$ block of grains, split among the outputs in proportion to $|M_{ij}x_j|$ — the share of $j$ that output $i$ used.

What piles up at the top is the provenance. One source, one colour. Everything averaged, grey.

import jax.numpy as jnp
import numpy as np
import jaxvis
import fad

fad.day(__file__)

n = 8
i = jnp.arange(n)
x = jnp.asarray([1.0, 2.0, 1.5, 3.0, 0.8, 2.4, 1.2, 2.8])

A permutation

$$M_{ij} = [\,i + j = n - 1\,]$$

Every input goes to exactly one output. Eight pure hues, reversed.

jaxvis.grains((i[:, None] + i[None, :] == n - 1) * 1.0, x,
              name="permutation")

A running total

$$M_{ij} = [\,i \ge j\,]$$

Pure on the left, every colour on the right. That gradient is the prefix sum.

jaxvis.grains((i[:, None] >= i[None, :]) * 1.0, x, name="running_total")

Local averaging

$$M_{ij} = \tfrac{1}{3}\,[\,|i - j| \le 1\,]$$

Three colours per pile, always its own and its neighbours'. That is what local means.

jaxvis.grains((jnp.abs(i[:, None] - i[None, :]) <= 1) / 3.0, x,
              name="local_average")

A learned weight matrix

$$M_{ij} \sim \mathcal{N}(0, 1)$$

Grey everywhere. "Where did this activation come from" has no short answer in a dense layer, and this is what no short answer looks like.

W = jnp.asarray(np.random.default_rng(7).normal(size=(n, n)) / 2.4)

jaxvis.grains(W, x, name="dense", spread=0.7)

Try it

jaxvis.grains(M, x, name="mine")

github.com/aminehd/jaxvis