jaxvis animates a JAX function by walking its jaxpr, so every frame is a real intermediate value and not a drawing of one. Annotate the function, run it, get the clip.
One formula a day, published when it turns out well.
If you want to see more randomly generated things — shapes, flows and clouds made for no reason at all — they are all on this page.
one dimension · softmax · came from
Every clip starts flat at zero, rises into $y = x$, and from there each bend is one more primitive. The caption names it.
The window is fixed, so a curve that leaves it is pinned dimly to the edge instead of disappearing.
import jax.numpy as jnp
import jaxvis
import fad
fad.day(__file__)
x = jaxvis.rail(-6.0, 6.0, 500)
$$\sigma(x) = \frac{1}{1 + e^{-x}}$$
Negate, exponentiate, add one, divide. The straight sloping line is $-x$. The exponential blows off the top; the division folds it back into a window one unit tall.
@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-0.35, 1.35), palette="vapor"))
def sigmoid(p):
return 1.0 / (1.0 + jnp.exp(-p))
$$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}$$
The same squash, centred on zero: one exponential running each way, and their difference over their sum.
@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-1.35, 1.35), palette="bloom"))
def tanh(p):
up, down = jnp.exp(p), jnp.exp(-p)
return (up - down) / (up + down)
$$\operatorname{softplus}(x) = \log(1 + e^{x})$$
The same exponential. The log turns a wall into a hinge.
@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-1.0, 6.5), palette="ultra",
fill=0.08))
def softplus(p):
return jnp.log1p(jnp.exp(p))
$$\operatorname{gelu}(x) \approx \tfrac{1}{2}x\left(1 + \tanh!\left[\sqrt{\tfrac{2}{\pi}}\left(x + 0.044715x^{3}\right)\right]\right)$$
A sigmoid-shaped gate multiplied back onto $x$. The dip left of zero, the part ReLU does not have, arrives in the last multiply.
@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-1.5, 6.5), palette="bloom"))
def gelu(p):
inner = 0.7978845608 * (p + 0.044715 * p ** 3)
return 0.5 * p * (1.0 + jnp.tanh(inner))
$$f(t) = \sin t + \tfrac{1}{3}\sin 3t + \tfrac{1}{5}\sin 5t + \tfrac{1}{7}\sin 7t$$
Each harmonic arrives at full height, gets divided down, settles into the corners. A self-rescaling window would hide every division.
s = jaxvis.rail(-jnp.pi, jnp.pi, 500)
@jaxvis.draw(s, **jaxvis.curve(s, ylim=(-1.45, 1.45), palette="vapor"))
def square_wave(p):
y = jnp.sin(p)
y = y + jnp.sin(3.0 * p) / 3.0
y = y + jnp.sin(5.0 * p) / 5.0
return y + jnp.sin(7.0 * p) / 7.0
$$y = e^{-x^{2}/2}$$
Square, negate, halve, exponentiate. The parabola falls off the bottom and comes back bounded.
@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-0.3, 1.25), palette="ember",
fill=0.10))
def bell(p):
return jnp.exp(-(p ** 2) / 2.0)
$$y = e^{-t/3}\sin 4t$$
The envelope is never drawn. It is one multiply, and you see it as the curve being pulled to the axis from the right.
r = jaxvis.rail(0.0, 12.0, 500)
@jaxvis.draw(r, **jaxvis.curve(r, ylim=(-1.15, 1.15), palette="ultra"))
def ring_down(p):
return jnp.exp(-p / 3.0) * jnp.sin(4.0 * p)
Twenty-four scores go in, twenty-four probabilities come out. The textbook writes two operations. The jaxpr shows four.
import jax.numpy as jnp
import jaxvis
import fad
fad.day(__file__)
x = jaxvis.rail(-6.0, 6.0, 24)
$$\operatorname{softmax}(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}}$$
Exponentiate, then divide by the total. Watch the sum arrive as a flat line: one number, broadcast back across all twenty-four.
@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-0.15, 1.15), palette="vapor"))
def softmax(p):
e = jnp.exp(p)
return e / jnp.sum(e)
$$\operatorname{softmax}(x)_i = \frac{e^{x_i - \max x}}{\sum_j e^{x_j - \max x}}$$
Subtracting the largest score changes nothing — it cancels, top and bottom. What it buys is that the biggest exponent is now $e^0 = 1$, so nothing can overflow. Two extra steps in the jaxpr, and the curve that comes out is identical.
@jaxvis.draw(x, **jaxvis.curve(x, ylim=(-0.15, 1.15), palette="bloom"))
def stable_softmax(p):
e = jnp.exp(p - jnp.max(p))
return e / jnp.sum(e)
Colour means which input, not value. Each input is a $9\times9$ block of grains, split among the outputs in proportion to $|M_{ij}x_j|$ — the share of $j$ that output $i$ used.
What piles up at the top is the provenance. One source, one colour. Everything averaged, grey.
import jax.numpy as jnp
import numpy as np
import jaxvis
import fad
fad.day(__file__)
n = 8
i = jnp.arange(n)
x = jnp.asarray([1.0, 2.0, 1.5, 3.0, 0.8, 2.4, 1.2, 2.8])
$$M_{ij} = [\,i + j = n - 1\,]$$
Every input goes to exactly one output. Eight pure hues, reversed.
jaxvis.grains((i[:, None] + i[None, :] == n - 1) * 1.0, x,
name="permutation")
$$M_{ij} = [\,i \ge j\,]$$
Pure on the left, every colour on the right. That gradient is the prefix sum.
jaxvis.grains((i[:, None] >= i[None, :]) * 1.0, x, name="running_total")
$$M_{ij} = \tfrac{1}{3}\,[\,|i - j| \le 1\,]$$
Three colours per pile, always its own and its neighbours'. That is what local means.
jaxvis.grains((jnp.abs(i[:, None] - i[None, :]) <= 1) / 3.0, x,
name="local_average")
$$M_{ij} \sim \mathcal{N}(0, 1)$$
Grey everywhere. "Where did this activation come from" has no short answer in a dense layer, and this is what no short answer looks like.
W = jnp.asarray(np.random.default_rng(7).normal(size=(n, n)) / 2.4)
jaxvis.grains(W, x, name="dense", spread=0.7)
jaxvis.grains(M, x, name="mine")