The Cooley-Tukey algorithm works very well when the input sequence’s length is a product of (mostly or entirely) small prime factors, which it can use to “break down” the problem. However, it provides little speedup once those small prime factors (if any exist) are exhausted. In particular, for prime input lengths, Cooley-Tukey provides no speedup at all, degrading to \(O(\lvert x \rvert^2)\) time complexity. In order to guarantee \(O(\lvert x \rvert \log \lvert x \rvert )\) time complexity regardless of what prime factors the length has, we will need another algorithm, such as Bluestein’s algorithm.

Bluestein’s Algorithm

Recall the definition of the discrete Fourier transform:

Substitute the identity \(jk = \frac{j^2 + k^2 - (k-j)^2}{2}\):

At this point, the right-hand side should look very similar to a cyclic convolution, however it still needs a few more manipulations:

Let there be sequences \(y\) and \(z\), both of some (yet-undecided) length \(N\) defined as follows:

Substituting these sequences to our previous equation:

For the discrete Fourier transform \(\mathcal{F}\{ x \}\), the only valid indices are from 0 to \(\lvert x \rvert - 1\), therefore:

Inside the summation, we know that:

If we choose an integer value for \(N\) such that \(2 \lvert x \rvert - 2 < N\), then \((k - j) \% N = k-j\):

Since \(y[j] = 0\) for all \(\lvert x \rvert \leq j \leq N - 1\), we can extend the summation’s bounds without changing its value:

Discrete Fourier Transform as Cyclic Convolution

We have thus far shown that the discrete Fourier transform of any sequence \(x\) can be calculated as the convolution of two sequences \(y\) and \(z\). \(y\) is a “zero-padded” copy of our input sequence, multiplied elementwise with a chirp signal, while \(z\) is itself a chirp signal. The advantage of this formulation lies in our freedom to choose the length of these sequences; their lengths \(N\) can be chosen to be any integer greater than \(2 \lvert x \rvert - 1\).

If we choose \(N\) to be the smallest power of 2 greater than or equal to \(2 \lvert x \rvert - 1\), then in the worst case scenario¹ where \(2 \lvert x \rvert - 1 = 2^p + 1\) (\(p \in \mathbb{N}\)), then \(N = 2^{p+1} = 4 \lvert x \rvert - 4\). If we apply the convolution theorem, here, we now see that the discrete Fourier transform \(\mathcal{F} \{ x \}\) can be calculated by taking the DFT’s of \(y\) and \(z\), then taking the inverse DFT of their elementwise product, and finally multiplying that result elementwise by another chirp signal:

All of these operations take linear time complexity with respect to \(N\), except for the inverse and forward discrete Fourier transforms. Since these DFT’s are calculated on sequences whose length is a power of 2, we know they can be calculated in \(O(\lvert x \rvert \log \lvert x \rvert)\) time using the Cooley-Tukey algorithm.

How Fast is a Fast Fourier Transform?

The guaranteed linearithmic time complexity of Bluestein’s algorithm unfortunately does not mean that the performance is just as good as we would get for Cooley-Tukey applied to a sequence of highly-composite length. Bluestein’s algorithm inflates the problem size to at least \(N = 2 \lvert x \rvert - 1\), while also adding an inverse Fourier transform step.

The limitations of Bluestein’s algorithm are apparent when one uses any fast Fourier transform implementation that relies on it, such as NumPy’s. In a Google Colab notebook, I ran numpy.fft.fft() on random real-numbered sequences ranging in length from \(2^{20}\) to \(2^{20}+24\); below is a graph showing the length of time taken to compute the discrete Fourier transform of each of the sequences:

You may notice that a longer sequence can take a shorter time to compute;² for example, the DFT for the sequence of length 1,048,580 took 0.155 seconds to compute, while for length 1,048,579 the execution time was 0.496 seconds, more than 3 times as long. This is because the greater length, 1,048,580, is highly composite, having 6 prime factors: 2, 2, 5, 13, 37, and 109. Meanwhile, the lesser length, 1,048,579, has only 3 prime factors: 7, 163, and 919.³

The “Ugly” Truth

Rather than being an alternative to Cooley-Tukey, Bluestein’s algorithm is better described as an (apparently high-overhead) “compatibility layer” for Cooley-Tukey. Bluestein’s allows Cooley-Tukey to efficiently process awkwardly-sized inputs, but never remotely as quickly as highly-composite lengths (even those of somewhat greater size).

The fast Fourier transform makes perennial appearances on discussions in internet fora and attention-grabbing listicles as among the most “beautiful” algorithms. If “fast Fourier transform” refers only to Cooley-Tukey applied to inputs of highly-composite lengths, I’m tempted to concur; that algorithm is just complex enough to be both interesting and elegant. But if “fast Fourier transform” refers to a program that uses a combination of Cooley-Tukey and Bluestein’s algorithm to efficiently process inputs of any length, then I can’t be sure I agree. Bluestein’s algorithm is, in my opinion, both fascinating and at least a bit unsatisfying. The fact that we can calculate a Fourier transform as a convolution is mind-boggling. The fact that this convolution is calculated using two Fourier transforms at least twice the length of our input size is a little disappointing.

Sources & Related Reading

• Smith, J.O., “Bluestein’s FFT Algorithm” in Mathematics of the Discrete Fourier Transform (DFT) with Audio Applications, Second Edition, http://ccrma.stanford.edu/~jos/mdft/, online book, 2007 edition, accessed 2026-05-01.
• L. Bluestein, “A linear filtering approach to the computation of discrete Fourier transform,” in IEEE Transactions on Audio and Electroacoustics, vol. 18, no. 4, pp. 451-455, December 1970, doi: 10.1109/TAU.1970.1162132.

Footnotes:

One of the most useful applications of the fast Fourier transform is in efficiently calculating cyclic convolutions. Paradoxically, cyclic convolutions can also be used to help efficiently calculate discrete Fourier transforms. This post will show the former; I will explain the latter point in future blog posts.

Discrete Cyclic Convolution

If \(y\) and \(z\) are two sequences of equal length \(N = \lvert y \rvert = \lvert z \rvert\), then the cyclic convolution of those two sequences, \(y \circledast z\), is defined as follows:

Where \(\%\) is the modulo operator.¹

Cyclic Convolution Visualization

The visualization below shows two input sequences \(y\) (vertically, along the left) and \(z\) (horizontally, along the top), the product of each of their elements in the big matrix in the middle. The output convolution, \(y \circledast z\) is shown on the bottom. You can discover the value and origin of an element by hovering the cursor over it. Those with a mouse can click-and-drag the input sequences to change their values.

\(N = \lvert y \rvert = \lvert z \rvert =\)

The Convolution Theorem

A cyclic convolution between two sequences \(y\) and \(z\) (length \(\lvert y \rvert = \lvert z \rvert = N\))—calculated naively (see above)—would require \(O(N^2)\) basic operations. However, using the Fourier transform, we can reduce this requirement to \(O(N \log(N))\) operations (assuming \(N\) is highly composite and we calculate the Fourier transforms using the Cooley-Tukey algorithm).

Let’s prove this:

The indices of the inner summation can be split² into two ranges, from \(0\) to \(m-1\) and from \(m\) to \(N-1\):

For the entire outer summation, note that:

or, if we define \(\mathcal{F} \{ y \} \odot \mathcal{F} \{ z \}\) as the elementwise product of \(\mathcal{F} \{ y \}\) and \(\mathcal{F} \{ z \}\), i.e.

then, applying the inverse discrete Fourier transform:⁴

A Note on Other Proofs

Most other proofs of the convolution theorem found on the public-facing web seem to forgo an explicit modulo operator in favor of (sometimes implicitly) treating the input sequences as “\(N\)-periodic” infinitely-repeating signals. For example, Professor Julius Orion Smith III’s free educational book, Mathematics of the Discrete Fourier Transform (DFT) with Audio Applications, does exactly this in its passage on the convolution theorem.

I suspect this convention is intuitive to anyone with sufficient background in signal processing. However, for newcomers such as myself, the unexplained elision of the modulo operator can be rather confusing. It’s also not immediately obvious how to represent and interact with an \(N\)-periodic infinitely-repeating signal in a computer program; the modulo operator, on the other hand, should be quite intuitive to anyone with a computer science background, and much more easily translatable into computer code.

Source and Additional Reading

• Smith, J.O. Mathematics of the Discrete Fourier Transform (DFT) with Audio Applications, Second Edition, http://ccrma.stanford.edu/~jos/mdft/, online book, 2007 edition, accessed 2026-04-27.

Footnotes:

為政以德，譬如北辰，居其所而眾星共之

He who exercises government by means of his virtue may be compared to the north polar star,¹ which keeps its place and all the stars turn towards it.

(Analects 2.1, trans. James Legge)

A 21st-century reader like you or me might think of those long-exposure photos of the northern sky, showing the North Star, Polaris, appearing almost perfectly still while the all other stars seem to revolve around it over the course of a night.

Is this the star that Confucius had in mind? Is Polaris (also known as α Ursae Minoris) a model for the ideal Confucian ruler? Well, no, in fact, because Earth’s north pole was not pointed anywhere near Polaris in Confucius’s time. The Earth’s celestial poles—each being a projection of its axis of rotation—“wobble”, a bit like a spinning top.

As every 2nd grader should know, the Earth’s axis of rotation is tilted (about 23.4°) relative to the plane of its orbit around the Sun. While the amount of tilt does not change much, the direction of that tilt has slowly but significantly wobbled, on a roughly 26,000-year cycle.

While thinking about the above passage by Confucius, I got the idea to try simulating what the night sky would look like at different points in the Earth’s cycle of axial precession. For illustrative purposes, I’ve removed the Sun and sped up the rate of rotation to 6 degrees per second (in reality, the Earth’s rate of rotation is about 15.04 degrees per hour).

First, here is what you would see looking due north in the night sky, assuming a view unobstructed by clouds (or, as would be the case at southerly latitudes, the ground beneath your feet):²

By virtue of its proximity to the north celestial pole, Polaris (the bright star at the center of our view) appears to move extremely little throughout the night.

Meanwhile, this is what the night sky would have looked like circa 479 BCE, the last year Confucius was alive:

Not only did the Earth’s north pole point nowhere near Polaris, the north pole isn’t particularly close to any bright star. In this time, Kochab, at just north of +82° declination, was the most northerly star of roughly the same brightness as Polaris (the celestial north pole is at +90° declination). Polaris was itself all the way down at +76° declination, quite far from the north pole. For a given Earthbound observer, Polaris could end the night more than 28° from where it had started; such a star can hardly be claimed to (as Confucius had described) “keep its place”!

Finding Confucius’s Beichen

So what, then, should we make of Confucius’s statement? Well the term I gave in translation above as “north polar star” might refer to something else entirely, so—in order not to bias the reader—I’ll leave it in transliterated Chinese: beichen (北辰). As for what on Earth (or rather, above it) beichen refers to, sources disagree:

Another Pole Star

Joseph Needham proposes³ that the ancient Chinese used a series of northern pole stars over the centuries. Needham does not outright say which of these stars he thinks Confucius is alluding to, but logically it must be:

• Kochab (β Ursae Minoris, HR 5563), a star nearly as bright as Polaris (though significantly further from the north pole at any time), which according to Needham was used as the pole star around 1000 BC, or
• “4339 Camelopardi”, an obscure star that was apparently a quite exact pole star circa 800 AD.⁴ I believe⁵ Needham is referring to the double star Σ 1694, or perhaps the brighter of its two components, HR 4893. According to Needham, this star was used as a pole star in the Han dynasty.

An Asterism

David Pankenier, in “A Brief History of Beiji (Northern Culmen)”⁶ alternately proposes that Confucius was referring collectively to the stars of the Northern Dipper (also known as Ursa Major, the Big Dipper, or the Plough). Indeed, Classical Chinese does not have grammatical number, so “beichen” could just as easily be “the north stars” as “the north star”.

The Pole Itself

Finally, E. Bruce Brooks and A. Taeko Brooks write:

There was in this period no literal pole star, the immediate circumpolar region being essentially empty until much later times… Whether we imagine a polar void or (as the text seems to require) a polar star, the thrust of the saying is the magical power of inactivity.⁷

Brooks & Brooks’s (tentative) identification of beichen with the empty celestial pole—rather than any luminous heavenly body—may receive some support from the oldest Chinese dictionary,⁸ though I think that evidence is weak at best.

Words from the Ancients

Finally, I’ll contribute a hypothesis of my own. Let’s recall that Confucius described himself as:

A transmitter and not a maker, believing in and loving the ancients

述而不作，信而好古”

(Analects 7.1)

Could this ancient philosopher be transmitting a statement about (or even from) a yet-more distant antiquity? As it happens, Thuban (α Draconis, HR 5291), a star significantly less bright than Polaris, though still easily visible with the naked eye, was at one point even closer to the celestial north pole than Polaris will ever be:

It would be quite remarkable indeed if Confucius was talking about this erstwhile pole star; the Earth’s celestial north pole was at its closest to Thuban around 2800 BC, almost as distant to the time of Confucius as his time is to mine. And there is another reason to discount this extraordinary possibility: Confucius makes no note of the north pole having drifted away from this former pole star–a shift that I imagine would carry great cosmological significance, and merit much commentary from Confucius.

It might be tempting to just pick one of these hypotheses and present it as the truth, but I think the most honest answer is that we simply don’t know! Confucius’s “beichen 北辰” could be an asterism, a point in the sky, or one of several stars that have approached the pole over the course of human history. Perhaps, we may never know which one it was.

Rendering the Sky of Spring and Autumn

To make the night-sky renderings you see above in this post, I created a program I’ve named “Beichen”. I made this program using Rust, WebAssembly, and JavaScript. I’m sure I could have easily used any of a number of pre-existing programs, but I decided to make something new anyway, for fun. For my star dataset I used the Yale Bright Star Catalogue (hereafter, “YBS”).

This catalogue and others generally give stars using right ascension and declination, which are a type of spherical (rather than Cartesian) coordinates. If you’re like me, you might imagine it would be inherently easier to rotate spherical coordinates, which are just a pair of angles.⁹ Indeed, it is very easy to rotate around the polar axis by simply adding-to or subtracting-from the azimuthal angle (which in our case would be right ascension), but it is apparently a pain to rotate around any other axis, so I instead simply convert the star coordinates to Cartesian coordinates using these formulae (where \(\theta\) is right ascension and \(\phi\) is declination):

and then rotate them using the usual rotation matrices that you’d learn about in a typical linear algebra class, (specifically using the ndarray crate, my only non-web-related dependency).

Rather than using WebGL (the recommended way of rendering 3D content on a CanvasRenderingContext2D), the program uses a 2D canvas context as a rasterization engine for the stars (projected onto a pinhole camera matrix). In retrospect, my approach was likely very suboptimal for performance; making all of those rendering calls from inside my WebAssembly program means lots of crossing the Wasm-JS boundary.

Skies Past, Present, and Future

Here is an interactive version of Beichen, the night sky viewer. You can click and drag to move, or scroll to zoom. Hovering over a star will show its designation, magnitude, and coordinates (unless you deselect that option) Try out the checkboxes below to show equatorial coordinate lines, the ecliptic, and the north and south pole’s circles of precession. The intended purpose of this program is to show the night sky at whatever year² in the past or future the user wants, but ultimately you can use it however you like.

What’s so Special About 2102?

Regardless of whether I’m reading a millennia-old Chinese text or conversing with my fellow 21st-century Americans, I often find that people believe themselves to be living in an era of decline—a lesser shadow of some glorious past. Back then, things were better; sons were filial, names were rectified, and coffee only cost a dime. I can’t say with certainty whether things used to be better here on Earth, but I can tell you that the state of affairs up in the heavens is very much the opposite. Whereas our ancestors (at least, for those of us with roots in the Northern Hemisphere) lived under a sky that appeared to revolve around an empty point, our 21st-century celestial dome converges on a brilliant gem of a star almost precisely at its center—that is, Polaris.¹⁰ What’s more, not only is the present greater than the past, the future promises to out-do our present!

Online pop science sources tend to be a bit vague about exactly how long the Earth’s precession cycle takes, generally claiming that the Earth’s axis of rotation will return to pointing at the same spot after approximately 26,000 years. Rather than doing more research to find a rigorous source, I thought it would be fun to try to calculate the cycle’s length using the data in the Yale Bright Star Catalogue. The YBS reports observations for two epochs: B1900.0 & J2000.0. By comparing the apparent position of stars in 1900 vs. 2000, we can estimate the amount that the Earth’s axis precesses in a period of 100 “years”.¹¹¹² If we take Wikipedia as our source of truth, trusting that the Earth’s northern orbital pole is at a declination of +66° 33′ 38.84″, then we find that the median star observed in the YBS has appeared to “rotate” about 1.39651° around the orbital pole.¹³ Therefore:

If we convert Polaris’s location to ecliptic coordinates¹², we find that its ecliptic longitude is about 1.4323328° away¹⁴ from that of the celestial north pole (both positions according to the J2000.0 epoch). Calculating the years from this angle, we find that the Earth’s celestial north pole will point closest to Polaris in \(\frac{1.4323328°}{360°} \cdot 25{,}778.47 \textrm{ yrs.} = 102.56\) years.¹⁵ Taking a “ year” as 365.25 days (see note¹¹), we can use a date calculator to find that our celestial north pole will point closest to Polaris on July 26, 2102, after which it will appear to slowly drift further and further away.

It’s also possible (practically guaranteed, really) that other factors such as nutation, proper motion, and who knows what else, could influence this date—but I believe only very slightly (as a sanity check, I’ve also played around with Stellarium and found that it generally agrees with my own software). If you think that my calculations are wrong, if you have any new arguments, evidence, or hypotheses for the meaning of the word beichen as used by Confucius, or if you just enjoyed my article, please leave a comment below, and I’ll try to get back to you! As always, thanks for reading.

Footnotes

The Discrete Fourier Transform

If \(x\) is a sequence of complex numbers with a length \(\lvert x \rvert\) and a starting index of 0, then the discrete Fourier transform of \(x\), \(\mathcal{F} \{ x \}\), is defined as follows:

Since complex exponentiation is so commonly used in Fourier transforms, we’ll define a helpful term \(W_N\) as follows:

i.e. \(W_N = W_N^1\) is a \(\frac{1}{N}\)-turn rotation in the complex plane (starting at 1). \(W_N^2\) is a \(\frac{2}{N}\)-turn rotation in the complex plane, etc. Substituting to the original discrete Fourier transform definition, we get:

Naïvely evaluating this equation for each of the \(\lvert x \rvert\) different output frequency buckets of the DFT (\(k = 0, 1, \ldots, \lvert x \rvert - 2, \lvert x \rvert - 1\)) requires a summation of complex products of the \(\lvert x \rvert\) samples in the signal, thus giving any naïve DFT algorithm a time complexity of \(O(\lvert x \rvert^2)\).

The Cooley-Tukey Algorithm

If \(\lvert x \rvert\) is a composite number, we can pick two natural numbers \(r\) and \(d\), such that:

This allows us to change the single summation over \(j\) into nested summations:

Similarly, we can define variables \(k_0\) and \(k_1\) such that \(k = k_1 d + k_0\). Let:

In other words, \(k_1\) is the quotient and \(k_0\) is the remainder of the Euclidean division¹ of \(k\) by \(d\).

This allows us to again re-formulate the discrete Fourier transform:

Since \(W_{\lvert x \rvert}^{j_1 k_1 \lvert x \rvert} = (e^{-i \frac{2 \pi \lvert x \rvert}{\lvert x \rvert}})^{j_1 k_1} = 1^{j_1 k_1} = 1\), therefore:

At this point, we can split the elements of \(x\) into sub-sequences corresponding to modulo classes. Let \(x_{r}^{j_0}\) be a sequence whose elements are equal to the elements of \(x\) whose indices are equivalent \(j_0\) modulo \(r\). More formally, these sequences (of which there are \(r\) total) can be defined as follows:

Substituting this sequence definition, we get:²

Let’s consider how long it will take to evaluate the discrete Fourier transform in this formulation:

• First, we need to pre-compute the discrete Fourier transforms of the \(r\) different sequences \(x_r^{j_0}\) (i.e. \(\mathcal{F} \{ x_r^0 \}, \mathcal{F} \{ x_r^1 \}, \ldots \mathcal{F} \{ x_r^{r-1} \}\)), each of length \(d\). Assuming each DFT is performed naïvely, this will take \(O(r \cdot d^2) = O(\lvert x \rvert \cdot d)\) operations.
• Then, we need to compute the \(\lvert x \rvert\) values of \(\mathcal{F} \{ x \}\), each of which requires a summation over \(r\) different values of \(j_0\) for \(\mathcal{F} \{ x^{(j_0)} \}[k_0]\), taking \(O(\lvert x \rvert \cdot r)\) operations.

Added together, these two sub-routines require \(O(\lvert x \rvert \cdot d + \lvert x \rvert \cdot r) = O(\lvert x \rvert \cdot (d + r))\) operations, possibly a significant improvement from the original \(O(\lvert x \rvert^2)\) complexity of the original naive formulation, depending on the values of \(r\) & \(d\). More importantly, this manipulation can be applied recursively. Specifically, each of the \(r\) discrete Fourier transforms of the \(d\)-length sequences \(x_r^{j_0}\) can be broken down into \(r'\) Fourier transforms of length \(d'\), assuming that two natural numbers exist such that \(d = r' \cdot d'\).³ In the ideal⁴ case where \(\lvert x \rvert = 2^n\), \(n \in \mathbb{N}\), calculating the Cooley-Tukey algorithm will require \(O(\lvert x \rvert \cdot (2 + 2 + \ldots + 2)) = O(\lvert x \rvert \cdot 2 \cdot \log_2(\lvert x \rvert)) = O (\lvert x \rvert \cdot \log(\lvert x \rvert))\) operations.

The Cooley-Tukey algorithm can also be used to calculate the inverse discrete Fourier transform with only very slight modification. In fact, the original Cooley-Tukey paper (see “related reading”) specifically described an algorithm to compute the inverse discrete Fourier transform, not the “forward” DFT. I will leave the Cooley-Tukey iDFT algorithm as an exercise for the reader.

However, note that the Cooley-Tukey algorithm gives no speed-up for input sequences of prime length, and provides relatively little speed-up when the factors of the input length contain large primes. To efficiently compute the DFT for sequences of prime or even non-highly-composite lengths, we will need additional algorithms. Ultimately, however, these other FFT algorithms generally depend on Cooley-Tukey for part of the computation. I plan to cover at least one of these techniques—Bluestein’s algorithm—in a future blog post(s).

Cooley-Tukey Interactive Visualization

This visualization shows how the discrete Fourier transform of some signal \(x\) is computed using the Cooley-Tukey algorithm. The black boxes at the very bottom are the input signal. While the DFT can be applied to complex signals, I’ve restricted the sample values of the input signal to be real numbers, for simplicity’s sake (this mimics some real-world applications, such as performing a DFT on an audio recording). You can click and drag on the input boxes to change their values.

The grey circles and the sometimes visible white “clock hands” inside of them represent the complex exponent \(W_N^x = e^{-2 i \pi \frac{x}{N}}\) for some \(N\) (e.g. \(\lvert x \rvert\), \(d\), \(d'\), etc.) and some \(x\). These complex exponents, which are equivalent to rotations in the complex plane, are applied to the relevant input value. Unlike the usual convention, I’ve decided to show the real component of the complex plane as vertical (“up” is positive-real) and the imaginary component as horizontal (“right” is imaginary-positive). You can see where the input value is drawn from by hovering the mouse over a given “rotation” box.

The sum of those rotated input values is added together to calculate one element of a discrete Fourier transform. Hover over a white “output” box of a discrete Fourier transform in the visualization to highlight the column of “rotation” boxes that it was summed from.

\(|x| =\)

Available factors:

A Quick Rant on Word Choice

I’ve noticed an irritating and confusing tendency among many people–including published authors–when talking about discrete Fourier transforms. I find they often use the phrase “fast Fourier transform” (or perhaps more often, the abbreviation “FFT”) when they mean “discrete Fourier transform” (or “DFT”). I think this is wrong and confusing; to understand why, imagine you have a list:

x = [10, 3, 2, 19, -2]

would it make sense to refer to the following list as the “mergesort” of the previous list?

y = [-2, 2, 3, 10, 19]

I think most people would find that very strange. We can say that the second list is the result of sorting the first list, but we don’t know anything about the specific algorithm that was used to sort the list. It could have been sorted using mergesort, quicksort, heapsort, bubblesort, bogosort, or any other sorting algorithm (in reality, I just sorted this one in my head). However, even if I had used mergesort, y still wouldn’t be the “mergesort” of x, it would still just be “the result of sorting x”.

Similarly, the output of an FFT algorithm should not be referred to as “an/the FFT of” anything. In theory, calculating the DFT of a sequence using Cooley-Tukey gives the exact same result as calculating that DFT using a naïvely-implemented DFT algorithm. In practice, Cooley-Tukey will probably give a slightly more accurate result since there are fewer total calculations and therefore fewer opportunities for floating point round-off error.

This is not just irritating to me; I think it causes confusion among the public. A friend of mine–an intelligent mathematics major and software engineer–recently asked me “what would I lose by taking the fast Fourier transform instead of the ‘normal’ Fourier transform? I’ve always just taken the normal Fourier transform”. He probably inferred from the way many people throw around the phrase “FFT” that the “fast Fourier transform” is some kind of approximation or related concept yet distinct from the discrete Fourier transform or Fourier transforms in general, rather than an algorithm for calculating such.

Sources & Related Reading

• An Algorithm for the Machine Calculation of Complex Fourier Series by James W. Cooley & John W. Tukey

Footnotes:

As a former volunteer at my alma mater’s international student program, I happen to have some knowledge about OPT and the policies concerning student (F-1) visa holders that led me to suspect there were major errors in Paul Gosar’s statement regarding the bill, as well as in the articles from advocacy groups that he cites. In addition to some superficial and obvious errors, such as claiming that OPT was “expanded by three years by the Obama Administration” (it was in fact expanded only by seven months), Gosar’s post claims that:

These foreign workers are exempt from payroll taxes[,] making them at least 10-15 percent [sic] cheaper than a comparable American worker. NumbersUSA reports OPT costs the Social Security and Medicare trust fund [sic] $4 billion annually.

The document from NumbersUSA (an immigration restrictionist advocacy group) cites a 2024 article from the Center for Immigration Studies (CIS, another restrictionist group), which in turn simply scales up an earlier estimate from a 2015 CIS article.

The author of the 2015 article, David North, tallies the number of OPT & OPT STEM extension approvals, then multiplies by 12 or 17 months respectively (the STEM extension was only 17 months long at the time) to determine the number of years an OPT worker will have worked in the United States without being liable for Social Security and Medicare taxes. He proposes an estimated average OPT salary based on that of the average college graduate, and estimates the total loss to Social Security & Medicare like so:

Many OPT Workers do in Fact Pay FICA

There are a few problems with this estimate, but I will start with the most glaring one: it is not true that all or even nearly all post-completion OPT workers (and their employers) are exempt from FICA. The reason that some OPT workers are exempt from FICA is that they are nonresident aliens. And while F-1 student visa holders (including OPT workers) are always considered nonresident aliens for immigration purposes, they are only treated as such for tax purposes until they pass the “substantial presence” test.⁴

The “substantial presence” test is rather complicated, but in a typical case, an international student will not be considered substantially present for their first 5 calendar years in the United States, due to a policy exempting F-1 visa holders for that period. The visa holder will usually come to be considered “substantially present”—assuming they spend a majority of that year in the United States—beginning January 1st of their 6th calendar year in the United States. The visa holder is then treated as a resident alien for tax purposes and is thus subject to the payroll taxes funding Social Security & Medicare, a.k.a. FICA. The only likely way for an international student not to be considered a resident alien is if they spend fewer than 183 days in the United States in their final calendar year in the United States.

Substantial presence is not some niche exception; it ultimately applies to a very large number—perhaps a majority—of post-completion OPT workers at some point in their OPT period. Keep in mind that degree programs are generally mis-aligned to calendar years and therefore push F-1 holders/OPT workers into resident alien tax status sooner than if they weren’t.⁵

Unfortunately, I couldn’t seem to find public reporting from ICE or USCIS to break down the share of OPT participants by level of degree attained. However, based on this report from the Niskanen center, which obtained additional statistics on OPT educational attainment levels via a FOIA request, we can see the different levels of education attained by OPT workers:

I have attempted to combine the Niskanen data, official ICE-reported numbers on OPT & OPT STEM extension approvals in 2022, and no small number of naïve and arbitrary assumptions⁶ to estimate the total number of years worked by post-completion OPT workers as nonresident aliens (NRAs) vs. as resident aliens (RAs) for tax purposes.

I estimated that in 2022, a narrow majority—about 53%—of all post-completion OPT time was worked while in resident alien status. In other words, if this estimate is accurate, most OPT workers and their employers at any given time are paying into the Social Security and Medicare trust funds at exactly the same rate as a US citizen and their employer would be. This estimate comes with much uncertainty; the result can shift quite dramatically depending on how many master’s graduates we assume to be substantially present by the time of graduation. That said, I think this estimation is useful as an exercise showing how complicated it is to estimate the amount of FICA paid by OPT workers, as well as a demonstration of the magnitude of CIS, NumbersUSA, and Paul Gosar’s error.

Nonresident Alien Tax Status is not Always Advantageous

Excepting FICA exemption, nonresident alien status generally leads to a higher tax burden. Nonresident aliens other than Indian residents⁷ cannot take the standard deduction, and almost no nonresident aliens⁸ can take the earned income tax credit, the American opportunity tax credit, the lifetime learning credit, or many other deductions and credits that an American worker would be able to take.

In the above chart I have plotted the additional tax burden caused by a lack of standard deduction compared to the tax burden avoided by being exempt from FICA.⁹ Note that the lower one’s income, the less of a tax advantage one is conferred by nonresident alien status. At income levels of around $20,000 or less, more than the entire employee FICA advantage for NRAs is wiped out by the increased tax burden due to a lack of standard deduction. You may think it unlikely that a post-completion OPT worker would make only $20,000 (or less) in a year; if so, I think you would be correct (see next section).

However, many international students likely earn wages in this range when they work on-campus while enrolled as a student. International students (like their domestic peers) can take part-time jobs at their home institutions, often helping to run facilities such as libraries and cafeterias or serving as teaching assistants (or even instructors, in the case of PhD students). In the case of these student workers, nonresident alien status loses its main advantage (that is, FICA exemption), because all student workers at colleges and universities—including U.S. citizens—are universally exempt from FICA. In other words, F-1 student visa holders are most likely to have nonresident alien status precisely when it is least advantageous, often increasing their total tax burden relative to a US citizen’s.

I would like to pause very briefly to bring up just how unfair the tax treatment of international students can be. Nonresident aliens are excluded from paying FICA for a specific reason—FICA specifically funds social insurance programs that most nonresident aliens will never be able to collect from (granted, this logic doesn’t particularly apply to their employers). What is the justification for why we take an extra 10% out of the wages of an international student paying their way through college by washing dishes in the cafeteria? If anyone can tell me a good reason, I’m all ears, but until then I’ll assume it’s nothing more nor less than “because we can”.

OPT Workers are Probably Paid Much More than the Average New College Graduate

In CIS’s 2015 estimate of lost FICA income due to OPT workers, author David North estimates the average income of an OPT worker as $50,000 per year, based on a 2011 report showing the average new college grad salary as $50,034.¹⁰ I’m a bit surprised he didn’t try to find a newer estimate or at least adjust for general wage growth, as this would have allowed him to claim that OPT was depriving Social Security and Medicare of an even larger amount of money.

On the other hand, I think this would contradict a larger narrative that Rep. Gosar and CIS seem to believe about OPT. Namely, they assert that OPT is fundamentally a sham program to funnel “inexpensive foreign labor” (in Paul Gosar’s words) into American jobs. Listening to immigration restrictionists, you’d get the impression that the average OPT worker has “graduated” from some fly-by-night for-profit diploma-cum-visa mill, or at best an un-selective associate’s degree program, in order to exploit the OPT “loophole” with their sham degree and work in some low-skill job with no real connection to their supposed field of study. As David North wrote for CIS in 2018, “many a pizza place is staffed with OPT workers”.

This certainly does not resemble the experiences of my international classmates at Macalester College, who often out-earned me at renowned firms such as Google, Ernst & Young, or Merck, to name a few. But perhaps my view from a selective liberal arts college is not representative of the typical OPT worker and their career. This spreadsheet published by ICE showing the top schools for active OPT records in 2022 might give us some insight into what kind of new college and university graduates are participating in OPT. Here are the top 20 schools:

Not only are these schools reputable, legitimate institutions of higher learning, many of them are the most prestigious universities in the entire world!¹¹ The top 100 schools for OPT authorizations includes almost the entire Ivy League (with the exceptions of Dartmouth and Princeton), Stanford, MIT, Duke, Johns Hopkins, and the University of Chicago. This is remarkable, considering that these exceptionally prestigious schools are generally not particularly large; for example, Harvard’s total enrollment of 21,278 is smaller than that of 11 California State University campuses, only 1 of which is among the top 100 OPT schools.

Using data on the top schools for OPT authorizations in 2022, I compared several schools’ share of all degrees awarded to their share of OPT workers. In that year, compared to the general population of new bachelor’s degree recipients, an OPT authorization recipient was:

• over 7 times as likely to have graduated from Stanford,
• over 11 times as likely to have graduated from MIT, and
• over 13 times as likely to have graduated from Harvard

this discrepancy is due in part to the fact that these institutions grant far more master’s degrees compared to similarly-sized institutions. However, even comparing to the average master’s recipient, an OPT worker is:

• 2.23 times as likely to have graduated from Stanford,
• 2.77 times as likely to have graduated from MIT, and
• 1.92 times as likely to have graduated from Harvard

Sources such as this article from the St. Louis Federal Reserve Bank, as well as common sense, tell us that graduates of more selective universities tend to significantly out-earn graduates of less selective institutions. We also know that most OPT recipients are master’s degree holders; according to this report from the Bureau of Labor Statistics, master’s degree holders out-earned bachelor’s degree holders by 15% in 2022. Contrary to Paul Gosar’s description of OPT workers as “inexpensive foreign labor”, I suspect that OPT workers significantly out-earn the average new college graduate.

Gosar et al.’s Assumptions about Employment are Likely Wrong

Representative Gosar, NumbersUSA, and CIS seem to assume that OPT workers’ only impact on the U.S. economy is to take jobs away from U.S. citizens and green card holders who could have worked these jobs and now must be unemployed as a result. In their counterfactual analysis, these critics of OPT don’t consider the possibility that the quantity and quality of labor available could itself influence the rate of formation, expansion, or closure for firms. They also don’t consider that the quantity and quality of labor available could affect the quality and price of goods & services, and by extension the real incomes of U.S. citizens. They do not entertain the possibility that some of the jobs worked by OPT workers could have otherwise been outsourced, taken by a foreign competitor, or simply gone unfilled. Nevermind that many former international students have founded multi-billion dollar companies that employ many thousands of U.S. citizens.

There is a large body of economics research analyzing the effects of skilled immigration on the US economy and workers. I am not by any means an expert in economics, however, The University of Chicago, itself home to one of the most highly-regarded economics departments in the world, conducts polls of economists on various economics questions. For example, on the question of whether reducing the number of H-1(b) visas would increase employment opportunities for American workers, 45% of polled economists responded “disagree”, 36% responded “strongly disagree”, and 19% responded “uncertain”; 0% of economists responded “agree” or “strongly agree”. Put differently, the field of economics generally believes that H-1(b)—a program that allows foreign skilled workers similar to those in OPT to work temporarily in the United States—does not significantly negatively impact the employment prospects of American workers.

I would be interested to see whether these experts feel similarly about OPT as they do H-1(b); if so (and if they are not all wrong), then each OPT worker may in fact not be simply displacing an American worker, but instead adding new value to the American economy that wouldn’t have existed otherwise. In that case, I think it is only fair to consider some share of the FICA paid by OPT workers to be a net gain relative to the counterfactual in which those OPT workers were not allowed to work in the United States. Rather than depriving Social Security & Medicare of badly needed funds, OPT workers may be contributing more than would otherwise be present.

This paper, which studies the effects of the random lottery for H-1(b) visas on firms, estimates that each H-1(b) lottery win for a firm increase that firm’s total employment by 0.83. This is only a firm-level estimate; the effect on total employment in the United States economy may be greater or lesser than this number.

Out of a desire to estimate the net effect of OPT on the Social Security & Medicare trust funds, I’ll split the difference between:

• Rep. Gosar, CIS, and NumbersUSA, who implicitly assume that each OPT worker adds roughly 0 new employment to the United States, and
• the 42 expert economists polled by the University of Chicago, who seem to think that each skilled worker (in their case, in the H-1(b) program) adds approximately 1 (or more) total comparably-employed workers to the United States economy.

therefore, I will assume that each OPT worker only increases total employment (comparable in compensation to the job worked by an OPT worker) in the United States by 0.5. Put differently, I’ll suppose, for the sake of estimation, that OPT workers really do displace American workers (for the duration of their OPT periods), but because a) they increase firm profitability b) lower the cost of goods & services to American consumers, and c) work some jobs that may have otherwise gone unfilled or taken by a foreign competitor, they generate enough additional economic value in the U.S. that the number of displaced workers is only half of the number OPT workers.

Conclusion

To estimate the net gain or loss to the Social Security and Medicare trust funds, we must compare the reality of the US with OPT workers to the counterfactual without OPT workers. First, the FICA revenue generated in reality by OPT workers (directly or indirectly):

minus the counterfactual (in which OPT does not exist):

where:

• \(\textrm{OPT}_{total}\) is the total number of OPT worker-years
• \(\textrm{OPT}_{RA}, \textrm{OPT}_{NRA}\) is the number of OPT worker-years spent in resident or nonresident alien status, respectively
• \(\textrm{NewEmployment}\) is the total (worker-years of) employment added to the US economy by each worker-year of OPT
• \(\textrm{AvgSalary}\) is the average salary of OPT workers
• \(\textrm{FICA}\) is the total FICA rate (employer- and employee-)

Subtracting reality from the counterfactual gives us:

Now I’ll substitute the values we estimated earlier. This report from the National Association of Colleges and Employers says that new bachelor’s graduates in 2022 took home a median starting salary of $60,028. To give a very rough estimate of the salary of OPT workers (who, on average, are master’s degree holders), I’ll increase that by 15%, to get an average salary of $69,032.20.¹²

I estimate that the total net gain to the Social Security & Medicare trust funds due to the existence of OPT could be $81,184,248.81. I don’t expect anyone to take this estimate too seriously; I had to make arbitrary guesses for many very important values.⁶

That said, this estimate is at least as valid as that of CIS, which through a chain of blindly trusting references, ended up being cited by a member of United States Congress as if it were fact—despite a blatant error, which could easily have been revealed by anyone through a simple web search.

OPT workers are by and large graduates of quite selective colleges and universities, who in fact do contribute significantly to American social insurance programs from which they ultimately might never collect. Even from the most self-interested, nationalist perspective, these are among the most desirable people in the world to have in any country. When I read Paul Gosar describe OPT workers—which many of my closest friends have been at one time or another—as “cheap foreigners”, I was more than a little insulted on their behalf. I hope this blog post serves to correct the record. Most OPT workers are anything but cheap, low-quality labor incentivized by unfair tax breaks; they are some of the best and brightest of our workforce.

Rather than making yet another futile attempt to eliminate the OPT program, I would suggest that Congressman Gosar instead propose legislation to reduce or eliminate the time (currently 5 calendar years) that F-1 student visa holders are exempt from the substantial presence test. If the years of exemption were eliminated entirely, this would cause virtually all OPT workers to be subject to FICA for the entirety of their OPT periods, making international students contribute even more to the coffers of Social Security and Medicare than they already do. However, this would come at a cost to the United States Treasury, increasing our national deficit, as international student workers would then be able to take credits and deductions otherwise denied to them.

Addendum: No Minor Blunder

David North, the author of the 2015 CIS article providing the original framework for estimating FICA revenue supposedly lost to OPT, writes that he volunteered to help graduate students—including international students—file their taxes. And yet, he writes:

The assumption is that all these years of tax-free status were used; in reality, probably a small fraction of the tax-free status was not used because the recent graduate either moved on to another visa category, left the nation, or, in a very few cases, died.

North gives no indication that a typical OPT worker would, in fact, be fully liable for FICA for a significant share (possibly most or all) of their OPT time. I’m tempted to presume that he knew how many OPT workers used OPT after getting master’s degrees, which in some cases could allow them to be FICA-exempt for nearly the entirety of 1-year OPT and a STEM extension (assuming they did not get their bachelor’s degrees in the US, too). However, he specifically uses the average salary of new college graduates as the estimated salary of OPT workers, indicating he assumes they are primarily bachelor’s degree holders.

This is no small blunder! Determining tax residency is literally the first (and probably most important) step in filing taxes as a noncitizen. It determines whether the noncitizen filer can submit a normal form 1040 (resident) or 1040-NR (nonresident) as well as which tax filing software the filer can use (e.g. TurboTax vs. Sprintax). So either:

• North was woefully incompetent to assist any noncitizen living in the U.S. with their taxes, possibly causing them to file completely incorrectly
• North lied about having helped international students with their taxes, or
• North knew that many OPT workers would not be exempt from FICA, but intentionally lied to his readers by writing that nearly all OPT workers are exempt from FICA

What’s more, North did not just make this misrepresentation in a blog post (that was ultimately read and cited by a member of the United States Congress), he made this same claim in an amicus brief submitted to a federal court in 2019!¹³ This level of carelessness from a professional think-tank writer who had written well over a dozen articles on this topic before submitting this amicus brief is remarkable, especially considering this information was easily available to the public on the IRS website as early as 2013.

Thank you to my friend Stephanie Hou, who helped to proofread this post. Any mistakes are my own.

Footnotes:

I recently (read: several months ago) watched a video about so-called “color-corrective lenses” that supposedly enable colorblind people to see the full range of colors that people with full color vision can see. The video’s creator argues—successfully, in my opinion—that these color corrective lenses are essentially a scam. They cannot restore full color vision, and only assist in distinguishing pairs of colors by entirely blocking the light of one of the colors. They may have some practical use, but only at significant cost; for example, green traffic lights may appear completely black, as occurred for one reporter who tried on the glasses and drove his car (!!!) with them on.

What really got my attention was the video creator showing some of the colorblind tests that he, a colorblind person, had tried and failed. These tests—known as Ishihara test plates—consist of circles filled with irregularly sized and placed dots I noticed that I too often could not read these test plates. Somewhat alarmed, I took several online colorblindness tests (including one furnished by a color corrective lens company). These tests generally indicated that I had mild deutan colorblindness. The cone receptors in my retinas that should be activated by green light are lacking in quantity or quality, and therefore my ability to distinguish red from green is significantly worse than someone with full color vision.

An example of an Ishihara plate, showing "74" in green dots surrounded by orange dots (from Wikipedia)

I was particularly shaken when I showed one of the Ishihara test plates to my friends & girlfriend. The plate above is the number 74 in green dots surrounded by orange dots; my friends & girlfriend told me they could easily perceive it as 74. This 74 was not (and still isn’t) clear at all to me! I can see that there’s a number, but I originally thought it might be a 21, and still can’t help but see it as a 21 sometimes.

I can't read this shirt. You can buy it from here, if you think it's funny

I became fascinated with Ishihara test plates and things like them (see above). I tried a few programs that I found to make my own Ishihara plates but none of them quite satisfied me in terms of power and customizability, so I decided to make my own using Rust and WebAssembly. This was my first time using WebAssembly, and I found the Conway’s Game of Life WASM tutorial very helpful, as well as Carl M. Kadie’s post Nine Rules for Running Rust in the Browser.

Creating the Algorithm

I decided to start with the simplest generation algorithm I could think of. First, we load the image as an array of pixels; we decide which pixels in the image should be “in” versus “out” depending on whether the pixel has a luma value greater or lesser than 0x7F (i.e. 127, or 50% of maximum illumination). Then we generate dots with random radii and coordinates within the image. If a dot doesn’t overlap with any already-added dot, we add the dot to the image. The dot’s color will depend on whether more than 50% of the pixels inside of it are marked as “on”; if so, it will be drawn with the “on” color, otherwise it will be drawn with the “off” color.

This algorithm worked surprisingly well, except that it slowed down very quickly as the number of dots grew; total checks for overlapping dots grew quadratically with the number of dots, i.e. \(O(n^2)\). To cut down on required operations, I kept the added list of dots sorted by x-coordinate and used binary search to narrow down the list of dots to check for overlap to just those dots whose x-coordinate could possibly be in range of the new, candidate dot.

The text "HELLO WORLD", with (bottom) and without (top) a set maximum share of the dot's area that can cross the on/off boundary

I also noticed that the “in” and “out” dots sometimes crossed “in/out” boundary quite significantly, which made the outline of the text or number represented unclear. To make up for this, I added a user-set “tolerance” parameter, which defines the maximum share of a dot that can contain pixels of the “wrong” on/off value.

Playing with the Ishihara Generator

I still don’t have a good pipeline for generating the black-and-white text images to use as input for the Ishihara test generator; my quick-and-dirty solution is to make a Google Doc with very large font bold text and take a screenshot of that. I’ve noticed that if you set padding (the minimum space between dots) and tolerance very low, you can end up with images where you can see the outline of text even with identically-colored dots:

Even without any coloring, you can sometimes see the outline of text if the tolerance parameter is set very low. Here, the tolerance is set to 0% and padding is set to 0

Technically, you don’t have to input a black-and-white image of text. It’s also fun to play around with wacky colors on any image with significant numbers of high-luma and low-luma pixels. For example, check out this cool visual output from a picture of an astronaut in Earth orbit:

Trippy!

Here’s the link to my program that I used to make all these images. The only thing you should need to use it is a modern web browser. Thanks to the power of WebAssembly, computation happens on the client-side, so you don’t even need a persistent network connection.

Other than a passing mention of the binomial distribution, Silver doesn’t “show his work” with much detail. Since probability math can be really easy to get wrong (at least for me!), I thought I’d take a stab at trying the brute force option of simulating polls in a hypothetical dead tie, i.e. exactly 50% of decided voters plan to vote for each of the major candidates, Harris & Trump (I also happen to be a computer programmer by hobby and profession, so maybe I’m just a hammer looking for a nail).

This little project was made possible thanks to Nate Silver’s blog, Silver Bulletin, collecting and distributing poll results. Here are the data files containing the poll results that I used for Wisconsin, Pennsylvania, and New Hampshire.¹

Simulating Wisconsin polls²

If the exact same number of likely or registered voters (depending on which poll) plan to vote for Harris as Trump, we can easily simulate the act of surveying them by flipping a coin. Even more easily, we can run the random number generator on my computer and checking whether the output floating point number is greater than 0.5; if it is, that’s a Trump voter. Otherwise, that’s a Harris voter.

After we simulate our polls, lets extract our statistic of interest: the mean absolute margin. For example, if I have three polls with margins:³

then their absolute margins are:

and the mean absolute margin for this universe of polls is:

Here’s what the actual polls⁴ in real world Wisconsin done by real pollsters look like:⁵

An average poll of Wisconsin has one candidate beating the other (some of them Trump beating Harris, some of them vice versa) by about 2% (or 0.203 0.0203, as shown in the graph). While the trendline is not terribly strong, we do find that the absolute margin of a poll goes down as sample size goes up, as we’d expect if the race were truly tied.

And here’s a simulation of what those polls could look like in an alternate universe where pollsters perfectly randomly sample the same number of people in a perfectly matched race between Donald Trump and Kamala Harris:

The output of this simulation certainly differs from our observed results–our mean absolute margin is a full percentage point higher than our observed one. That doesn’t prove anything on its own, though; maybe this simulation of the Wisconsin polls just happened to result in a high mean absolute margin by chance.

Simulating a multiverse of polls

What happens if we run that simulation many, many times, keeping track of the resulting mean absolute margin for each simulation? Let’s look at the histogram we get when we do that:

Whoa! Our observed mean absolute margin of polls (the dashed red line to the left) is way lower than any of the MAMs in the multiverse where Harris and Trump are neck-and-neck. In fact, the lowest MAM out of 100,000 simulated universes is 0.02092 or 2.092%, still 0.06 percentage points higher than our observed MAM. Does this mean something is wrong? Well, I can’t think of any way these polls could get consistently closer margins than our simulations while still remaining scientifically valid. It’s hard to get a low variance estimate of a mean without increasing your sample size; that’s why the sample size \(n\) is so important in scientific papers.

Recall also that we generously assumed the candidates had exactly even shares of decided voters. The more imbalanced the share of voters between the candidates, the higher we would expect the mean absolute margin to be. If you’re not convinced, look at this graph of simulations with varied shares for the candidates:

So, assuming the presidential race in Wisconsin isn’t exactly tied, the poll margins would look even more suspiciously close to zero than they already do!

With all that in mind, it seems hard to deny that there could be some systemic bias distorting these polls away from being true random samples of their populations–possibly herding driven by an aversion to publishing too strong of a poll for one or more of the candidates.

Some other states’ poll margins

Nate Silver noted that he observed herding in Pennsylvania as well, and our simulations reveal as much as well:

(the unnatural bias towards a tie looks even worse for Pennsylvania than it did for Wisconsin). However, the polls in New Hampshire apparently don’t suffer from herding:

You can see that not only is the mean absolute margin for New Hampshire not well below (to the left) the simulation’s distribution, it is actually far above (to the right of) it. This makes sense; New Hampshire appears to be nowhere near tied, with nearly all polls giving a strong Harris lead. Note that our simulations actually wouldn’t be able to detect herding if its not occurring around a near-tie polling average; therefore all we can say is that Nate Silver could be right to acquit New Hampshire pollsters of the herding accusation.

How is this happening?

To be clear, no individual poll–even one with a very close margin–is by itself indicative of foul play by the pollster who created it. Rather, the aggregation of poll results for each of multiple swing states indicate systemic bias. I know almost nothing about political polls, but I recently read a great book about systemic problems in modern science called Science Fictions and it seems like there’s a lot of ways to manipulate your data–even without intending to or realizing that you are doing it.

It’s totally plausible to me that pollsters are just focusing a lot more scrutiny on any result that shows a strong swing toward one candidate or another; maybe they’re more likely to throw out outliers or keep collecting more data if they start to see “too” wide of a margin. These hypotheses may sound very foolish to people more familiar with how polls are typically conducted; I’ll stop speculating before I make too much of a fool of myself, but suffice it to say there are a lot of ways for data to get distorted in any field of science and I would expect no less of political polling.

Why does this Matter?

After election results (either from exit polls or counting the votes) are announced, it may turn out that one or more of the swing states goes to either Trump or Harris by a very wide margin. Some people might look back at these polls and conclude that they constitute evidence of interference, cheating, voter suppression, fraud, or the like. After all, the pollsters nearly all agreed that these swing states were right on the margin. However, the high level of agreement between pollsters is not evidence that we know what the results will be, but rather that we can’t trust these polls, and therefore should be very uncertain about the outcome of this race.

Footnotes:

The following walkthrough assumes that you have a running Minikube cluster and have installed kubectl.

Un-installing Snap Docker

First, if you are on Ubuntu, you need to make sure that you are not running the Snap version of Docker; the Docker client on your host machine will need to authenticate to the Docker daemon on the Minikube host using a cert file that is inaccessible to the Snap version of Docker, due to that Snap’s containment policy. So make sure you have installed Docker Desktop from the downloadable .deb file, or add Docker’s package repository and install Docker CE using Apt.

Connecting to the Minikube’s Docker daemon

In order to connect to the Docker daemon inside the Minikube VM, we will need to change the values of certain environment variables. Luckily, Minikube makes it easy for us to get these values with the following command:

minikube docker-env

(you will need to instead run minikube -p <PROFILE-NAME> docker-env if you want to connect to a Minikube profile other than the currently activated one)

this should return an output similar to the following:

export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.58.2:2376"
export DOCKER_CERT_PATH="/home/your-username/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="profile-name"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p profile-name docker-env)

Export these values to your current terminal’s environment by running the command described in the last line of the output, i.e.:

eval $(minikube -p profile-name docker-env)

(note: profile-name will likely be a different value when run on your machine, you should copy & run the output of your minikube docker-env command, not the one on this webpage)

You can verify that your Docker client has successfully connected to the Minikube VM Docker daemon by running:

$ docker images
REPOSITORY                                         TAG                                        IMAGE ID       CREATED         SIZE
registry.k8s.io/kube-apiserver                     v1.30.0                                    c42f13656d0b   4 months ago    117MB
registry.k8s.io/kube-controller-manager            v1.30.0                                    c7aad43836fa   4 months ago    111MB
registry.k8s.io/kube-scheduler                     v1.30.0                                    259c8277fcbb   4 months ago    62MB
registry.k8s.io/kube-proxy                         v1.30.0                                    a0bf559e280c   4 months ago    84.7MB
...

your output should similarly contain several images from the Kubernetes official registry.

NOTE: the above will have to be re-run every time you open a new terminal, open a new SSH session, restart the computer, etc.

Running the Image on Minikube

To test that we can actually build an image to the Minikube VM’s Docker daemon, let’s start by making a directory named test-docker, then make a Dockerfile in it with the following contents:

FROM python
CMD python -c "print('Hello, world. This is Python, inside a Docker container, possibly on a Kubernetes cluster')"

In a terminal that has connected to the Minikube VM’s Docker daemon (by following the instructions above), cd into the parent directory of test-docker, then run:

docker build --tag my-python test-docker/

Now check that the image is available by running:

$ docker images
REPOSITORY                                         TAG                                        IMAGE ID       CREATED         SIZE
my-python                                          latest                                     17f99b663100   10 days ago     1.02GB
registry.k8s.io/kube-apiserver                     v1.30.0                                    c42f13656d0b   4 months ago    117MB
registry.k8s.io/kube-scheduler                     v1.30.0                                    259c8277fcbb   4 months ago    62MB
registry.k8s.io/kube-controller-manager            v1.30.0                                    c7aad43836fa   4 months ago    111MB
...

Now run a pod using this image with the following command:

$ kubectl run --image my-python --image-pull-policy Never my-python-pod
pod/my-python-pod created

(the --image-pull-policy Never is necessary because Kubernetes looks for images in a default registry, without even considering images in its own Docker daemon)

Check that the pod has run:

$ kubectl get pods
NAME                                             READY   STATUS                   RESTARTS         AGE
...
my-python-pod                                    0/1     Completed                2 (13s ago)      15s

(the pod’s STATUS may eventually change to CrashLoopBackOff; I think this is because Kubernetes does not expect pods to execute one command and then terminate)

You can see that the pod has completed the command described in the Dockerfile’s CMD directive by running:

$ kubectl logs my-python-pod
Hello, world. This is Python, inside a Docker container, possibly on a Kubernetes cluster

And there you have it! A Docker image built and run on your local Minikube cluster.

>>> sklearn.__version__
'1.3.0'
>>> sklearn.metrics.f1_score(y_true=list(range(104)), y_pred=list(range(100)) + [101, 102, 103, 104], average='macro', zero_division=1.0)
0.9809523809523809  # incorrect

compare to (the exact same expression in an earlier version of Scikit-Learn):

>>> sklearn.__version__
'1.2.2'
>>> sklearn.metrics.f1_score(y_true=list(range(104)), y_pred=list(range(100)) + [101, 102, 103, 104], average='macro', zero_division=1.0)
0.9523809523809523  # correct

Similar cases for zero_division=np.nan (which was introduced in 1.3.0, so I can’t directly compare to the output in 1.2.2):

>>> sklearn.metrics.f1_score([0, 1], [1, 0], average='macro', zero_division=np.nan)
nan  # should be 0.0
>>> sklearn.metrics.f1_score([0, 1, 2], [1, 0, 2], average='macro', zero_division=np.nan)
1.0  # should be ~0.67

Both myself and the Scikit-Learn maintainers consider the behavior in 1.3.X to be incorrect. While a pull request to fix this behavior was just merged, the fix has not yet shipped on any released version of Scikit-Learn. Therefore, the easiest solution to this specific problem is to revert to Scikit-Learn 1.2.2, or use zero_division=0.0 if possible, while being careful to understand how this parameter change will affect precision, recall, & F-1 (see below for an explainer on the purpose and function of the zero_division parameter).

(EDIT 2024-01-24: Scikit-Learn 1.4.0 has been released as of a week ago and contains a fix for this bug. Go and update now!)

The problem is that F-1 for an individual class is getting calculated as 1.0 or np.nan when precision & recall are both 0.0 (which is not the desired behavior for the zero_division parameter).

How did this happen?

Let’s take a look at some formulae for classification metrics:

There are three different places here where a division by zero can occur:

• in precision, if true positive + false positive = 0 (the classifier made no positive predictions for the class)
• in recall, if true positive + false negative = 0 (there are no truly positive examples of the class in the dataset)
• in F-1, if precision = recall = 0 (the classifier has made a nonzero number of exclusively incorrect predictions)

Two of these are interesting cases where reasonable people could disagree on what the correct behavior should be:

• When the classifier has made zero positive predictions for the class, should that count as a precision of 1.0? If “perfect precision” is interpreted as “no false positives”, then this is totally reasonable behavior.
• When the gold dataset has zero true positive examples of the class, should that count as a recall of 1.0? This is a much more unusual scenario than the “zero positive predictions” example–a good evaluation dataset should almost never be entirely missing a class. However, this can realistically occur when evaluating on subsets of a large multiclass dataset. Again, if the definition of “perfect recall” is taken as “no false negatives”, then assigning a recall of 1.0 in this case is totally reasonable behavior.

For F-1, however, the “division by zero” case is not interesting or controversial in any way. If a classifier has achieved a recall of 0.0 (all negative predictions are false) and a precision of 0.0 (all positive predictions are false), I don’t think any reasonable person would disagree what the F-1 score should be: 0.0. Indeed, this is exactly how Scikit-Learn calculated F-1 right up to (and including) version 1.2.2, regardless of the value of the zero_division parameter.

However, in Scikit-Learn 1.3.0, the zero_division parameter was turned into a kind of monkey’s paw that defines the behavior of any division-by-zero that happens to occur during the calculation of an F-1 score, leading to the bizarre scenario where a 100% wrong classifier can get an F-1 score of 100%:²

>>> sklearn.__version__
'1.3.0'
>>> print(sklearn.metrics.classification_report(y_true=[0, 1, 2, 3, 4], y_pred=[1, 2, 3, 4, 0], zero_division=1.0))
              precision    recall  f1-score   support

           0       0.00      0.00      1.00       1.0
           1       0.00      0.00      1.00       1.0
           2       0.00      0.00      1.00       1.0
           3       0.00      0.00      1.00       1.0
           4       0.00      0.00      1.00       1.0

    accuracy                           1.00       5.0
   macro avg       0.00      0.00      1.00       5.0
weighted avg       0.00      0.00      1.00       5.0

Why? Because precision and recall are both 0, which means the denominator of the F-1 formula is 0, and zero_division=1.0 now (as of Scikit-Learn 1.3.0) applies to the F-1 calculation itself, so that means F-1 is calculated (incorrectly) as 1.0!

Why does this matter?

I don’t know if there are rigorous statistics on this, but I’d wager that macro average F-1 is the most commonly used metric for multiclass classification by a wide margin. Scikit-Learn’s f1_score() function is in turn very likely the most commonly used implementation of F-1. Try asking Google or ChatGPT how to calculate F-1; the first results will very likely tell you to use this exact function in Scikit-Learn.

The kinds of tasks F-1 could be used for range from low-risk, like sentiment analysis on customer reviews, to some conceivably really safety-critical things. Imagine a researcher at an autonomous car company thinks their computer vision system is performing really well at recognizing all categories of objects & entities on the road. But actually, their classifier is completely missing every single example of a few classes!

Ideally, any machine learning practitioner probably should notice this bug well before a classifier is put into production or reporting results in a submitted journal paper. On the other hand, you really would not expect the definition of F-1 to change from one version of Scikit-Learn to the next! While just about any programmer should be able to implement an F-1 calculator in very little time, most of us prefer to just import Scikit-Learn’s specifically to avoid gotcha edge cases like this one.

What should I do now?

If your project:

• is using, has used, or may have used any Scikit-Learn version starting with 1.3.0 (released 2023-06-30)
• contains any call to classification_report(), f1_score(), or fbeta_score(), and
• that call contains the parameter zero_division=1.0 or zero_division=np.nan

it may have been affected by this bug. To determine if any particular F-1 score calculation was impacted by this bug, first change that F-1 score calculation to a classification_report() if possible. If any class in that classification report contains a precision of 0.0, a recall of 0.0, and an f1-score of 1.0 or nan, then the F-1 score for this classifier has been calculated incorrectly.

Any call using zero_division=1.0 can be fixed by reverting to Scikit-Learn version 1.2.2. Unfortunately, the parameter zero_division=np.nan did not exist in Scikit-Learn 1.2.2, and I don’t believe there is any easy way to replicate it.

(EDIT 2024-01-24: Scikit-Learn 1.4.0 has been released, and you should update to it ASAP!)

Footnotes: