How does this compare with GumTree? It’s weird that the page doesn’t even mention existing state-of-the-art tools for this task.

edit: I’ve compared GumTree and difftastic myself while working on a project this past week. Difftastic is harder to use programatically (the JSON format is unstable and leaves something to be desired) but other than that it’s miles and miles better.

Only the number of shuffles is linear. Shuffling an array and marking/deleting correctly-placed elements still take linear time even with a “placement oracle.” It’s at best O(n^2) so the algorithm still wouldn’t be a good sorting algorithm.

It’s like doing selection sort, except we’re selecting a random set of elements (from that poisson distribution) instead of the smallest one.

I’m not sure the median is what you want. The worst case behavior is unbounded. There is no guarantee that such an algorithm ever actually terminates, and in fact (with very low probability) it may not! But that means there is no well-defined median; we can’t enumerate the space.

So let’s instead ask about the average, which is meaningful, as the increasingly high iteration-count datapoints are also decreasingly likely, in a way that we can compute without trying to enumerate all possible sequences of shuffles.

Consider the problem like this: at every iteration, remove the elements that are in the correct positions and continue sorting a shorter list. As long as we keep getting shuffles where nothing is in the correct position, we can go forever. Such shuffles are called derangements, and the probability of getting one is 1/e. That is, the number of derangements of n items is the nearest integer to n!/e, so the probability of a derangement would be 1/n! * [n!/e]. This number converges to 1/e incredibly quickly as n grows - unsurprisingly, the number of correct digits is on the order of the factorial of n.

We’re now interested in partial derangements D_{n,k}; the number of permutations of n elements which have k fixed points. D_{n,0} is the number of derangements, as established that is [n!/e]. Suppose k isn’t 0. Then we can pick k points to be correctly sorted, and multiply by the number of derangements of the others, for a total of nCk * [(n-k)!/e]. Note that [1/e] is 0, indeed, it’s not possible for exactly one element to be out of place.

So what’s the probability of a particular partial derangement? Well now we’re asking for D_{n,k}/n!. That would be nCk/n! * [(n-k)!/e]. Let’s drop the nearest integer bit and call it an approximation, then (nCk * (n-k)!)/(n! * e) = 1/(k!*e). Look familiar? That’s a Poisson distribution with λ = 1!

But if we have a Poisson distribution with λ = 1, then that means that on average we expect one new sorted element per shuffle, and hence we expect to take n shuffles. I’ll admit, I was not expecting that when I started working this out. I wrote a quick program to average some trials as a sanity check and it seems to hold.