With a bit of reverse engineering, I think I was able to see how Hal was able to get to these results.

First, it's a pretty well known result of Fermat's little theorem that if p is a prime number and g is a generator for the field Z/pZ, then:

g ^ (p - 1) = 1

Note, don't confuse this abstract generator g with the generator for the secp256k1 group G. Now, given the above equation, it's not a big leap to see that:

(g ^ ((p - 1)/3)^3 = g ^ (p - 1) = 1

Thus, we can find λ and β by first finding generators for Z/NZ and Z/PZ (N and P being the parameters given in the original question), and then raising them to the (N-1)/3 and (P-1)/3 powers, respectively. You can check that both N-1 and P-1 are divisible by 3.

The generator that it seems Hal used for λ is 3, and for β is 2. I'm not sure why he picked those, there are plenty of other good generators to choose from. It was probably on a trial and error basis.

Using sage mathematics notebook, I was able to produce the same values for λ and β.

Quoting cryptography.stackexchange.com:

Given that N and P are prime, one obvious way to do this is to select a random value g from [1,N−1], and compute g^((N−1)/3) mod N; assuming that N≡1(mod 3), this resulting value will either be 1, the displayed value of λ, or N−λ−1 (with equal probabilities of each). If N≢1(mod 3), then the only modular cube root of 1 will be 1.

And, to compute β, you do the same with P.

The reason this works is due to Fermat's little theorem which states:

g^(N-1) ≡ 1 (mod N)

which implies

(g^((N-1)/3))^3 ≡ 1 (mod N)

which implies

g^((N-1)/3) is our potential λ. If it's not 1, it'll work for the purposes of endomorphism.