Btukfyl

I would appreciate some help finding $GCD(a^3-3ab^2, b^3-3ba^2)$; $a,b in mathbb{Z}$. So far I've got here: if $GCD(a,b)=d$ then $exists alpha, beta$ so that $GCD(alpha, beta)=1$ and $alpha d=a$, $beta d=b$.

Therefore we know that $GCD(a^3-3ab^2, b^3-3ba^2)=d^3 GCD(alpha^3-3alpha beta^2, beta^3-3beta alpha^2)$. However I don't know to figure out $GCD(alpha^3-3alpha beta^2, beta^3-3beta alpha^2)$ given that $GCD(alpha,beta)=1$.

As you've already shown, factoring out the cube of the GCD of $a$ and $b$ gives a new equation

$$e = gcdleft(alpha^3-3alpha beta^2, beta^3-3beta alpha^2 right) tag{1}label{eq1}$$

where

$$gcdleft(alpha,beta right) = 1 tag{2}label{eq2}$$

Update: Here is a simpler solution than what I originally wrote. First, note that no factor of $e$ may divide $alpha$ or $beta$. If any do, let's say $alpha$, then it must divide $beta^3 - 3betaalpha^2$ and, thus, must divide $beta^3$, which is not possible due to eqref{eq2}. Thus, from the first term of eqref{eq1}, since $alpha^3-3alpha beta^2 = alphaleft(alpha^2 - 3beta^2right)$, this means that $e mid alpha^2 - 3beta^2$. Similarly, for the second term, $beta^3-3beta alpha^2 = betaleft(beta^2 - 3alpha^2right)$ gives that $e mid beta^2 - 3alpha^2$. Also, $e$ must divide any linear combination of these values, including $e | alpha^2 - 3beta^2 + 3left(beta^2 - 3alpha^2right) = -8alpha^2$. Thus, $e$ can only be a power of $2$. To finish this off, go to the second last paragraph. Otherwise, for the rest of the original, longer solution, continue reading.

$ $

Next, note that if $f = gcd(g,h)$, then $f$ divides $g$ and $h$ and, thus, will also divide any linear combination of $g$ and $h$, including their sum & difference. From eqref{eq1}, first check the sum of the $2$ inside values:

begin{align}
alpha^3-3alpha beta^2 + beta^3-3beta alpha^2 & = alpha^3 + beta^3 -3left(alpha betaright)beta - 3left(alpha betaright)alpha \
& = left(alpha + betaright)left(alpha^2 - alphabeta + beta^2right) - 3alphabetaleft(alpha + betaright) \
& = left(alpha + betaright)left(alpha^2 - 4alphabeta + beta^2right) tag{3}label{eq3}
end{align}

Suppose there's a factor $m gt 1$ which divides $e$ and $alpha + beta$. Then, $alpha equiv -beta pmod m$, so $alpha^3-3alpha beta^2 equiv 2beta^3 pmod m$. From eqref{eq2}, this means that $m = 2$, and that any other factors of $e$ must divide $alpha^2 - 4alphabeta + beta^2$.

From eqref{eq1}, next check the difference of the $2$ inside values:

begin{align}
alpha^3-3alpha beta^2 - beta^3 + 3beta alpha^2 & = alpha^3 - beta^3 -3left(alpha betaright)beta + 3left(alpha betaright)alpha \
& = left(alpha - betaright)left(alpha^2 + alphabeta + beta^2right) + 3alphabetaleft(alpha - betaright) \
& = left(alpha - betaright)left(alpha^2 + 4alphabeta + beta^2right) tag{4}label{eq4}
end{align}

Suppose there's a factor $n gt 1$ which divides $e$ and $alpha - beta$. Then, $alpha equiv beta pmod n$, so $alpha^3-3alpha beta^2 equiv -2beta^3 pmod m$. From eqref{eq2}, this means that $n = 2$, and that any other factors of $e$ must divide $alpha^2 + 4alphabeta + beta^2$.

This shows that any factor, other than $2$, which divides $e$ must divide both $alpha^2 - 4alphabeta + beta^2$ and $alpha^2 + 4alphabeta + beta^2$. Thus, it must also divide their difference, which is $8alphabeta$. This can only be true for $2$, $4$ or $8$.

At this shows overall, only powers of $2$ may possibly divide $e$. Since $e$ is relatively prime to $alpha$ & $beta$, this means they must both be odd. From $alpha^3 - 3alphabeta^2 = alphaleft(alpha^2 - 3beta^2right)$, note that $alpha^2 equiv beta^2 equiv 1 pmod 4$, so $alpha^2 - 3beta^2 equiv -2 pmod 4$. In other words, there will only be $1$ factor of $2$.

In summary, with your original equation of $d = gcdleft(a,bright)$, we get that $gcdleft(a^3-3ab^2, b^3-3ba^2right)$ is $2d^3$ if both $frac{a}{d}$ and $frac{b}{d}$ are odd, else it's $d^3$.

Note that $alpha^3 - 3 alpha beta^2$ and $3betaalpha^2 - beta^3$ are the real and imaginary parts respectively of $(alpha + i beta)^3$; this suggests that it will be useful to work in the ring of Gaussian integers $mathbb{Z}[i]$.

Now, suppose that for some integer prime $p$, $p mid alpha^3 - 3alpha beta^2$ and $pmid 3betaalpha^2 - beta^3$; then $p mid (alpha + i beta)^3$. If $p equiv 3 pmod{4}$, then $p$ remains irreducible in $mathbb{Z}[i]$, so $p mid alpha + i beta$, implying that $p mid gcd_{mathbb{Z}}(alpha, beta)$ which gives a contradiction. Similarly, if $p equiv 1 pmod{4}$, then the factorization of $p$ into irreducibles is of the form $p = (c + di) (c - di)$ for some $c, d in mathbb{Z}$. Thus, $c + di mid (alpha + i beta)^3$ implies $c + di mid alpha + ibeta$ and $c - di mid (alpha + ibeta)^3$ implies $c - di mid alpha + i beta$. Since $c + di$ and $c - di$ are relatively prime in $mathbb{Z}[i]$ (being irreducibles which do not differ by multiplication by a unit), this implies $(c + di) (c - di) mid alpha + i beta$; in other words, $p mid alpha + i beta$, again giving a contradiction.

The only remaining possibility is $p = 2$ which factors as $-i (1+i)^2$ in $mathbb{Z}[i]$. Now, by a similar argument to the above we have $(1 + i)^2 nmid alpha + beta i$, so the order of $1+i$ in $(alpha + beta i)^3$ is either 0 or 3. In the former case we get that $gcd(alpha^3 - 3alphabeta^2, 3betaalpha^2 - beta^3) = 1$, and in the latter case we get that $gcd(alpha^3 - 3alphabeta^2, 3betaalpha^2 - beta^3) = 2$.

Note that this solution is very closely tied to the exact form of the polynomials under consideration; whereas the general idea of John Omielan's answer should be more generally applicable to other cases.