Btukfyl

I tried to write • as the binary operation for a group (G, •), but cdot seems too small and bullet seems too large, so what symbol is normally used?

Normally, it's just cdot, it's what people are used to and it doesn't look small in $(G,{cdot})$ to me.

Anyway, if you want a bigger dot, you can load the package usepackage{bm} after any other font package, and then define newcommandbigcdot{bm{cdot}} and use this.

This is a matter of topic, audience, and personal taste. For a group operation consider an asymmetric binary symbol like oslash, though perhaps not arrow-like symbols like triangleright which suggests a mapping. But if oslash is commonly associated with another operation in your field, then another asymmetric symbol could be used. For an abelian group operation consider a more symmetric symbol like cdot, ast, bullet, etc..

There is no right way unless you are working under house rules, but it can be helpful to have symbols that graphically suggest properties (e.g. asymmetry) or avoid suggesting properties not assumed (e.g. commutativity).

Consider including

newcommandgop{mathbin{oslash}}

Then you can use agop b and modify globally ad lib.