BEGIN RSA PRIVATE KEYindicates "PKCS#1" or "PEM" key format, which is Base64 encoding of an ASN.1 DER serialized structure. It's a basic ASN.1 sequence containing the RSA parameters (n, e, d, p, q, etc).
OpenSSH actually has traditionally used this format as well – because it already uses OpenSSL's cryptographic code, so the "load key" and "write key" functions were also conveniently available.
This means you can use
ssh-keygen -m PEMto generate or convert such keys.
BEGIN PRIVATE KEYindicates "PKCS#8" key format (unencrypted); the contents are very similar to the above format, with the same RSA parameters nested inside another structure that indicates that it is indeed an RSA key.
Compared to PEM, the PKCS#8 format more cleanly separates the "payload" (key algorithm, encryption) from the outer Base64 wrapper. It doesn't change the meaning of the data though.
OpenSSH will recognize this format too (due to its OpenSSL usage for key loading).
BEGIN OPENSSH PRIVATE KEYis a format invented by OpenSSH for OpenSSH. This time it uses the SSHv2 packet serialization instead of DER.
OpenSSH now uses this format so that it could avoid depending on OpenSSL entirely, and/or so that it could add new key algorithms without waiting for PKIX to standardize an ASN.1 serialization format (which otherwise has absolutely nothing to do with SSH) and then waiting for OpenSSL to implement it.
PuTTY has its own .ppk format. You can use /usr/bin/puttygen to convert between all of them.
See also: OpenSSH public key file format?