It took me several days to figure out how to connect to a checkpoint vpn server with certificate-authorization. That's why i want to put some quick info here.
First: you need a kernel with the ipsec-stuff (not everything shown here is needed for checkpoint-vpns):
Networking support (NET) [Y/n/?] y
*
* Networking options
*
PF_KEY sockets (NET_KEY) [Y/n/m/?] y
IP: AH transformation (INET_AH) [Y/n/m/?] y
IP: ESP transformation (INET_ESP) [Y/n/m/?] y
IP: IPsec user configuration interface (XFRM_USER) [Y/n/m/?] y
Cryptographic API (CRYPTO) [Y/n/?] y
HMAC support (CRYPTO_HMAC) [Y/n/?] y
Null algorithms (CRYPTO_NULL) [Y/n/m/?] y
MD5 digest algorithm (CRYPTO_MD5) [Y/n/m/?] y
SHA1 digest algorithm (CRYPTO_SHA1) [Y/n/m/?] y
DES and Triple DES EDE cipher algorithms (CRYPTO_DES) [Y/n/m/?] y
AES cipher algorithms (CRYPTO_AES) [Y/n/m/?] y
(see: http://www.spenneberg.com/ipsec-2.6.pdf)
Second you need to install the ipsec-tools package. For Gentoo Linux this would be: emerge ipsec-tools.
After that you need to setup the /etc/ipsec.conf and /etc/racoon/racoon.conf files. Click on the filenames for examples. As a last step you need to convert the provided p12 (pkcs12) file to pem (cert) files. This can be done with openssl (use this little script). After executing this script you get 2 .pem files. One of them named "filename.split.me.into.2.cert.files.pem". You need to edit this file and split it into two files. It contains two certificates. The vpn-ca (certification authority of the vpn-server) and the client-cert. Normally the first one is the ca-cert (look at the subject-line - the ca-cert has your vpn-domain-name as subject) and the second the client-cert (normally the subject contains the user-dn). Copy these two certs together with the key to your /etc/racoon/certs directory. Now edit your ipsec.conf and racoon.conf files.
Finally: Prey... ;)
How to get info about errors (like: "racoon: ERROR: none message must be encrypted"):
Check if my_identifier, encryption_algorithm, hash_algorithm, dh_group, authentication_algorithm, pfs_group and the sainfo lines are correct (examples above work for me and the checpoint vpn). Best Way is to use wireshark (formely ethereal) to check for "ISAKMP: Informational" Messages.
Then click on "Internet Security Association and Management Protocol", "Notification Payload" and look at the "Message Type" line. This is the error. Best way is to google for it ;) Errors i encountered: INVALID-CERTIFICATE: export the cacert, the cert and the key from the pkcs12 file and add them to the config, INVALID-ID-INFORMATION: my_identifier is incorrect (perhaps does not match certificate?)
Usefull links:
http://felipe-alfaro.org/blog/category/ipsec/
http://www.spenneberg.com/
http://www.fw-1.de/aerasec/ng/vpn-racoon/CP-VPN1-NG-Linux-racoon.html
