Privia Security was chosen as one of Türkiye's fastest growing companies!

Read the News Read the News
17 August 2021

What Are the VPN Protocols?

What Are the VPN Protocols?

The most important criterion of a VPN tunnel is encryption, and many methods and protocols are used to meet encryption requirements. There are several specific network protocols for VPNs. The two most frequently used protocols for this purpose are Point-to-Point Tunneling Protocol (PPTP) and Layer 2 Tunneling Protocol (L2TP).

The part of the connection where the data is encapsulated is called the tunnel. Among these protocols, L2TP is generally combined with IPSec to provide a high level of security. 

We would also like to remind you that VPN security is of critical importance in the cyber world. The connection between two systems is encrypted using a VPN and a secure connection is established. However, while establishing this connection, one of the most important points is to select the correct protocols and provide an efficient working environment.

PPTP – Point-to-Point Tunneling Protocol

PPTP is a tunneling protocol that enables the encapsulation of PPP (Point-to-Point Protocol) packets, an older connection protocol, within Internet Protocol (IP) packets and their transmission over any IP network, including the Internet. PPTP is generally used to create VPNs. PPTP is an older protocol than L2TP and IPSec. Some experts consider PPTP to be less secure than L2TP or IPSec. However, it consumes fewer resources and is supported by almost every VPN application. Essentially, it is a secure extension of PPP.

PPTP was originally proposed as a standard in 1996 by the PPTP Forum, a group of companies including Ascend Communications, ECI Telematics, Microsoft, 3Com and US Robotics. The aim of this group was to design a protocol that would enable remote users to communicate securely over the Internet.

Although newer VPN protocols are available, PPTP continues to be widely used as almost all VPN vendors support PPTP. Another important benefit of PPTP is that it operates at layer 2 (data link layer) of the OSI model and allows different network protocols to run over a PPTP tunnel. 

When connecting users to a remote system, encrypting data transmissions is not the only aspect of security. You also need to authenticate the user. PPTP supports two separate technologies to accomplish this. These technologies are the Extensible Authentication Protocol (EAP) and the Challenge Handshake Authentication Protocol (CHAP).

Extensible Authentication Protocol (EAP)

EAP is specifically designed with PPTP and is designed to operate as part of PPP. EAP works from PPP’s authentication protocol. It provides a framework for several different authentication methods. EAP is intended to support proprietary authentication systems and includes various authentication methods including passwords, challenge-response tokens and public-key infrastructure certificates.

Challenge Handshake Authentication Protocol (CHAP)

CHAP is essentially a three-part handshake (a term used to denote TCP authentication operations) procedure. After the connection is established, the server sends a challenge message to the client machine that is establishing the connection. The originator responds by returning a value calculated using a one-way hash function. The server checks the response against its own calculation of the expected hash value. If the values match, authentication is acknowledged. Otherwise, the connection is terminated. This means there are three stages to authorising the client connection.

What makes CHAP interesting is that it repeats the process periodically. This means that even after a client connection has been authenticated, CHAP continually attempts to re-authenticate the client, providing a strong level of security.

L2TP

Layer 2 Tunneling Protocol is an extension and enhancement of the Point-to-Point Tunneling Protocol, which is generally used to run virtual private networks over the Internet. Essentially, it is a new and improved version of PPTP. As the name suggests, it operates at the data link layer of the OSI model (like PPTP). Both PPTP and L2TP are considered by many experts to be less secure than IPSec. However, you can also frequently see IPSec used together with L2TP to create a secure VPN connection.

Like PPTP, L2TP supports EAP and CHAP. However, it also offers support for a total of six other authentication methods…

MS-CHAP

As the name suggests, MS-CHAP is a Microsoft-specific authentication method for CHAP. Microsoft developed MS-CHAP to authenticate remote Windows workstations. The aim is to provide remote users with the functionality available on the LAN while integrating the encryption and hashing algorithms used on Windows networks.

Where possible, MS-CHAP is consistent with standard CHAP. However, some basic differences between MS-CHAP and standard CHAP are as follows.

PAP

The Password Authentication Protocol (PAP) is the most basic form of authentication. With PAP, a user’s name and password are transmitted over a network and compared to a table of name-password pairs. The transmission of the passwords as cleartext, unencrypted, is a weakness for PAP. The basic authentication feature built into the HTTP protocol uses PAP. This method is no longer used and is mentioned only for historical purposes and information.

SPAP

The Shiva Password Authentication Protocol (SPAP) is a proprietary version of PAP. Most experts view SPAP as slightly more secure than PAP because, when the username and password are sent, unlike PAP, encryption is applied during transmission. 

Because SPAP encrypts passwords, someone who captures authentication packets cannot read the SPAP password. However, SPAP is still sensitive to data theft attacks. Replay attacks are also possible because SPAP always uses the reversible encryption method for sending passwords over the wire.

Kerberos

Kerberos is one of the most recognised network authentication protocols. It was developed at MIT and takes its name from the legendary three-headed dog that guarded the gates of Hades. Kerberos works by sending messages back and forth between client and server. 

The actual password (or even a hash of the password) is never sent. This makes it impossible for someone to intercept it. Instead, the username is sent. The server then looks up the stored hash of this password and uses this data as an encryption key to encrypt and send back to the client. The client then takes the password entered by the user and uses it as a key to decrypt the data. If the user enters an incorrect password, it will never be decrypted. In this case, it is a clever way to verify the password without transmitting the data. Authentication occurs with UDP (User Datagram Protocol) on port number 88.

After the user’s username is sent to the authentication service (AS), the AS uses the stored password hash as a secret key to encrypt the following two messages sent to the client.

Remember, both of these messages are encrypted using the key generated by the AS.

Then the user attempts to decrypt message A with the secret key generated by the client, which hashes the password entered. If the entered password does not match the AS password in the database, the hashes will not match and decryption will not work. If it does work, message A contains the Client/TGS session key that can be used for communication with the TGS. Message B is encrypted with the TGS secret key and cannot be decrypted by the client.

The user is now authenticated. However, when the user actually requests a service, further message communication is required. When requesting a service, the client sends the following messages to the TGS:

After receiving messages C and D, the TGS retrieves message B from message C. It decrypts message B using the TGS secret key. This gives it the “Client/TGS session key”. Using this key, the TGS decrypts message D (Authenticator) and sends the following two messages to the client:

Once messages E and F are received from the TGS, the client has enough information to authenticate itself to the Service Server (SS). The client connects to the SS and sends the following two messages:

The SS decrypts the ticket (message E) using its own secret key to obtain the client/server session key. Using the session key, the SS decrypts the Authenticator and sends the following message to the client to confirm its identity and willingness to serve the client:

The client decrypts the confirmation (message H) using the client/server session key and checks whether the timestamp is correct. If so, the client can trust the server and start giving service requests to the server. The server provides the requested services to the client.

IPSec

Internet Protocol Security (IPSec) is a technology used to create virtual private networks. IPSec is used in addition to the IP protocol, which adds security and privacy to TCP/IP communication. IPSec is integrated with Microsoft operating systems and many other operating systems. 

For example, the security settings that come with Windows XP and later versions in the Internet Connection Firewall allow users to turn on IPSec for transmissions. IPSec is a set of protocols developed by the IETF (Internet Engineering Task Force; www.ietf.org) to support the secure exchange of packets. IPSec is widely deployed to implement VPNs.

IPSec has two encryption modes. These are called transport and tunnel. Transport mode works by encrypting the data in each packet but leaves the header unencrypted. This means that the source and destination addresses, as well as other header information, are not encrypted. Tunnel mode encrypts both the header and the data. 

This is more secure than transport mode but can run slower. At the receiving end, an IPSec-compliant device decrypts each packet. For IPSec to work, the sending and receiving devices must share a key, which is an indication that IPSec is a single-key encryption technology. IPSec also offers two more protocols beyond the two modes described earlier:

Either protocol can be used to protect an IP packet, or both protocols can be applied to the same IP packet.

IPSec can also work in two modes. These modes are transport mode and tunnel mode. Transport mode is the mode in which IPSec encrypts the data. However, it does not encrypt the packet header. Tunneling mode encrypts both the packet data and the header.

There are other protocols that make IPSec work. IKE, or Internet Key Exchange, is used to establish security associations in IPSec. A security association is established by the two endpoints of the VPN tunnel after deciding how to encrypt and authenticate. 

Internet Key Exchange (IKE and IKEv2) is used to negotiate protocols and algorithms to create an SA and to generate the encryption and authentication keys to be used.

The Internet Security Association and Key Management Protocol (ISAKMP) provides a framework for authentication and key exchange. After the IKE protocol sets up the SA, it is time to actually perform the authentication and key exchange.

The first exchange between VPN endpoints establishes the basic security policy. The initiator proposes the encryption and authentication algorithms it wishes to use. The responder selects the appropriate proposal and sends it to the initiator. The next exchange transmits Diffie-Hellman public keys and other data. 

These Diffie-Hellman public keys will be used to encrypt the data sent between the two endpoints. The third exchange authenticates the ISAKMP session. This process is called main mode. After the IKE SA is established, IPSec negotiation (Quick Mode) begins.

Quick Mode IPSec negotiation, or Quick Mode, is similar to Aggressive Mode IKE negotiation, except that negotiation must be protected within an IKE SA. Quick Mode negotiates the SA for data encryption and manages the key exchange for that IPSec SA. 

In other words, Quick Mode uses the Diffie-Hellman keys exchanged in main mode to continue exchanging the symmetric keys to be used for the actual encryption in the VPN.

Aggressive Mode compresses the IKE SA negotiation into three packets with all the data required for the SA passed by the initiator. The responder sends the proposal, key material and identity, and authenticates the session in the next packet. The initiator responds by authenticating the session. The negotiation is faster and the initiator and responder identity pass in the clear.

SSL / TLS

A new type of firewall, it uses SSL (Secure Sockets Layer) or TLS (Transport Layer Security) to provide VPN access through a web portal. Essentially, TLS and SSL are protocols used to secure websites. If you see a website starting with HTTPS, the traffic to and from that website is encrypted using SSL or TLS. 

In some VPN solutions, the user logs into a website secured with SSL or TLS, and is then granted access to the virtual private network. However, visiting a website that uses SSL or TLS does not mean you are on a VPN. As a general rule, most websites, such as banking websites, allow you to access only a very limited set of data, such as your account balance. A VPN allows you to access the network with the same or similar access as you would have if you were physically on that network.

Whether you are using SSL to connect to an e-commerce website or to establish a VPN, an SSL handshake process is required to establish a secure / encrypted connection.

  1. The client sends the server the client’s SSL version number, cipher settings, session-specific data and other information the server needs to communicate with the client using SSL.
  2. The server sends the client the server’s SSL version number, cipher settings, session-specific data and other information the client needs to communicate with the server over SSL. The server also sends its own certificate, and if the client is requesting a server resource that requires client authentication, the server requests the client’s certificate.
  3. The client uses the information sent by the server to authenticate the server; for example, in the case of a web browser connecting to a web server, the browser checks whether the subject name of the received certificate actually matches the name of the server being communicated with, whether the issuer of the certificate is a trusted certificate authority, whether the certificate has expired and ideally whether the certificate has been revoked… If the server cannot be authenticated, the user is warned of the issue and informed that an encrypted and authenticated connection cannot be established. If the server can be successfully authenticated, the client proceeds to the next step.
  4. Using all the data generated in the handshake so far, the client (with the cooperation of the server depending on the cipher used) creates the pre-master secret for the session, encrypts it with the server’s public key (which is sent through the server’s certificate (step 2)), and then sends the encrypted pre-master secret to the server.
  5. If the server requests client authentication (an optional step in the handshake), the client also signs another piece of data that is unique to this handshake and known by both the client and the server. In this case, the client sends both the signed data and the client’s own certificate along with the encrypted pre-master secret to the server.
  6. If the server requested client authentication, the server attempts to authenticate the client. If the client cannot be authenticated, the session ends. If the client can be successfully authenticated, the server uses its private key to decrypt the pre-master secret and then performs a series of steps (which it also performs starting from the same pre-master secret) to generate the master secret.
  7. Both the client and the server use the master secret to generate session keys, which are symmetric keys used to encrypt and decrypt information exchanged during the SSL session and to verify its integrity (i.e. to detect changes in the data over time).
  8. The client sends a message to the server informing it that future messages from the client will be encrypted with the session key. It then sends a separate (encrypted) message indicating that the client part of the handshake is complete.
  9. The server sends a message to the client informing it that future messages from the server will be encrypted with the session key. It then sends a separate (encrypted) message indicating that the server part of the handshake is complete.

For the penetration testing and security hardening services you need, you can contact our experts to obtain a quote.

You May Be Interested In These