SSL handshake

A HTTP-based SSL connection is always initiated by the client using a URL starting with https:// instead of with http://. At the beginning of an SSL session, an SSL handshake is performed. This handshake produces the cryptographic parameters of the session. A simplified overview of how the SSL handshake is processed is shown in the diagram below.

1. The client sends a client "hello" message that lists the cryptographic capabilities of the client (sorted in client preference order), such as the version of SSL, the cipher suites supported by the client, and the data compression methods supported by the client. The message also contains a 28-byte random number.
2. The server responds with a server "hello" message that contains the cryptographic method (cipher suite) and the data compression method selected by the server, the session ID, and another random number.
   Note:
   The client and the server must support at least one common cipher suite, or else the handshake fails. The server generally chooses the strongest common cipher suite.
3. The server sends its digital certificate. (In this example, the server uses X.509 V3 digital certificates with SSL.)
   If the server uses SSL V3, and if the server application (for example, the Web server) requires a digital certificate for client authentication, the server sends a "digital certificate request" message. In the "digital certificate request" message, the server sends a list of the types of digital certificates supported and the distinguished names of acceptable certificate authorities.
4. The server sends a server "hello done" message and waits for a client response.
5. Upon receipt of the server "hello done" message, the client (the Web browser) verifies the validity of the server's digital certificate and checks that the server's "hello" parameters are acceptable.
   If the server requested a client digital certificate, the client sends a digital certificate, or if no suitable digital certificate is available, the client sends a "no digital certificate" alert. This alert is only a warning, but the server application can fail the session if client authentication is mandatory.
6. The client sends a "client key exchange" message. This message contains the pre-master secret, a 46-byte random number used in the generation of the symmetric encryption keys and the message authentication code (MAC) keys, encrypted with the public key of the server.
   If the client sent a digital certificate to the server, the client sends a "digital certificate verify" message signed with the client's private key. By verifying the signature of this message, the server can explicitly verify the ownership of the client digital certificate.
   Note:
   An additional process to verify the server digital certificate is not necessary. If the server does not have the private key that belongs to the digital certificate, it cannot decrypt the pre-master secret and create the correct keys for the symmetric encryption algorithm, and the handshake fails.
7. The client uses a series of cryptographic operations to convert the pre-master secret into a master secret, from which all key material required for encryption and message authentication is derived. Then the client sends a "change cipher spec" message to make the server switch to the newly negotiated cipher suite. The next message sent by the client (the "finished" message) is the first message encrypted with this cipher method and keys.
8. The server responds with a "change cipher spec" and a "finished" message of its own.
9. The SSL handshake ends, and encrypted application data can be sent.

A TCP Tutorial

TCP SUMMARY

TCP provides a connection oriented, reliable, byte stream service. The term connection-oriented means the two applications using TCP must establish a TCP connection with each other before they can exchange data. It is a full duplex protocol, meaning that each TCP connection supports a pair of byte streams, one flowing in each direction. TCP includes a flow-control mechanism for each of these byte streams that allows the receiver to limit how much data the sender can transmit. TCP also implements a congestion-control mechanism.

 

Two processes communicating via TCP sockets. Each side of a TCP connection has a socket which can be identified by the pair < IP_address, port_number >. Two processes communicating over TCP form a logical connection that is uniquely identifiable by the two sockets involved, that is by the combination < local_IP_address, local_port, remote_IP_address, remote_port>.

 

TCP provides the following facilities to: 

Stream Data Transfer

From the application's viewpoint, TCP transfers a contiguous stream of bytes. TCP does this by grouping the bytes in TCP segments, which are passed to IP for transmission to the destination. TCP itself decides how to segment the data and it may forward the data at its own convenience.

 

Reliability TCP assigns a sequence number to each byte transmitted, and expects a positive acknowledgment (ACK) from the receiving TCP. If the ACK is not received within a timeout interval, the data is retransmitted. The receiving TCP uses the sequence numbers to rearrange the segments when they arrive out of order, and to eliminate duplicate segments. 

 

Flow Control The receiving TCP, when sending an ACK back to the sender, also indicates to the sender the number of bytes it can receive beyond the last received TCP segment, without causing overrun and overflow in its internal buffers. This is sent in the ACK in the form of the highest sequence number it can receive without problems. 

 

Multiplexing To allow for many processes within a single host to use TCP communication facilities simultaneously, the TCP provides a set of addresses or ports within each host. Concatenated with the network and host addresses from the internet communication layer, this forms a socket. A pair of sockets uniquely identifies each connection.

 

Logical Connections

The reliability and flow control mechanisms described above require that TCP initializes and maintains certain status information for each data stream. The combination of this status, including sockets, sequence numbers and window sizes, is called a logical connection. Each connection is uniquely identified by the pair of sockets used by the sending and receiving processes. Full Duplex TCP provides for concurrent data streams in both directions. 

TCP HEADER

TCP data is encapsulated in an IP datagram. The figure shows the format of the TCP header. Its normal size is 20 bytes unless options are present. Each of the fields is discussed below:

 

The SrcPort and DstPort fields identify the source and destination ports,respectively. These two fields plus the source and destination IP addresses, combine to uniquely identify each TCP connection.

 

The sequence number identifies the byte in the stream of data from the sending TCP to the receiving TCP that the first byte of data in this segment represents.

 

The Acknowledgement number field contains the next sequence number that the sender of the acknowledgement expects to receive. This is therefore the sequence number plus 1 of the last successfully received byte of data. This field is valid only if the ACK flag is on. Once a connection is established the Ack flag is always on.

 

The Acknowledgement, SequenceNum, and AdvertisedWindow fields are all involved in TCP's sliding window algorithm.The Acknowledgement and AdvertisedW indow fields carry information about the flow of dat going in the other direction. In TCP's sliding window algorithm the reciever advertises a window size to the sender. This is done using the AdvertisedWindow field. The sender is then limited to having no more than a value of AdvertisedWindow bytes of un acknowledged data at any given time. The receiver sets a suitable value for the AdvertisedWindow based on the amount of memory allocated to the connection for the purpose of buffering data.

 

The header length gives the length of the header in 32-bit words. This is required because the length of the options field is variable.

 

The 6-bit Flags field is used to relay control information between TCP peers. The possible flags include SYN, FIN, RESET, PUSH, URG, and ACK.

• The SYN and Fin flags are used when establishing and terminating a TCP connection, respectively.
• The ACK flag is set any time the Acknowledgement field is valid, implying that the receiver should pay attention to it.
• The URG flag signifies that this segment contains urgent data. When this flag is set, the UrgPtr field indicates where the non-urgent data contained in this segment begins.
• The PUSH flag signifies that the sender invoked the push operation, whic h indicates to the receiving side of TCP that it should notify the receiving process of this fact.
• Finally, the RESET flag signifies that the receiver has become confused and so wants to abort the connection.

 

The Checksum covers the TCP segment: the TCP header and the TCP data. This is a mandatory field that must be calculated by the sender, and then verified by the receiver.

 

The Option field is the maximum segment size option, called the MSS. Each end of the connection normally specifies this option on the first segment exchanged. It specifies the maximum sized segment the sender wants to recieve.

 

The data portion of the TCP segment is optional.

TCP STATE TRANSITION DIAGRAM

 

The two transitions leading to the ESTABLISHED state correspond to the opening of a connection, and the two transitions leading from the ESTABLISHED state are for the termination of a connection. The ESTABLISHED state is where data transfer can occur between the two ends in both the directions.

If a connection is in the LISTEN state and a SYN segment arrives, the connection makes a transition to the SYN_RCVD state and takes the action of replying with an ACK+SYN segment. The client does an active open which causes its end of the connection to send a SYN segment to the server and to move to the SYN_SENT state. The arrival of the SYN+ACK segment causes the client to mo ve to the ESTABLISHED state and to send an ack back to the server. When this ACK arrives the server finally moves to the ESTABLISHED state. In other words, we have just traced the THREE-WAY HANDSHAKE.

In the process of terminating a connection, the important thing to kee p in mind is that the application process on both sides of the connection must independently close its half of the connection. Thus, on any one side there are three combinations of transition that get a connection from the ESTABLISHED state to the CLOSED state:

• This side closes first:

ESTABLISHED -> FIN_WAIT_1-> FIN_WAIT_2 -> TIME_WAIT -> CLOSED.

• The other side closes first:

ESTABLISHED -> CLOSE_WAIT -> LAST_ACK -> CLOSED.

• Both sides close at the same time:
  ESTABLISHED -> FIN_WAIT_1-> CLOSING ->TIME_WAIT -> CLOSED.

The main thing to recognize about connection teardown is that a connection in the TIME_WAIT state cannot move to the CLOSED state until it has waited for two times the maximum amount of time an IP datagram might live in the Inter net. The reason for this is that while the local side of the connection has sent an ACK in response to the other side's FIN segment, it does not know that the ACK was successfully delivered. As a consequence this other side might re transmit its FIN segment, and this second FIN segment might be delayed in the network. If the connection were allowed to move directly to the CLOSED state, then another pair of application processes might come along and open the same connection, and the delayed FIN segment from the earlier incarnation of the connection would immediately initiate the termination of the later incarnation of that connection.

SLIDING WINDOW

 

The sliding window serves several purposes:

(1) it guarantees the reliable delivery of data

(2) it ensures that the data is delivered in order,

(3) it enforces flow control between the sender and the receiver.

Reliable and ordered delivery

The sending and receiving sides of TCP interact in the following manner to implement reliable and ordered delivery:

Each byte has a sequence number.

ACKs are cumulative.

Sending side

• LastByteAcked <=LastByteSent
• LastByteSent <= LastByteWritten
• bytes between LastByteAcked and LastByteWritten must be buffered.

Receiving side

• LastByteRead < NextByteExpected
• NextByteExpected <= LastByteRcvd + 1
• bytes between NextByteRead and LastByteRcvd must be buffered.

Flow Control

Sender buffer size : MaxSendBuffer

Receive buffer size : MaxRcvBuffer

Receiving side

• LastByteRcvd - NextBytteRead <= MaxRcvBuffer
• AdvertisedWindow = MaxRcvBuffer - (LastByteRcvd - NextByteRead)

Sending side

• LastByteSent - LastByteAcked <= AdvertisedWindow
• EffectiveWindow = AdvertisedWindow - (LastByteSent - LastByteAcked)
• LastByteWritten - LastByteAcked <= MaxSendBuffer
• Block sender if (LastByteWritten - LastByteAcked) + y > MaxSendBuffer

Always send ACK in response to an arriving data segment

Persist when AdvertisedWindow = 0

Adaptive Retransmission

TCP guarantees reliable delivery and so it retransmits each segment if an ACK is not received in a certainperiod of time. TCP sets this timeout as a function of the RTT it expects between the two ends of the connection. Unfortunately, given the range of possible RTT's between any pair of hosts in the Internet, as well as the variation in RTT between the same two hosts over time, choosing an appropriate timeout value is not that easy. To address this problem, TCP uses an adaptive retransmission medchanism. We describe this mechanism and how it has evolved over time.

 

Original Algorithm

Measure SampleRTT for each segment/ACK pair

Compute weighted average of RTT

EstimatedRTT = a*EstimatedRTT + b*SampleRTT, where a+b = 1

a between 0.8 and 0.9

b between 0.1 and 0.2

Set timeout based on EstimatedRTT

TimeOut = 2 * EstimatedRTT

 

Karn/Partridge Algorithm

Do not sample RTT when retransmitting

Double timeout after each retransmission

Jacobson/Karels Algorithm

New calculation for average RTT

Difference = SampleRTT - EstimatedRTT

EstimatedRTT = EstimatedRTT + ( d * Difference)

Deviation = Deviation + d ( |Difference| - Deviation)), where d is a fraction between 0 and 1

Consider variance when setting timeout value

Timeout = u * EstimatedRTT + q * Deviation, where u = 1 and q = 4

Congestion Control

Slow Start

It operates by observing that the rate at which new packets should be injected into the network is the rate at which the acknowledgments are returned by the other end.

Slow start adds another window to the sender's TCP: the congestion window, called "cwnd". When a new connection is established with a host on another network, the congestion window is initialized to one segment (i.e., the segment size announced by the other end, or the default, typically 536 or 512). Each time an ACK is received, the congestion window is increased by one segment. The sender can transmit up to the minimum of the congestion window and the advertised window. The congestion window is flow control imposed by the sender, while the advertised window is flow control imposed by the receiver. The former is based on the sender's assessment of perceived network congestion; the latter is related to the amount of available buffer space at the receiver for this connection.

The sender starts by transmitting one segment and waiting for its ACK. When that ACK is received, the congestion window is incremented from one to two, and two segments can be sent. When each of those two segments is acknowledged, the congestion window is increased to four. This provides an exponential growth, although it is not exactly exponential because the receiver may delay its ACKs, typically sending one ACK for every two segments that it receives.

At some point the capacity of the internet can be reached, and an intermediate router will start discarding packets. This tells the sender that its congestion window has gotten too large.

Early implementations performed slow start only if the other end was on a different network. Current implementations always perform slow start.

Congestion Avoidance

Congestion can occur when data arrives on a big pipe (a fast LAN) and gets sent out a smaller pipe (a slower WAN). Congestion can also occur when multiple input streams arrive at a router whose output capacity is less than the sum of the inputs. Congestion avoidance is a way to deal with lost packets.

The assumption of the algorithm is that packet loss caused by damage is very small (much less than 1%), therefore the loss of a packet signals congestion somewhere in the network between the source and destination. There are two indications of packet loss: a timeout occurring and the receipt of duplicate ACKs.

Congestion avoidance and slow start are independent algorithms with different objectives. But when congestion occurs TCP must slow down its transmission rate of packets into the network, and then invoke slow start to get things going again. In practice they are implemented together.

Congestion avoidance and slow start require that two variables be maintained for each connection: a congestion window, cwnd, and a slow start threshold size, ssthresh. The combined algorithm operates as follows:

1. Initialization for a given connection sets cwnd to one segment and ssthresh to 65535 bytes.

2. The TCP output routine never sends more than the minimum of cwnd and the receiver's advertised window.

3. When congestion occurs (indicated by a timeout or the reception of duplicate ACKs), one-half of the current window size (the minimum of cwnd and the receiver's advertised window, but at least two segments) is saved in ssthresh. Additionally, if the congestion is indicated by a timeout, cwnd is set to one segment (i.e., slow start).

4. When new data is acknowledged by the other end, increase cwnd, but the way it increases depends on whether TCP is performing slow start or congestion avoidance.

If cwnd is less than or equal to ssthresh, TCP is in slow start; otherwise TCP is performing congestion avoidance. Slow start continues until TCP is halfway to where it was when congestion occurred (since it recorded half of the window size that caused the problem in step 2), and then congestion avoidance takes over.

Slow start has cwnd begin at one segment, and be incremented by one segment every time an ACK is received. As mentioned earlier, this opens the window exponentially: send one segment, then two, then four, and so on. Congestion avoidance dictates that cwnd be incremented by segsize*segsize/cwnd each time an ACK is received, where segsize is the segment size and cwnd is maintained in bytes. This is a linear growth of cwnd, compared to slow start's exponential growth. The increase in cwnd should be at most one segment each round-trip time (regardless how many ACKs are received in that RTT), whereas slow start increments cwnd by the number of ACKs received in a round-trip time.

Fast Retransmit

TCP may generate an immediate acknowledgment (a duplicate ACK) when an out- of-order segment is received. This duplicate ACK should not be delayed. The purpose of this duplicate ACK is to let the other end know that a segment was received out of order, and to tell it what sequence number is expected.

Since TCP does not know whether a duplicate ACK is caused by a lost segment or just a reordering of segments, it waits for a small number of duplicate ACKs to be received. It is assumed that if there is just a reordering of the segments, there will be only one or two duplicate ACKs before the reordered segment is processed, which will then generate a new ACK. If three or more duplicate ACKs are received in a row, it is a strong indication that a segment has been lost. TCP then performs a retransmission of what appears to be the missing segment, without waiting for a retransmission timer to expire.

Fast Recovery

After fast retransmit sends what appears to be the missing segment, congestion avoidance, but not slow start is performed. This is the fast recovery algorithm. It is an improvement that allows high throughput under moderate congestion, especially for large windows.

The reason for not performing slow start in this case is that the receipt of the duplicate ACKs tells TCP more than just a packet has been lost. Since the receiver can only generate the duplicate ACK when another segment is received, that segment has left the network and is in the receiver's buffer. That is, there is still data flowing between the two ends, and TCP does not want to reduce the flow abruptly by going into slow start.

The fast retransmit and fast recovery algorithms are usually implemented together as follows.

1. When the third duplicate ACK in a row is received, set ssthresh to one-half the current congestion window, cwnd, but no less than two segments. Retransmit the missing segment. Set cwnd to ssthresh plus 3 times the segment size. This inflates the congestion window by the number of segments that have left the network and which the other end has cached .

2. Each time another duplicate ACK arrives, increment cwnd by the segment size. This inflates the congestion window for the additional segment that has left the network. Transmit a packet, if allowed by the new value of cwnd.

3. When the next ACK arrives that acknowledges new data, set cwnd to ssthresh (the value set in step 1). This ACK should be the acknowledgment of the retransmission from step 1, one round-trip time after the retransmission. Additionally, this ACK should acknowledge all the intermediate segments sent between the lost packet and the receipt of the first duplicate ACK. This step is congestion avoidance, since TCP is down to one-half the rate it was at when the packet was lost.

What is the difference between FTP, FTPS and SFTP?

The most common protocols for file transfer in use today are FTP, FTPS and SFTP.  While these protocols may sound the same, there are some key differences among them.  Learning these differences can eliminate some of the common problems experienced when performing a file transfer.

FTP

The FTP (File Transfer Protocol) protocol has been around for quite some time.  The protocol itself is described in RFC 959 which was last updated in 1985.  The FTP protocol consists of two channels, the command channel and data channel.  These channels are responsible for exchanging commands and data in an FTP client session.

The command channel typically runs on server port 21 and is responsible for handling the exchange of simple commands between FTP server and client.  The USER and PASS commands used for authenticating an FTP user are examples of commands that are exchanged on the command channel.  The command channel remains open until the client sends the QUIT command to disconnect or the server forcibly disconnects the client.

The data channel runs on temporary random ports listening on the server (passive mode) or on the client (active mode) and are responsible for exchanging data in the form of file transfers and directory listings.  The LIST command used for getting a FTP server directory listing is an example of a command that opens a data channel.  Unlike the command channel which remains alive during the entire FTP session, the data channel automatically shuts down once the transfer of data is complete.

FTPS

When the FTP protocol was initially drafted security was not a primary concern.  Since then many things have changed and sending data over the Internet or any other insecure network without encryption is considered a big no-no.  In order to address this issue a set of security extensions to the original FTP protocol were proposed in RFC 2228 that protect FTP data as it travels over the network using SSL encryption.  However, just to make things more complicated, FTPS is available in two forms known as FTPS Implicit SSL and FTPS Explicit SSL.

FTPS Implicit SSL

In implicit SSL mode a required SSL session is established between client and server before any data is exchanged.  In other words, the use of SSL is implied because any attempt made by a non-SSL client would automatically be refused by the server.  Typically FTPS implicit SSL services run on port 990.

FTPS Explicit SSL

In explicit SSL mode the client can optionally switch from unencrypted mode to SSL.  This is useful in that the server can support both unencrypted FTP and encrypted FTPS sessions on a single port, typically port 21. In an explicit SSL session the client first establishes an unencrypted connection to FTP service.  Prior to sending user credentials, the client then requests that the server switch the command channel to an SSL encrypted channel using the client AUTH TLS or AUTH SSL commands.  Upon successful setup of the SSL channel the client then sends user credentials to the FTP server.  These credentials along with any other commands sent to server during the FTP session are automatically encrypted by the SSL channel. 

SFTP

SFTP is most often confused with FTPS and vice-versa.  However, unlike FTP and FTPS these protocols are not at all related.  SFTP is actually a sub-system of the SSH (Secure Shell) protocol and typically runs on port 22.  Unlike FTP/S, SFTP does not have the concept of separate command and data channels.  Instead both data and commands are transferred in specially formatted packets via a single connection.  Furthermore, unlike FTPS explicit SSL, SFTP encrypts the entire session and does not offer the ability to switch between unencrypted and encrypted mode. 

NTLM VS Kerberos

NTLM Authentication: Challenge- Response mechanism.

In the NTLM protocol, the client sends the user name to the server; the server generates and sends a challenge to the client; the client encrypts that challenge using the user’s password; and the client sends a response to the server.If it is a local user account, server validate user’s response by looking into the Security Account Manager; if domain user account, server forward the response to domain controller for validating and retrive group policy of the user account, then construct an access token and establish a session for the use. 

Kerberos authentication: Trust-Third-Party Scheme.

Kerberos authentication provides a mechanism for mutual authentication between a client and a server on an open network.The three heads of Kerberos comprise the Key Distribution Center (KDC), the client user and the server with the desired service to access. The KDC is installed as part of the domain controller and performs two service functions: the Authentication Service (AS) and the Ticket-Granting Service (TGS). When the client user log on to the network, it request a Ticket Grant Ticket(TGT) from the AS in the user’s domain; then when client want to access the network resources, it presents the TGT, an authenticator and Server Principal Name(SPN) of the target server, contact the TGS in the service account domain to retrive a session ticket for future communication w/ the network service, once the target server validate the authenticator, it create an access token for the client user.