General Linux server network performance guide

Environment: Linux web server serves browser client and Database server on the local LAN serves the application server which is also on the same LAN
Objective: maximum the network performance which is one of the 4 performance bottlenecks(CPU,Memory,Storage and Network I/O)


Parameters:

1. net.core.wmem_default(/proc/sys/net/core/wmem_default) and net.core.rmem_default(/proc/sys/net/core/rmem_default) (the following settings are also recommended by Oracle 11gR1 installation)

net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 4194304

For Oracle database, it's not recommended to configure net.ipv4.tcp_rmem and net.ipv4.tcp_wmem.

as stated on metalink.

2. net.core.netdev_max_backlog (/proc/sys/net/core/netdev_max_backlog), default is 1000 in Linux RHEL 5 kernel 2.6

net.core.netdev_max_backlog=30000

set maximum number of incoming packets that will be queued for delivery to the device queue.

3. net.core.somaxconn(/proc/sys/net/core/somaxconn), default is 128

maximum accept queue backlog that can be specified via the listen() system call. or the number of pending connection requests.

net.core.somaxconn=1536

4. optmem_max (/proc/sys/net/core/optmem_max)

maximum initialization size of socket buffers, expressed in bytes.

increase this  

TCP options:

5. net.ipv4.tcp_window_scaling (/proc/sys/net/ipv4/tcp_window_scaling), enable

6. disable net.ipv4.tcp_sack(/proc/sys/net/ipv4/tcp_sack)

on LAN, disabling this tcp_sack can actually improve performance.

when tcp_sack is disabled, you should also disable 7 and 8

7. net.ipv4.tcp_dsack(/proc/sys/net/ipv4/tcp_dsack)

8. net.ipv4.tcp_fack(/proc/sys/net/ipv4/tcp_fack)

9. net.ipv4.tcp_max_syn_backlog(/proc/sys/net/ipv4/tcp_max_syn_backlog)

controls the length of the tcp syn queue for each port. If client experience failures

connecting to busy servers, this value should be increased.

10. net.ipv4.tcp_synack_retries(/proc/sys/net/ipv4/tcp_synack_retries) set to 3

controls the number of times kernel tries to resend a response to an incoming syn/ack segments

11. net.ipv4.tcp_retries2 (/proc/sys/net/ipv4/tcp_retries2) set to 5

controls the number of times kernel tries to resend data to a remote host with which it has an established connection.

12. net.ipv4.tcp_max_tw_buckets (/proc/sys/net/ipv4/tcp_max_tw_buckets)

increase this to double value. 

13. net.ipv4.tcp_orphan_retries (/proc/sys/net/ipv4/tcp_orphan_retries)  set to 0

14. net.ipv4.tcp_fin_timeout set to 30

15. net.ipv4.tcp_keepalive

tcp_keepalive_time=120

tcp_keepalive_probes=3

tcp_keepalive_intvl=30

16. ip_local_port_range (net.ipv4.ip_local_port_range)

1024 65000

17. net.ipv4.tcp_window_scaling = 1

18. net.ipv4.tcp_timestamps = 1 

Partitions and File system performance:

1. database raid:

For oracle database server hard disk raid. Use raid1 for redo log, archivelog,

including flash recovery area archivelog, temporary tablespace.

use raid1+0 for database files

2. for partition on individual hard disk, the first partition for /boot, the second is 

for swap, the third is for /var, the fourth is for /usr, the last is for /home and /

for other partitions, the first partition is at the outer side of the hard disk which 

is much faster then the inside. outsider partition can be accessed faster than insider ones.

3. add 'noatime' for those often accessed partitions in /etc/fstab.

4. switch to another I/O scheduler 

root (hd0,0) kernel /vmlinuz-2.6.18-8.el5 ro root=/dev/sda2 elevator=deadline

5. swap partition size:

  RAM               Swap Space
  --------------------------------------------
  1 GB - 2 GB       1.5 times the size of RAM
  2 GB - 8 GB       Equal to the size of RAM
  more than 8GB     0.75 times the size of RAM

6. use hugepage and ramfs to improve performance.