Most of the information about carp + PF + pfsync clusters on the internet is ob, so I found some machines and moved them to FB. the experiment was successful and there may be some problems. Please discover and correct them !!!!
Hardware:
Five servers (actually only four servers are used, two are used for loadbalance (one for Master/Slave nodes, each server requires at least three NICS), and two are used for application services .)
Cisco2950 2
System: FreeBSD 7.0 release (minimal installation)
Goals:
1. Implement High Availability (ha) of loadbalance through pfsync );
2. Load Balancing of the server pool;
Test address of my deployment environment (the cluster cannot be retained for a long time. The cluster is deployed in the internal network, so the external network is redirected to carp0 ):
The test address is not put.
I. network topology:
Internet: 192.168.1.0/24
Intranet: 192.168.10.0/24
Ha dedicated: 10.10.10.250/10.10.10.20.
Screen. width * 0.7) {This. resized = true; this. width = screen. width * 0.7; this. style. cursor = 'hand'; this. alt = 'click here to open new window \ nctrl + mouse wheel to zoom in/out';} "onclick =" If (! This. resized) {return true;} else {window. open ('HTTP: // loads);} "alt =" "src =" http://bbs.chinaunix.net/attachments/month_0805/20080515_659d2f4c6701ca7fd002d4DihrUqt2c3.png "onLoad =" If (this. width> screen. width * 0.7) {This. resized = true; this. width = screen. width * 0.7; this. alt = 'click here to open new window \ nctrl + mouse wheel to zoom in/out';} "border = 0>
Screen. width * 0.7) {This. resized = true; this. width = screen. width * 0.7; this. style. cursor = 'hand'; this. alt = 'click here to open new window \ nctrl + mouse wheel to zoom in/out';} "onclick =" If (! This. resized) {return true;} else {window. open ('HTTP: // bbs.chinaunix.net/images/default/attachimg.gif');} "alt =" "src =" http://bbs.chinaunix.net/images/default/attachimg.gif "onLoad =" If (this. width> screen. width * 0.7) {This. resized = true; this. width = screen. width * 0.7; this. alt = 'click here to open new window \ nctrl + mouse wheel to zoom in/out';} "border = 0>
Load Balance Configuration:
A. kernel settings:
The kernel configurations of Master/Slave are the same.
Device Carp
Device pf # Start a virtual network device to record traffic (through BPF)
Device pflog # Start a virtual network device to monitor the network status
Device pfsync
Options altq
Options altq_cbq # classification-based sorting (CBQ)
Options altq_red # Random Early Detection (red)
Options altq_rio # Red the incoming and outgoing packets
Options altq_hfsc # hierarchical package scheduler (hfsc)
Options altq_priq # sort by priority (priq)
Options altq_nopcc # This parameter must be used when the SMP kernel is integrated. The read clock is disabled.
Recompile the kernel.
B. Configure RC. conf.
Only configuration items related to this chapter are written here.
Master part:
Gateway_enable = "yes"
Defaultrouter = "192.168.1.1"
Hostname = "master.cluster.org"
Cloned_interfaces = "carp0 carp1"
# External public interface (for the secondary firewall use a different public IP .)
Ifconfig_em0 = "Inet 192.168.1.52 netmask 255.255.255.0"
# External public carp Interface
# Ifconfig_carp0 = "vhid 1 pass 11111 192.168.1.51/24"
Ifconfig_carp0 = "vhid 1 pass 11111 192.168.1.51/24 advskew 10"
# Internal interface (for the secondary firewall change the IP address to 192.168.10.11)
Ifconfig_em1 = "Inet 192.168.10.10 netmask 255.255.255.0"
# Internal carp Interface
Ifconfig_carp1 = "vhid 1 pass 22222 192.168.10.100/24 advskew 10"
# Heartbeat interface (for the secondary firewall, change the IP address to 10.10.10.20)
Ifconfig_vr0 = "10.10.10.250 netmask 255.255.255.0"
# Pfsync Interface
Ifconfig_pfsync0 = "Up syncif vr0"
Pf_enable = "yes"
Pf_rules = "/etc/PF. conf"
Pf_flags = ""
Pflog_enable = "yes"
Pflog_logfile = "/var/log/pflog"
Pflog_flags = ""
Slave part:
# The slave configuration is approximately similar to the master configuration. The major changes are the priority values of the local IP address and advskew. Note: The carp IP address must be public.
Gateway_enable = "yes"
Defaultrouter = "192.168.1.1"
Hostname = "slave.cluster.org"
Cloned_interfaces = "carp0 carp1"
# External public interface (for the primary firewall use a different public IP .)
Ifconfig_em0 = "Inet 192.168.1.53 netmask 255.255.255.0"
# External public carp Interface
# Ifconfig_carp0 = "vhid 1 pass 11111 192.168.1.51/24"
Ifconfig_carp0 = "vhid 1 pass 11111 192.168.1.51/24 advskew 20"
# Internal interface (for the primary firewall change the IP address to 192.168.10.10)
Ifconfig_em1 = "Inet 192.168.10.11 netmask 255.255.255.0"
# Internal carp Interface
Ifconfig_carp1 = "vhid 1 pass 22222 192.168.10.100/24 advskew 20"
# Heartbeat interface (for the primary firewall, change the IP address to 10.10.10.250)
Ifconfig_fxp0 = "10.10.10.20.netmask 255.255.255.0"
# Pfsync Interface
Ifconfig_pfsync0 = "Up syncif fxp0"
Pf_enable = "yes"
Pf_rules = "/etc/PF. conf"
Pf_flags = ""
Pflog_enable = "yes"
Pflog_logfile = "/var/log/pflog"
Pflog_flags = ""
C. Pf. conf rules
Except for the nic id, the master and slave are the same.
######################################## ########################################
# Macro and lists
######################################## ########################################
Lop_if = "lo0"
Ext_if = "em0"
Int_if = "Em1"
Sync_if = "vr0"
Ext_carp = "carp0"
Web_ports = "{80,443 }"
# Web_servers = "{192.168.10.20, 192.168.10.21, 192.168.10.22 }"
# Web_servers = "{192.168.10.20 }"
Web_servers = "{192.168.10.20, 192.168.10.21 }"
######################################## ########################################
# Options, scrub and Nat
######################################## ########################################
Set block-policy drop
Set skip on $ lop_if
Scrub in
Nat on $ ext_if from $ int_if: Network to any-> $ ext_if
######################################## ########################################
# Redirection
######################################## ########################################
# RDR on $ ext_if proto TCP from any to any port 80-> $ web_servers round-robin sticky-address
RDR on $ ext_if proto TCP from any to $ ext_carp port $ web_ports-> $ web_servers round-robin sticky-address
######################################## ########################################
# Filtering rules
######################################## ########################################
Pass quick on {$ sync_if} proto pfsync keep state (no-sync)
Pass on {$ ext_if, $ int_if} proto carp keep state
D. Configure sysctl. conf
# Master and slave are the same.
Net. inet. Carp. preempt = 1
Net. inet. tcp. blackhole = 2
Net. inet. UDP. blackhole = 1
Net. inet. tcp. sendspace = 65536
Net. inet. tcp. recvspace = 65536
E. Restart the server after configuration. Check the system status.
Master
Master # ifconfig
Vr0: Flags = 8843 metric 0 MTU 1500
Options = 8
Ether 00: 05: 5D: 85: 84: D8
Inet 10.10.10.250 netmask 0xffffff00 broadcast 10.10.10.255
Media: Ethernet Autoselect (100 basetx)
Status: Active
Em0: Flags = 8943 metric 0 MTU 1500
Options = 9B
Ether 00: C0: 9f: 31: 25: A2
Inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
Media: Ethernet Autoselect (100 basetx)
Status: Active
Em1: Flags = 8943 metric 0 MTU 1500
Options = 9B
Ether 00: C0: 9f: 31: 25: A3
Inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255
Media: Ethernet Autoselect (100 basetx)
Status: Active
Pflog0: Flags = 141 metric 0 MTU 33204
Lo0: Flags = 8049 metric 0 MTU 16384
Inet 127.0.0.1 netmask 0xff000000
Pfsync0: Flags = 41 metric 0 MTU 1460
Pfsync: syncdev: vr0 syncpeer: 224.0.0.240 maxupd: 128
Carp0: Flags = 49 metric 0 MTU 1500
Inet 192.168.1.51 netmask 0xffffff00
Carp: Master vhid 1 advbase 1 advskew 10
Carp1: Flags = 49 metric 0 MTU 1500
Inet 192.168.10.100 netmask 0xffffff00
Carp: Master vhid 1 advbase 1 advskew 10
On slave:
Slave # ifconfig
Fxp0: Flags = 8843 metric 0 MTU 1500
Options = 8
Ether 00: 07: E9: 1b: 4b: CD
Inet 10.10.20.netmask 0xffffff00 broadcast 10.10.10.255
Media: Ethernet Autoselect (100 basetx)
Status: Active
Em0: Flags = 8943 metric 0 MTU 1500
Options = 9B
Ether 00: C0: 9f: 38: BD: af
Inet 192.168.1.53 netmask 0xffffff00 broadcast 192.168.1.255
Media: Ethernet Autoselect (100 basetx)
Status: Active
Em1: Flags = 8943 metric 0 MTU 1500
Options = 9B
Ether 00: C0: 9f: 38: BD: B0
Inet 192.168.10.11 netmask 0xffffff00 broadcast 192.168.10.255
Media: Ethernet Autoselect (100 basetx)
Status: Active
Lo0: Flags = 8049 metric 0 MTU 16384
Inet 127.0.0.1 netmask 0xff000000
Pflog0: Flags = 141 metric 0 MTU 33204
Pfsync0: Flags = 41 metric 0 MTU 1460
Pfsync: syncdev: fxp0 syncpeer: 224.0.0.240 maxupd: 128
Carp0: Flags = 49 metric 0 MTU 1500
Inet 192.168.1.51 netmask 0xffffff00
Carp: Backup vhid 1 advbase 1 advskew 20
Carp1: Flags = 49 metric 0 MTU 1500
Inet 192.168.10.100 netmask 0xffffff00
Carp: Backup vhid 1 advbase 1 advskew 20
Ii. Server pool configuration.
Unified configuration: point the gateway to 192.168.10.100.
Defaultrouter = "192.168.10.100"
I deployed the same application (Lighttpd + PHP-fcgi) on the two servers, so I will not write the configuration.
Because the server is not enough, MySQL is deployed on S2, and two servers are connected to it.
Iii. Test.
A. Static Page:
Separate and create an index.htm page on s12.
The content of index.htm on s1is:
"Hi, this is No.1 server"
The content of index.htm on s2is:
"Hi, this is No. 2 server"
Access address:
Http: // 192.168.1.51/index.htm
The returned content is S1 and S2 respectively. note: The status is maintained during scheduling. therefore, you need to close the browser and then open the browser to see different content. it is recommended that several people access the service together to return different results.
B. dynamic page:
Phpwind is deployed on S1 and S2. Assume that phpwind is under./BBS.
Access address:
Http: // 192.168.1.51/BBS/
The returned content remains consistent,
I have not done anything about file sharing. The actual method is to use NFS and iSCSI. I cannot call iSCSI storage, and I don't want to use NFS.
Master # pfctl-S state
All carp 192.168.1.52-> 224.0.0.18 single: no_traffic
All pfsync 224.0.0.240 224.0.0.240 single: no_traffic
All carp 224.0.0.18 224.0.0.18 single: no_traffic
All TCP 192.168.10.21: 65346-> 192.168.1.52: 61575-> 130.104.5.67: 25 fin_wait_2: fin_wait_2
All TCP 192.168.10.20: 80 metric 0 MTU 1500.
Inet 192.168.1.51 netmask 0xffffff00
Carp: Master vhid 1 advbase 1 advskew 20
You can perform some tests.
Finally, some optimizations can be made :)
References:
Http://www.countersiege.com/doc/pfsync-carp/
Http://blog.randomutterings.com/articles/2007/06/15/redundant-failover-firewall-with-pf-pfsync-and-carp-on-freebsd
