A single linux server allocates multiple ADSL connections and mounts multiple fixed public ip addresses at the same time. the squid is used to implement the proxy pool scheme.

I. background: in the big data era, who owns data can have a future. many Internet companies do not generate data themselves. what should they do?, so a considerable portion of the current huge network traffic is crawling traffic, such as search engine spider, crawling of commodity data on e-commerce websites

I. background:

In the big data era, who owns data can have a future. many Internet companies do not generate data themselves. what should they do? , so a considerable portion of the current huge network traffic is crawling traffic, such as search engine spider and crawling of commodity data on e-commerce websites. As described at the beginning of this article, data is the future. if there is data for you to grasp, it will impose some access frequency restrictions on user access, if this parameter is exceeded, the ip address is blacklisted (for example, the ip address is blocked, the user agent is blocked, and the url is blocked, blocking means that normal users cannot access the service. The problem solved in this article is how to quickly and easily switch the ip address and change the ip address as needed after the ip address is blocked.

II. scheme diagram in use:

III. Hardware Requirements:

1. a two-layer switch that supports VLANs. for example, a maximum of 23 adsl cats can be connected with a maximum of 24 ports.

2. configure a general linux server, preferably a gigabit Nic and three NICs (theoretically, one Nic can also be used to avoid mutual influence and fault tolerance, so that the network structure is clear, use 3 Gigabit NICs)

3. common intranet switches.

IV. core technical principles:

1. a single ADSL vlan is used as a trunk for the port connected to the server. Why is a single adsl vlan required? Because China Unicom, China Telecom, and other carriers impose mac address restrictions on each adsl account, one mac address can only dial one number, this is why only one adsl instance is available when two or more adsl instances are simultaneously dialing.

2. virtual interfaces with the same vlan ID as adsl are created on the linux server. Why create vlan virtual interfaces in linux? To isolate data from adsl and reduce mutual interference. each ad is independent of each other. if a sub-Nic is used, the mac address cannot be modified, even if the modification is successful, all the sub-nics and the mac of the primary Eni are the same and cannot meet the requirements of multiple adsl connections at the same time. Therefore, vlan virtual interfaces must be used.

3. the Nic connected to the optical fiber uses a sub-Nic to Mount multiple public ip addresses

4. the linux server uses the source address routing technology. Why? Squid's tcp_outgoing_address only supports ip addresses, and does not support device names (such as ppp0 and eth0). the ip addresses obtained by adsl dialing also change. Therefore, you need to configure a private fixed ip address for the vlan sub-interface, bind the egress adsl through the source address route.

5. select tcp_outgoing_address for export ip through squid account reference http://blog.csdn.net/xuyaqun/article/details/9623635

V. configuration
1. vlan switch configuration: Omitted

The relationship between the port and vlan ID: port1 --> vlan 101, port2 --> vlan 102..., port23 --> vlan 123, port24 --> Trunk

2. linux configuration
The dynamic vlan virtual interface must correspond to the vlan ID on the vlan switch:
Modprobe 8021q
Vconfig add eth0 101
Ip link set dev eth0.101 address 00: 40: 50: 60: 70: 11
Ifconfig-s eth0.101 10.1.1.1 broadcast 10.1.1.0 netmask 255.255.255.0 up

Vconfig add eth0 102
Ip link set dev eth0.102 address 00: 40: 50: 60: 70: 12
Ifconfig-s eth0.102 10.1.2.1 broadcast 10.1.2.0 netmask 255.255.255.0 up

Or write the configuration file:

Boot and load the vlan module:

Echo "modprobe 8021q">/etc/rc. local

Create a vlan interface configuration file:

# More/etc/sysconfig/network-scripts/ifcfg-eth1.101
DEVICE = eth1.101
VLAN = yes
BOOTPROTO = none
ONBOOT = yes
HWADDR = 00: 50: 56: BA: 1D: 00
TYPE = Ethernet
NETMASK = 255.255.255.0
IPADDR = 192.168.101.1

View vlan virtual interfaces
Cat/proc/net/vlan/config
Cat/proc/net/vlan/eth0.101
Cat/proc/net/dev

3. ADSL dialing configuration

ADSL account configuration file:

# More/etc/ppp/chap-secrets
# Secrets for authentication using CHAP
# Client server secret IP addresses
"Root" * "123456"
"U1" * "123456"
"U2" * "123456"

ADSL interface configuration file (note the red part ):

# More/etc/sysconfig/network-scripts/ifcfg-ppp1
USERCTL = yes
BOOTPROTO = dialup
NAME = DSLppp1
DEVICE = ppp1
TYPE = xDSL
ONBOOT = no
PIDFILE =/var/run/pppoe-adsl-ppp1.pid
FIREWALL = NONE
PING =.
PPPOE_TIMEOUT = 80
LCP_FAILURE = 3
LCP_INTERVAL = 20
Clampm SS = 1412
CONNECT_POLL = 6
CONNECT_TIMEOUT = 60
DEFROUTE = no
SYNCHRONOUS = no
ETH = eth0: 1
PROVIDER = DSLppp1
USER = u2
PEERDNS = no
DEMAND = no

ADSL dialing/hanging up:

Ifup/ifdown ppp0

Adsl-start/adsl-stop/etc/sysconfig/network-scripts/ifcfg-ppp0

View adsl dialing status

Ifconfig

Pppoe-status/etc/sysconfig/network-scripts/ifcfg-ppp0

4. configure the sub-Nic: Omitted

Effect of sub-Nic configuration:

Eth1: 163 Link encap: Ethernet HWaddr 00: 0A: F7: 0F: 4A: E8
Inet addr: 8.8.148.163 Bcast: 8.8.148.175 Mask: 255.255.255.0
Up broadcast running multicast mtu: 1500 Metric: 1

Eth1: 164 Link encap: Ethernet HWaddr 00: 0A: F7: 0F: 4A: E8
Inet addr: 8.8.148.164 Bcast: 8.8.148.175 Mask: 255.255.255.0
Up broadcast running multicast mtu: 1500 Metric: 1

Eth1: 165 Link encap: Ethernet HWaddr 00: 0A: F7: 0F: 4A: E8
Inet addr: 8.8.148.165 Bcast: 8.8.148.175 Mask: 255.255.255.0
Up broadcast running multicast mtu: 1500 Metric: 1

5. ADSL source address routing configuration:

You only need to route the source address to the vlan interface IP address that corresponds to the adsl link.
Iptables-t nat-a postrouting-s 10.0.1.1/255.255.255.255-o ppp0-j MASQUERADE
Ip rule add from 10.0.1.1 lookup 5
Ip route add default dev ppp0 table 5

Iptables-t nat-a postrouting-s 10.0.2.1/255.255.255.255-o ppp1-j MASQUERADE
Ip rule add from 10.0.2.1 lookup 6
Ip route add default dev ppp1 table 6

6, squid Configuration: Omitted, see http://blog.csdn.net/xuyaqun/article/details/9623635

6. test whether the network interfaces are normal:

Ping the specified device name or interface IP address

Ping-I eth0 8.8.8.8
Ping-I ppp0 8.8.8.8
Ping-I ppp1 8.8.8.8

Ping-I 8.8.245.163 8.8.8.8

Or

Test the device name specified by traceroute

Traceroute-I eth0 8.8.8.8
Traceroute-I ppp0 8.8.8.8
Traceroute-I ppp1 8.8.8.8

7. after all the configurations are completed, the usage results are as follows:

8. monitoring and alerting proxy pools

Cacti monitoring results:

Nagios monitoring results:

Reference connection:

Http://www.junxiwang.com/forum.php? Mod = viewthread & tid = 76
Http://www.itxbo.com/network/860.html
Http://www.imjune.com/cuier/240.html
Http:// OS .51cto.com/art/200712/62152_all.htm