Network Design and Implementation of warden in cloud foundry-iptable Rule Configuration

In cloud foundry V2, the platform uses warden technology to control and isolate resources for running user application instances.

In a brief introduction, warden, that is, if dea_ng needs to run the user application instance (this article does not consider the staging packaging environment provided by Warden container), it will send the corresponding request to Warden server, the warden server creates warden container and runs the application instance inside warden container. In the specific implementation of warden container, kernel Virtualization Technologies such as cgroups are used, ensures resource control and isolation of Process Groups in the iner. For the architecture and implementation of warden, refer to my blog post: Architecture and Implementation of warden in cloud foundry.

Warden Network Architecture

The functions of warden container can be considered to be similar to those of traditional virtual machines, but traditional virtual machines use hypervisor and other technologies to control and isolate resources, while warden container achieves this goal through the virtualization kernel. In terms of network, Warden container technology creates a virtual network card for internal use of warden container, in addition, the virtual network card inside the warden container is paired with a virtual network card on the host machine where the warden server is located to act as the external gateway of the container. Only two virtual NICs can be created, in principle, to ensure physical "connectivity", but it is difficult to achieve "connectivity" for inter-network communication ", therefore, when physical and virtual physical resources are complete, Warden server designs and adds multiple rules on the host machine through iptables to ensure that the communication of the entire network can meet the requirements of warden container, at the same time, it does not affect the communication between the host machine.

The following figure shows the relationship between the physical environment of the host where the warden server is located, the network layer of the iptables, the operating system, and the user application layer:

Warden container Network Communication Mode
In cloud foundry, because Warden container is used to run user application instances, there are currently two types of communication between Application User instances and the outside world:

1. The user application instance acts as a web server to provide HTTP access services;
2. When an application instance uses the service, it establishes a connection with the service instance.

Of course, some cloud foundry platform developers will surely think that user application instances can be used as a client to access other application instances or resources outside the cloud through URLs. Here, we need to declare that the cloud foundry cluster does not expose IP addresses to application instances by default. Secondly, if the cloud foundry cluster does not exist internally or cannot connect to a powerful DNS server, the URL used to parse external resources.
The above two points will be used as the main research content in this article to analyze the network of the host machine where Warden is located, and focus on iptables.


The net. Sh file of warden iptables configuration involves two parts:

1. When the warden server is started, call the warden/Warden/root/Linux/net. Sh file to develop some iptables rules for the host where the warden server is located;
2. When creating a warden container, call the warden/Warden/root/Linux/skeleton/NET file. SH: Specifies the iptables rules to be added after Warden container is created (including setting iptables for creating only one warden container and iptables for port ing for a warden container ).

Net. Sh setup_filter of warden Server
For details about Warden/root/Linux/net. Sh, you can refer to the two most critical methods in this shell script: setup_filter and setup_nat. The following is the source code implementation of setup_filter:

function setup_filter() {  teardown_filter  # Create or flush forward chain  iptables -N ${filter_forward_chain} 2> /dev/null || iptables -F ${filter_forward_chain}  iptables -A ${filter_forward_chain} -j DROP  # Create or flush default chain  iptables -N ${filter_default_chain} 2> /dev/null || iptables -F ${filter_default_chain}  # Always allow established connections to warden containers  iptables -A ${filter_default_chain} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT  for n in ${ALLOW_NETWORKS}; do    if [ "$n" == "" ]    then      break    fi    iptables -A ${filter_default_chain} --destination "$n" --jump RETURN  done  for n in ${DENY_NETWORKS}; do    if [ "$n" == "" ]    then      break    fi    iptables -A ${filter_default_chain} --destination "$n" --jump DROP  done  iptables -A ${filter_default_chain} --jump REJECT  # Accept packets related to previously established connections  iptables -I INPUT -m state --state ESTABLISHED,RELATED --jump ACCEPT -m comment --comment 'related-traffic'  # Reject traffic from containers to host  if [ "$ALLOW_HOST_ACCESS" != "true" ]; then    iptables -A INPUT -s ${POOL_NETWORK} --jump REJECT -m comment --comment 'block-traffic-to-host'  fi  # Forward outbound traffic via ${filter_forward_chain}  iptables -A FORWARD -i w-+ --jump ${filter_forward_chain}  # Forward inbound traffic immediately  default_interface=$(ip route show | grep default | cut -d' ' -f5 | head -1)  iptables -I ${filter_forward_chain} -i $default_interface --jump ACCEPT}

Next, we will analyze the meaning of each part of iptables step by step, and then connect each iptables to analyze its functions. The brief flowchart of iptables in the kernel is as follows:

The following step-by-step analysis of the filter_setup method:
1. In this net. Sh file, the teardown_filter method does the following work: Clear certain warden-related iptable chains and rules. The following two chains will be created: filter_forward_chain and filter_forward_chain. If these two chains exist previously, the two chains will be created later.
2. For connections that have been established with Warden container, iptables performs the accept operation. The code is:

  # Always allow established connections to warden containers  iptables -A ${filter_default_chain} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Note: This rule is added to the filter_default_chain chain. When the matching condition is: this packet indicates that the data packet is uploaded and transmitted in the established connection, the accept operation is performed on the data packet. This rule is well understood from a functional point of view, that is, if warden container initiates a request internally to establish a connection to the outside, such packets will always be accepted when a packet is returned on the connection. However, I have always had a lot of doubts about the underlying implementation of this rule. After analyzing more knowledge about the warden network background, we will discuss it further.
3. In filter_default_chain, always take the accept action for the network host address in allow_networks, and use the drop action for the network host address in deny_networks. Here, we can briefly describe the difference between drop and reject in iptables. Drop: directly discard the data packet. Reject: discard the original data packet, and then return an information packet similar to an ICMP error to the sender of the data packet.
4. The subsequent shell code is iptables-A $ {filter_default_chain} -- jump reject, because there are a number of iptables <br
Rule matching of data packets in the same chain starts from the first one. If the matching succeeds, the corresponding operation is executed. Otherwise, the operation continues, therefore, this rule indicates that all data packets in this face cannot be matched. The kernel will take the reject operation to discard the data packets and return an error data packet. At this point, the rule settings in filter_default_chain are basically complete.
5. add Rules in the input chain, the so-called input chain, that is, when the network adapter finds the destination IP address of the received network packet, consistent with the IP address of the network adapter, put the data packet into the input chain for rule processing. The code here is as follows:

  # Accept packets related to previously established connections  iptables -I INPUT -m state --state ESTABLISHED,RELATED --jump ACCEPT -m comment --comment 'related-traffic'  # Reject traffic from containers to host  if [ "$ALLOW_HOST_ACCESS" != "true" ]; then    iptables -A INPUT -s ${POOL_NETWORK} --jump REJECT -m comment --comment 'block-traffic-to-host'  fi

Set rules in the input chain to ensure that the packets sent from the original external connection on the warden host machine will all adopt the accept operation. Secondly, the allow_host_access variable is used to determine whether to allow communication between the warden iner and the host where the warden server is located. If the variable is false, the reject operation is performed on all packets sent from pool_networks, make sure that the warden container cannot access the host machine of the warden server. At this point, the rules for the input chain have been completed.
6. Define the forward chain in iptables.
Create net. sh chain, that is, when the network finds that the destination IP address of the received network packet is inconsistent with the IP address of the network adapter, it is considered that the forward operation must be performed for the packet, put the data packet into the forward chain for rule processing. The code here is as follows:

  # Forward outbound traffic via ${filter_forward_chain}  iptables -A FORWARD -i w-+ --jump ${filter_forward_chain}  # Forward inbound traffic immediately  default_interface=$(ip route show | grep default | cut -d' ' -f5 | head -1)  iptables -I ${filter_forward_chain} -i $default_interface --jump ACCEPT

The first rule indicates that a device name is sent from a network device interface named 'W-+ '(a regular expression, that is, the device name of the virtual network card virtualized for Warden container, at the same time, the packet that needs to be forwarded will jump to the filter_forward_chain chain. This ensures that all packets sent from Warden iner, not to the NIC of the host machine, enter the forward chain and directly enter the filter_forward_chain chain. The subsequent rules indicate that the data packet in filter_forward_chain will take the accept operation as long as it is from default_interface. That is to say, when the data packet in filter_forward_chain is found to be from the physical Nic of the host where Warden server is located, the accept operation is immediately performed on the data packet.
So far, this net. the filter_setup method in the sh file has been analyzed step by step. Then, readers may still be confused about the entire execution flow, such as when to bind the filter_default_chain chain to enable data packets to enter the chain. Later, after analyzing another net. Sh file, we will discuss it in a unified manner.

Setup_nat
The following is a brief analysis of the setup_nat method:

function setup_nat() {     teardown_nat      # Create prerouting chain     iptables -t nat -N ${nat_prerouting_chain} 2> /dev/null || true      # Bind chain to PREROUTING     (iptables -t nat -S PREROUTING | grep -q "\-j ${nat_prerouting_chain}\b") ||            iptables -t nat -A PREROUTING             --jump ${nat_prerouting_chain}      # Bind chain to OUTPUT (for traffic originating from same host)     (iptables -t nat -S OUTPUT | grep -q "\-j ${nat_prerouting_chain}\b") ||            iptables -t nat -A OUTPUT             --out-interface "lo"             --jump ${nat_prerouting_chain}      # Create postrouting chain      iptables -t nat -N ${nat_postrouting_chain} 2> /dev/null || true      # Bind chain to POSTROUTING      (iptables -t nat -S POSTROUTING | grep -q "\-j ${nat_postrouting_chain}\b") ||            iptables -t nat -A POSTROUTING             --jump ${nat_postrouting_chain}      # Enable NAT for traffic coming from containers      (iptables -t nat -S ${nat_postrouting_chain} | grep -q "\-j SNAT\b") ||            iptables -t nat -A ${nat_postrouting_chain}             --source ${POOL_NETWORK}             --jump SNAT             --to $(external_ip)}

This part is easy to understand:

1. First create the nat_prerouting_chain chain, and then redirect all packets in the prerouting chain to the nat_prerouting_chain;

2. Then, bind the chain to nat_prerouting_chain to ensure that all packets sent through the lo network device are redirected to the chain;

3. Create a nat_postrouting_chain chain to ensure that all data packets in the default postrouting chain are redirected to the created chain;

4. SNAT is used for all data packets sent from Warden container, so that the source IP address of the data packet is replaced with the IP address of the warden server host machine.

In fact, after completing the above two steps, the following code is also executed:

 # Enable forwardingecho 1 > /proc/sys/net/ipv4/ip_forward

To ensure that the forwarding can be effective, you can view the functions of/proc/sys/NET/IPv4.

Create net. Sh for Warden container

In the previous article, we analyzed the iptbales settings for the host machine where the warden server is located when the warden server is started. during the whole process, no iner is involved. This section describes how to set iptables rules for the host machine where the warden server is located when a specific warden container is started. The file address is as follows: Warden/root/Linux/skeleton/net. Sh.

Setup_filter

First, analyze the setup_filter method. The source code is as follows:

function setup_filter() {       teardown_filter       # Create instance chain       iptables -N ${filter_instance_chain}       iptables -A ${filter_instance_chain}            --goto ${filter_default_chain}       # Bind instance chain to forward chain       iptables -I ${filter_forward_chain} 2            --in-interface ${network_host_iface}            --goto ${filter_instance_chain}       # Create instance log chain       iptables -N ${filter_instance_log_chain}       iptables -A ${filter_instance_log_chain}            -p tcp -m conntrack --ctstate NEW,UNTRACKED,INVALID -j LOG --log-prefix "${filter_instance_chain} "       iptables -A ${filter_instance_log_chain}            --jump RETURN}

1. This method creates a chain for the instance, and then jumps all data packets in the chain of the warden container instance to filter_default_chain, that is, the filter_default_chain mentioned in the previous chapter;

2. Bind the forward chain of the warden container instance to the filter_foward chain. That is to say, all data packets entering the filter_foward_chain will enter the corresponding filter_instance_chain according to the network interface device from which the data packets are sent. That is, from the network_host_iface Network Device"Receive"All the packets (network_host_iface is network_container_iface = "w-$ {ID}-1") are transferred to filter_instance_chain.

3. Create a log chain.

Setup_nat

When creating a warden container, Nat is also set. The source code is as follows:

function setup_nat() {      teardown_nat      # Create instance chain      iptables -t nat -N ${nat_instance_chain}      # Bind instance chain to prerouting chain      iptables -t nat -A ${nat_prerouting_chain}              --jump ${nat_instance_chain}}

The brief analysis is: Create the corresponding nat_instance_chain, and then redirect all the packets in nat_prerouting_chain to the nat_instance_chain.

Net_in port ing implementation

This allows the user application instance of warden container in cloud foundry to provide web services for external users. Therefore, the host machine of warden container and warden server performs port ing, that is, the net_in interface provided by Warden server. The actual execution part is the net. sh file.

The following is the source code implementation of net_in:

 "in")       if [ -z "${HOST_PORT:-}" ]; then           echo "Please specify HOST_PORT..." 1>&2           exit 1       fi       if [ -z "${CONTAINER_PORT:-}" ]; then           echo "Please specify CONTAINER_PORT..." 1>&2           exit 1       fi       iptables -t nat -A ${nat_instance_chain}            --protocol tcp            --destination "${external_ip}"            --destination-port "${HOST_PORT}"            --jump DNAT            --to-destination "${network_container_ip}:${CONTAINER_PORT}"        ;;

It can be seen that the most important part of this code block is the part for creating DNAT rule conversion. That is, in the corresponding instance rule chain, If the destination IP address and destination port of the TCP packet are the IP address and the mapped port of the host machine where the warden server is located, execute the DNAT conversion to the warden container IP address and warden container port.

Summary

The above is a brief analysis of iptables rules for warden network design in cloud foundry V2. However, there are a lot of designs that only leave interfaces and are not actually implemented.

In addition, I would like to thank Gao Xianglin, an intern at the vlis lab of Zhejiang University for his joint research.

Indicate the source for reprinting.

This document is more out of my understanding and is certainly flawed and wrong in some places. I hope this article will be helpful to anyone who has access to the warden module in cloud foundry V2. If you are interested in this and have better ideas and suggestions, please contact me.

My mailbox: [email protected]
Sina Weibo: @ lianzifu ruqing