openstack 负载均衡组件负载均衡 openwrt

转载

代码探险家 2023-08-08 16:28:17

文章标签 openstack 负载均衡组件 ci IP Code 文章分类 OpenStack 云计算

2021年11月更新：如果是K2P A2版本硬件，请参考更简易的步骤，更加高效，不需要太多代码知识。 K2P刷机-坐标北京Unicom

前题

硬件：路由器，刷入OpenWrt
一些背景知识和动手能力

目标效果图

openstack 负载均衡组件负载均衡 openwrt_openstack 负载均衡组件

openstack 负载均衡组件负载均衡 openwrt_ci_02

步骤

使用SSH 登陆路由器。I.e. ssh root@192.168.2.1
运行/usr/bin/duobo。日志类似：

sh: 2: unknown operand
Killed
Warning: Unable to locate ipset utility, disabling ipset support
 * Flushing IPv4 filter table
 * Flushing IPv4 nat table
 * Flushing IPv4 mangle table
 * Flushing IPv4 raw table
 * Flushing conntrack table ...
 * Populating IPv4 filter table
   * Zone 'lan'
   * Zone 'wan'
   * Rule 'Allow-DHCP-Renew'
   * Rule 'Allow-Ping'
   * Rule '51413'
   * Rule '9091'
   * Rule '9000'
   * Rule '6800'
   * Forward 'lan' -> 'wan'
 * Populating IPv4 nat table
   * Zone 'lan'
   * Zone 'wan'
 * Populating IPv4 mangle table
   * Zone 'lan'
   * Zone 'wan'
 * Populating IPv4 raw table
   * Zone 'lan'
   * Zone 'wan'
 * Set tcp_ecn to off
 * Set tcp_syncookies to on
 * Set tcp_window_scaling to on
 * Running script '/etc/firewall.user'
 * Running script '/usr/share/miniupnpd/firewall.include'
 * Running script '/usr/share/qos_gargoyle/firewall.include'
   ! Failed with exit code 1

___________________________________________________
开始第1次拔号...........
正在并发拔号中.............
等待3秒.............
[3]拔[0]拔成功, 小于设定的[2]拔，将重新拔号...

_________________________

View Code
如果没有成功并发多播，根据提示信息调整duobo的内部参数设置，重新运行。分享一些技巧：

number=10 #number是重拔次数，本脚本启动后一共尝试的次数
n=7 #n是几拔，同时发出几拨，理论上设置越大成功概率越大！
ok=2 #ok是拔上几次后退出拔号, 要实现的预期目标
wait=5 #wait time 每次单线多拨失败后，重试的等待时间

原则：失败多次以后，建议冷重启路由器，多个运行中的脚本会互相冲突，把问题复杂化。
多拨失败以后，耐心等上1分钟；很多次的多播成功都是脚本完成以后退出，然后60秒内获取了超过预期的多条链接。
把n调到7，这样会虚拟出很多wan接口，根据硬件性能，理论上设置越大成功概率越大！
将ok调低一点，比如2。我觉得别太贪心，如果能稳定多播2个ADSL的IP，那其实概率上更折中了稳定性和高负载下的带宽均衡。
wait的时间我感觉5秒－8秒差不多，这个要看运行日志，做相应调整。
看懂程序的原理，根据实际情况调整脚本程序。比如：这个版本的程序是这么多播的，ifup $prefix$i

遗留问题

openstack 负载均衡组件负载均衡 openwrt_ci_03

openstack 负载均衡组件负载均衡 openwrt_openstack 负载均衡组件_04

1 开始第3次拔号...........
 2 正在并发拔号中.............
 3 等待10秒.............
 4 pppoe-wan Link encap:Point-to-Point Protocol  
 5 pppoe-wan3 Link encap:Point-to-Point Protocol  
 6 [3]拔[2]拔成功, 大于或等于设定的[2]拨，退出拔号...
 7 Error: an inet address is expected rather than "dev".
 8 iptables v1.4.10: Couldn't load target `zone_wan_notrack':File not found
 9 
10 Try `iptables -h' or 'iptables --help' for more information.
11 iptables v1.4.10: Couldn't load target `zone_wan_nat':File not found
12 
13 Try `iptables -h' or 'iptables --help' for more information.
14 iptables: No chain/target/match by that name.
15 iptables v1.4.10: Couldn't load target `zone_wan':File not found
16 
17 Try `iptables -h' or 'iptables --help' for more information.
18 iptables: No chain/target/match by that name.
19 iptables: No chain/target/match by that name.
20 iptables: No chain/target/match by that name.
21 iptables: No chain/target/match by that name.
22 iptables: No chain/target/match by that name.
23 iptables: No chain/target/match by that name.
24 iptables: No chain/target/match by that name.
25 Error: an inet address is expected rather than "dev".
26 iptables v1.4.10: Couldn't load target `zone_wan_notrack':File not found
27 
28 Try `iptables -h' or 'iptables --help' for more information.
29 iptables v1.4.10: Couldn't load target `zone_wan_nat':File not found
30 
31 Try `iptables -h' or 'iptables --help' for more information.
32 iptables: No chain/target/match by that name.
33 iptables v1.4.10: Couldn't load target `zone_wan':File not found
34 
35 Try `iptables -h' or 'iptables --help' for more information.
36 iptables: No chain/target/match by that name.
37 iptables: No chain/target/match by that name.
38 iptables: No chain/target/match by that name.
39 iptables: No chain/target/match by that name.
40 iptables: No chain/target/match by that name.
41 iptables: No chain/target/match by that name.
42 iptables: No chain/target/match by that name.
43 Error: an IP address is expected rather than "dev"
44 192.168.2.0/24 dev br-lan  proto kernel  scope link  src 192.168.2.1 
45 221.218.232.1 dev pppoe-wan3  proto kernel  scope link  src 123.112.248.148 
46 221.218.232.1 dev pppoe-wan  proto kernel  scope link  src 221.218.236.51 
47 Kernel IP routing table
48 Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
49 192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 br-lan
50 221.218.232.1   0.0.0.0         255.255.255.255 UH    0      0        0 pppoe-wan3
51 221.218.232.1   0.0.0.0         255.255.255.255 UH    0      0        0 pppoe-wan

View Error Code

显然这个0.3版本的脚本不完全适用于我的路由器，有时间我需要查查原因。

估计都是关于iptables

有没有大侠探讨一下？

修正了下面版本中的一个错误

line 19：if [ $j -ge $ok ] ;

多拨代码(V0.3)

openstack 负载均衡组件负载均衡 openwrt_ci_03

openstack 负载均衡组件负载均衡 openwrt_openstack 负载均衡组件_04

1 #!/bin/sh
  2 #modified by muziling v0.3
  3 #并发多拨脚本
  4 
  5 #number是重拔次数
  6 #n是几拔
  7 #ok是拔上几次后退出拔号
  8 #wait time
  9 
 10 number=10
 11 n=3
 12 ok=2
 13 wait=5
 14 # avoid same with feixiang's N-WAN naming and must start with "wan"
 15 prefix=wan
 16 vthprefix=vth
 17 
 18 j=$(ifconfig | grep pppoe-wan | wc -l)
 19 if [ "$j" >= "$ok" ] ;
 20 then
 21     echo 已经是[$j]拔了，退出拔号...
 22     exit 0
 23 fi
 24 
 25 if [ -f /etc/config/nwannumset ] ;
 26 then
 27     uci set nwannumset.@macvlan_numset[0].macvlan_num=1
 28     uci commit nwannumset
 29 fi
 30 for i in $( seq 1 $(($n-1)))
 31 do
 32     ifname=$prefix$i
 33     ifvth=$vthprefix$i
 34     #ifwan=$(uci get network.wan.ifname)
 35     pppoe_name=$(uci get network.wan.username) 
 36     pppoe_pw=$(uci get network.wan.password) 
 37 
 38     if [ $(ip link | grep " ${ifvth}@eth0.2:" | wc -l) == "0" ] ;
 39     then
 40         macfac=$(ifconfig | grep eth0.2 | tr -s " " | cut -d " " -f5 | cut -b 1-8)
 41         mac="$macfac:"$(md5sum /proc/sys/kernel/random/uuid | sed 's/\(..\)/&:/g' | cut -b 1-8 | tr [a-f] [A-F])
 42         ip link add link eth0.2 $ifvth type macvlan
 43         ifconfig $ifvth hw ether $mac
 44     fi
 45 
 46     # add /etc/config/network
 47     uci delete network.$ifname
 48     uci set network.$ifname=interface
 49     uci set network.$ifname.ifname=$ifvth
 50     #uci set network.$ifname._orig_ifname=eth0.2
 51     #uci set network.$ifname._orig_bridge=false
 52     uci set network.$ifname.proto=pppoe
 53     uci set network.$ifname.username=$pppoe_name
 54     uci set network.$ifname.password=$pppoe_pw
 55     uci set network.$ifname.auto=0
 56     uci set network.$ifname.defaultroute=0
 57     uci set network.$ifname.peerdns=1
 58     uci set network.$ifname.pppd_options="plugin rp-pppoe.so syncppp $n"
 59 
 60     # add /etc/config/dhcp
 61     uci delete dhcp.$ifvth
 62     uci set dhcp.$ifvth=dhcp 
 63     uci set dhcp.$ifvth.interface=$ifname
 64     uci set dhcp.$ifvth.ignore=1 
 65 
 66     if [ -f /etc/config/nwan ] ;
 67     then
 68         uci delete nwan.$ifname
 69         uci set nwan.$ifname=interface 
 70         uci set nwan.$ifname.name=telecom 
 71         uci set nwan.$ifname.route=balance 
 72         uci set nwan.$ifname.weight=1 
 73         uci set nwan.$ifname.uptime=0day,0hour,0min
 74         uci commit nwan
 75     fi
 76 done
 77 uci set network.wan.defaultroute=0
 78 uci set network.wan.peerdns=1
 79 uci set network.wan.pppd_options="plugin rp-pppoe.so syncppp $n"
 80 uci commit network
 81 uci commit dhcp
 82 
 83 fw_wan_list=$(uci show network |grep =interface |grep -v lan|grep -v loopback |cut -d"." -f2 | awk -F "=" '{printf $1" "}')
 84 uci set firewall.@zone[1].network="$fw_wan_list"
 85 uci commit firewall
 86 /etc/init.d/firewall restart
 87 
 88 for q in $( seq 1 $number )
 89 do
 90     echo
 91     echo ___________________________________________________
 92     echo 开始第$q次拔号...........
 93     killall -q -SIG pppd
 94     if [ "$q" == "1" ] ;
 95     then
 96         for i in $( seq 1 $(($n-1)))
 97         do
 98             ifup $prefix$i
 99         done
100     fi
101 
102     echo 正在并发拔号中.............
103     echo 等待$wait秒.............
104     sleep $wait
105 
106     j=$(ps | grep pppd | wc -l)
107     ! [ "$j" -ge "$n" ]  && ifup ${prefix}1
108 
109     ifconfig|grep pppoe
110     j=$(ifconfig | grep pppoe-wan | wc -l)
111 
112     ! [ "$j" -ge "$ok" ] && echo [$n]拔[$j]拔成功, 小于设定的[$ok]拔，将重新拔号...
113     [ "$j" -ge "$ok" ] && echo [$n]拔[$j]拔成功, 大于或等于设定的[$ok]拨，退出拔号...
114 
115     if [ "$j" -ge "$ok" ] ;
116     then
117         for i in $( seq 0 $(($n-1)))
118         do
119             if [ "$i" == "0" ] ;
120             then
121                 interface=wan
122             else
123                 interface=$prefix$i
124             fi
125             if [ $(ifconfig | grep "pppoe-$interface " | wc -l) == "0" ] ;
126             then
127                 ifdown $interface
128             fi
129         done
130         break
131     fi
132 done
133 # kill ddns sleep and re-check wan ip change
134 killall sleep
135 
136 j=$(ifconfig | grep pppoe-wan | wc -l)
137 ! [ "$j" -ge 0 ] && reboot
138 
139 #ppoename=$(ifconfig |grep 'ppoe-' |awk '{print substr($1,7)}'|tr '\n' ' ')
140 ppoename=$(ifconfig |grep 'ppoe-' |awk '{print $1}'|tr '\n' ' ')
141 i=0
142 vias=""
143 for wan_ifname in $ppoename
144 do
145     vias="$vias nexthop via $wan_ip dev $wan_ifname weight 1 "
146     let "rt=100+$i"
147     i=$(($i+1))
148     ip route flush table $rt
149     ip route add default via $wan_ip dev $wan_ifname table $rt
150     ip route add table $rt to $(ip route | grep br-lan)
151 
152     if [ $(iptables -t nat -vxnL POSTROUTING | grep -c " $wan_ifname ") == "0" ] ;
153     then
154         iptables -t raw -A PREROUTING -i $wan_ifname -j zone_wan_notrack
155         iptables -t nat -A PREROUTING -i $wan_ifname -j zone_wan_prerouting
156         iptables -t nat -A POSTROUTING -o $wan_ifname -j zone_wan_nat
157         iptables -t filter -A forward -i $wan_ifname -j zone_wan_forward
158         iptables -t filter -A input -i $wan_ifname -j zone_wan
159         iptables -t filter -A zone_wan_ACCEPT -o $wan_ifname -j ACCEPT
160         iptables -t filter -A zone_wan_ACCEPT -i $wan_ifname -j ACCEPT
161         iptables -t filter -A zone_wan_DROP -o $wan_ifname -j DROP
162         iptables -t filter -A zone_wan_DROP -i $wan_ifname -j DROP
163         iptables -t filter -A zone_wan_REJECT -o $wan_ifname -j reject
164         iptables -t filter -A zone_wan_REJECT -i $wan_ifname -j reject
165     fi
166     iptables -A PREROUTING -t mangle -i $wan_ifname -j MARK --set-mark $rt
167     iptables -t mangle -A zone_wan_MSSFIX -o $wan_ifname -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
168     ip rule add fwmark $rt table $rt prio $rt
169 done
170 ip route del default
171 ip route add default scope global $vias
172 ip route flush cache
173 ip route list
174 route -n
175 root@OpenWrt:~#

View Code (duobo)

修订版（V0.3.1）

openstack 负载均衡组件负载均衡 openwrt_ci_03

openstack 负载均衡组件负载均衡 openwrt_openstack 负载均衡组件_04

1 #!/bin/sh
  2 #modified by muziling v0.3
  3 #并发多拨脚本
  4 #modified by carl v0.3.1 (修正一处bug，改善友好提示信息)
  5 
  6 #number是重拔次数，本脚本启动后一共尝试的次数
  7 #n是几拔，同时发出几拨，理论上设置越大成功概率越大！
  8 #ok是拔上几次后退出拔号, 要实现的预期目标
  9 #wait time 每次单线多拨失败后，重试的等待时间
 10 
 11 number=15
 12 n=7
 13 ok=2
 14 wait=8
 15 # avoid same with feixiang's N-WAN naming and must start with "wan"
 16 prefix=wan
 17 vthprefix=vth
 18 
 19 j=$(ifconfig | grep pppoe-wan | wc -l)
 20 if [ $j -ge $ok ] ;
 21 then
 22     echo 已经是[$j]拔了，退出拔号程序。
 23     exit 0
 24 fi
 25 
 26 if [ -f /etc/config/nwannumset ] ;
 27 then
 28     uci set nwannumset.@macvlan_numset[0].macvlan_num=1
 29     uci commit nwannumset
 30 fi
 31 
 32 for i in $( seq 1 $(($n-1)))
 33 do
 34     ifname=$prefix$i
 35     ifvth=$vthprefix$i
 36     #ifwan=$(uci get network.wan.ifname)
 37     pppoe_name=$(uci get network.wan.username) 
 38     pppoe_pw=$(uci get network.wan.password) 
 39 
 40     if [ $(ip link | grep " ${ifvth}@eth0.2:" | wc -l) == "0" ] ;
 41     then
 42         macfac=$(ifconfig | grep eth0.2 | tr -s " " | cut -d " " -f5 | cut -b 1-8)
 43         mac="$macfac:"$(md5sum /proc/sys/kernel/random/uuid | sed 's/\(..\)/&:/g' | cut -b 1-8 | tr [a-f] [A-F])
 44         ip link add link eth0.2 $ifvth type macvlan
 45         ifconfig $ifvth hw ether $mac
 46         echo 更换MAC完毕$ifvth.
 47     fi
 48 
 49     # add /etc/config/network
 50     uci delete network.$ifname
 51     uci set network.$ifname=interface
 52     uci set network.$ifname.ifname=$ifvth
 53     #uci set network.$ifname._orig_ifname=eth0.2
 54     #uci set network.$ifname._orig_bridge=false
 55     uci set network.$ifname.proto=pppoe
 56     uci set network.$ifname.username=$pppoe_name
 57     uci set network.$ifname.password=$pppoe_pw
 58     uci set network.$ifname.auto=0
 59     uci set network.$ifname.defaultroute=0
 60     uci set network.$ifname.peerdns=1
 61     uci set network.$ifname.pppd_options="plugin rp-pppoe.so syncppp $n"
 62 
 63     # add /etc/config/dhcp
 64     uci delete dhcp.$ifvth
 65     uci set dhcp.$ifvth=dhcp 
 66     uci set dhcp.$ifvth.interface=$ifname
 67     uci set dhcp.$ifvth.ignore=1 
 68 
 69     if [ -f /etc/config/nwan ] ;
 70     then
 71         uci delete nwan.$ifname
 72         uci set nwan.$ifname=interface 
 73         uci set nwan.$ifname.name=unicom 
 74         uci set nwan.$ifname.route=balance 
 75         uci set nwan.$ifname.weight=1 
 76         uci set nwan.$ifname.uptime=0day,0hour,0min
 77         uci commit nwan
 78     fi
 79 done
 80 
 81 uci set network.wan.defaultroute=0
 82 uci set network.wan.peerdns=1
 83 uci set network.wan.pppd_options="plugin rp-pppoe.so syncppp $n"
 84 uci commit network
 85 uci commit dhcp
 86 
 87 fw_wan_list=$(uci show network |grep =interface |grep -v lan|grep -v loopback |cut -d"." -f2 | awk -F "=" '{printf $1" "}')
 88 uci set firewall.@zone[1].network="$fw_wan_list"
 89 uci commit firewall
 90 /etc/init.d/firewall restart
 91 
 92 for q in $( seq 1 $number )
 93 do
 94     echo
 95     echo ___________________________________________________
 96     echo 开始第$q次拔号...........
 97     killall -q -SIG pppd
 98     if [ "$q" == "1" ] ;
 99     then
100         for i in $( seq 1 $(($n-1)))
101         do
102             ifup $prefix$i
103         done
104     fi
105 
106     echo 正在并发拔号中.............
107     echo 等待$wait秒.............
108     sleep $wait
109 
110     j=$(ps | grep pppd | wc -l)
111     ! [ "$j" -ge "$n" ]  && ifup ${prefix}1
112 
113     ifconfig | grep pppoe
114     j=$(ifconfig | grep pppoe-wan | wc -l)
115 
116     ! [ "$j" -ge "$ok" ] && echo [$n]拔[$j]拔成功, 小于设定的[$ok]拔，将重新拔号...
117     [ "$j" -ge "$ok" ] && echo [$n]拔[$j]拔成功, 大于或等于设定的[$ok]拨，退出拔号...
118     
119     if [ "$j" -ge "$ok" ] ;
120     then
121         for i in $( seq 0 $(($n-1)))
122         do
123             if [ "$i" == "0" ] ;
124             then
125                 interface=wan
126             else
127                 interface=$prefix$i
128             fi
129             if [ $(ifconfig | grep "pppoe-$interface " | wc -l) == "0" ] ;
130             then
131                 ifdown $interface
132             fi
133         done
134         break
135     fi
136 done # done/tried all tring times $number
137 
138 # kill ddns sleep and re-check wan ip change
139 killall sleep
140 
141 # reboot the machine if failed tried times
142 #sleep $wait
143 j=$(ifconfig | grep pppoe-wan | wc -l)
144 ! [ "$j" -gt 0 ] && reboot
145 
146 
147 echo ___________________________________________________
148 echo 开始N-WAN负载均衡功能...
149 #ppoename=$(ifconfig |grep 'ppoe-' |awk '{print substr($1,7)}'|tr '\n' ' ')
150 ppoename=$(ifconfig|grep 'ppoe-' |awk '{print $1}'|tr '\n' ' ')
151 i=0
152 vias=""
153 for wan_ifname in $ppoename
154 do
155     vias="$vias nexthop via $wan_ip dev $wan_ifname weight 1 "
156     let "rt=100+$i"
157     i=$(($i+1))
158     ip route flush table $rt
159     #REMOVE ERROR
160     ip route add default via $wan_ip dev $wan_ifname table $rt
161     ip route add table $rt to $(ip route | grep br-lan)
162 
163     if [ $(iptables -t nat -vxnL POSTROUTING | grep -c " $wan_ifname ") == "0" ] ;
164     then
165         #REMOVE ERROR
166         iptables -t raw -A PREROUTING -i $wan_ifname -j zone_wan_notrack
167         iptables -t nat -A PREROUTING -i $wan_ifname -j zone_wan_prerouting
168         #REMOVE ERROR
169         iptables -t nat -A POSTROUTING -o $wan_ifname -j zone_wan_nat
170         iptables -t filter -A forward -i $wan_ifname -j zone_wan_forward
171         #REMOVE ERROR
172         iptables -t filter -A input -i $wan_ifname -j zone_wan
173         iptables -t filter -A zone_wan_ACCEPT -o $wan_ifname -j ACCEPT
174         iptables -t filter -A zone_wan_ACCEPT -i $wan_ifname -j ACCEPT
175         iptables -t filter -A zone_wan_DROP -o $wan_ifname -j DROP
176         iptables -t filter -A zone_wan_DROP -i $wan_ifname -j DROP
177         iptables -t filter -A zone_wan_REJECT -o $wan_ifname -j reject
178         iptables -t filter -A zone_wan_REJECT -i $wan_ifname -j reject
179     fi
180     
181     iptables -A PREROUTING -t mangle -i $wan_ifname -j MARK --set-mark $rt
182     iptables -t mangle -A zone_wan_MSSFIX -o $wan_ifname -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
183     ip rule add fwmark $rt table $rt prio $rt
184 done
185 
186 ip route del default
187 
188 echo ___________________________________________________
189 echo 下面执行ip route add default scope global $vias
190 ip route add default scope global $vias
191 
192 ip route flush cache
193 
194 echo ___________________________________________________
195 echo 下面输出ip route list
196 ip route list
197 
198 echo ___________________________________________________
199 echo 下面输出route -n
200 route -n

View Code

参考文献

OpenWrt For AR71xx系列 ar2 Tr 脱机 N-WAN r48549

自己动手 4530R 脱机 Samba U-BOOT 多拨（11-04更新部分问题说明，请看2楼）

[0916更新]WR703N WR720N 及其他各类 OpenWRT类路由实现一号多拨带宽叠加教程

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。