Multivan and routing on Mikrotik RouterOS

Introduction

Besides vanity, the depressing frequency of questions on this topic in the relevant groups of the Russian-speaking telegram community prompted me to take up the article. This article is aimed at novice administrators of Mikrotik RouterOS (hereinafter ROS). It considers only a multivan, with an emphasis on routing. As a bonus, there are minimally sufficient settings to ensure safe and convenient operation. Those who are looking for the disclosure of those queues, load balancing, vlanov, bridges, multi-stage in-depth analysis of the condition of the channel and the like - can not waste time and effort reading.

Initial data

As a test subject, a five-port Mikrotik router with ROS version 6.45.3 was selected. It will route traffic between two local area networks (LAN1 and LAN2) and three providers (ISP1, ISP2, ISP3). The channel to ISP1 has a static “gray” address, ISP2 is “white” received via DHCP, ISP3 is “white” with PPPoE authentication. The connection diagram is shown in the figure:







The task is to configure the MTK router based on the scheme so that:

1. Provide automatic switching to the backup provider. The main provider is ISP2, the first reserve is ISP1, the second reserve is ISP3.
2. To organize LAN1 access to the Internet only through ISP1.
3. Provide the ability to route traffic from local networks to the Internet through the selected provider based on the address-list.
4. To provide the possibility of publishing services from a local area network to the Internet (DSTNAT)
5. Configure a firewall filter to provide the minimum sufficient security from the Internet.
6. The router could issue its own traffic through any of the three providers, depending on the selected source address.
7. Provide routing of response packets to the channel from which they came (including LAN).

Comment. We will configure the router “from scratch” in order to guarantee that there are no surprises in the starting configurations “out of the box” changing from version to version. Winbox was chosen as the configuration tool, where the changes will be visually displayed. The settings themselves will be set by commands in the Winbox terminal. The physical connection for configuration is through a direct connection to the Ether5 interface.

A little discussion about what a multivan is, whether it is a problem or cunning clever people weave networks of conspiracies around

An inquisitive and attentive admin, setting up such or such a scheme on his own, suddenly suddenly realizes that it is already working normally. Yes, yes, without these your custom routing tables and other route rules, which are full of most articles on this topic. Check it out?



Can we configure addressing on interfaces and default gateways? Yes:



The address and gateway with distance = 2 and check-gateway = ping were registered on ISP1 .

On ISP2, the default dhcp setting for the client is the distance, respectively, equal to one.

On ISP3 in the pppoe settings of the client with add-default-route = yes set default-route-distance = 3 .



Do not forget to register NAT for the output:



/ ip firewall nat add action = masquerade chain = srcnat out-interface-list = WAN



As a result, users of lokalka seals have fun loading through the main ISP2 provider and there is a reservation of the channel using the check gateway mechanism ^{. See Note 1}



Point 1 of the task is implemented. Where is the multivan with its tags? Not…



Farther. You need to release specific clients from the LAN through ISP1:



/ ip firewall mangle add action = route chain = prerouting dst-address-list =! BOGONS \

passthrough = yes route-dst = 100.66.66.1 src-address-list = Via_ISP1

/ ip firewall mangle add action = route chain = prerouting dst-address-list =! BOGONS \

passthrough = no route-dst = 100.66.66.1 src-address = 192.168.88.0 / 24



Items 2 and 3 of the task are implemented. Tags, stamps, route rules, where are you ?!



Need to give access to your favorite OpenVPN server with the address 172.17.17.17 for clients from the Internet? You are welcome:



/ ip cloud set ddns-enabled = yes



We give the output result to the clients as a feast: “ : put [ip cloud get dns-name] ”



We register port forwarding from the Internet:



/ ip firewall nat add action = dst-nat chain = dstnat dst-port = 1194 \

in-interface-list = WAN protocol = udp to-addresses = 172.17.17.17



Point 4 is ready.



We set up a firewall and other security for point 5, at the same time we are glad that everything is working for users and we are reaching for a container with your favorite drink ...

BUT! The tunnels are still forgotten.



The l2tp client configured according to the googled article has risen to the beloved Dutch VDS? Yes.

l2tp-server with IPsec rose and clients by DNS name from IP Cloud (see above.) cling? Yes.

Leaning back, taking a sip of the drink, we lazily examine items 6 and 7 of the problem. We think - do we need it? Everything works like that (s) ... So if it isn’t necessary, then that’s all. Multivan implemented.



What is a multivan? This is the connection of several Internet channels to one router.



You can’t read the article further, because what can be there besides a show-off of doubtful applicability?



With those who remain, who are interested in points 6 and 7 of the task, and also feel the itch of perfectionism, we dive deeper.



The most important task of implementing multivans is the correct routing of traffic. Namely: no matter in which (or which) ^{See Note 3,} the provider’s channel (s) looks at the default route on our router, it should return the answer exactly to the channel from which the packet came. The task is clear. Where is the problem? Indeed, in a simple local network the task is the same, but no one bothers with additional settings and does not feel trouble. The difference is that any routed node on the Internet is accessible through each of our channels, and not through a strictly specific one, as in a simple LAN. But the “trouble” is that if we received a request for the IP address of ISP3, then in our case the answer will go through the ISP2 channel, since the default gateway is directed there. It will leave and be discarded by the provider as incorrect. We decided on the problem. How to solve it?



The solution is divided into three stages:

1. Preset. At this stage, the basic settings of the router will be set: local network, firewall, address lists, hairpin NAT, etc.
2. Multivan. At this stage, the necessary connections will be marked and sorted according to the routing tables.
3. Connection to ISP. At this stage, the interfaces that provide Internet connection will be configured, routing and the mechanism for reservation of Internet channels will be involved.

Comment. Three different types of connection to the ISP were chosen specifically to show that there is nothing unsolvable in setting up multivans with dynamic addresses and demonstrate one of the solution options.

Important! To switch channels according to the algorithm specified using the cost of distance routes, the check gateway mechanism is used . ^{See note 1}

The scripts given in the article have nothing to do with channel reservation.

1. Preset

1.1. We clear the router configuration with the command:

/system reset-configuration skip-backup=yes no-defaults=yes
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

agree with “ Dangerous! Reset anyway? [y / N]: ”and, after rebooting, we connect to Winbox via MAC. At this stage, the configuration and user base are cleared.



1.2. Create a new user:

 /user add group=full name=knight password=ultrasecret comment=”Not horse”
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

log in under it and delete the default:

 /user remove admin
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. It is the removal and not the disconnection of the default user that the author considers more secure and recommends for use.



1.3. We create basic interface lists for the convenience of operating in a firewall, discovery settings and other MAC servers:

 /interface list add name=WAN comment="For Internet" /interface list add name=LAN comment="For Local Area"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

We sign interfaces with comments

 /interface ethernet set ether1 comment="to ISP1" /interface ethernet set ether2 comment="to ISP2" /interface ethernet set ether3 comment="to ISP3" /interface ethernet set ether4 comment="to LAN1" /interface ethernet set ether5 comment="to LAN2"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

and fill out the interface lists:

 /interface list member add interface=ether1 list=WAN comment=ISP1 /interface list member add interface=ether2 list=WAN comment=ISP2 /interface list member add interface=ether3 list=WAN comment="to ISP3" /interface list member add interface=ether4 list=LAN comment="LAN1" /interface list member add interface=ether5 list=LAN comment="LAN2"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. Writing clear comments is worth the time spent on this plus greatly facilitates Troubleshoot and understanding the configuration.



The author considers it necessary, for security reasons, to add the ether3 interface to the interface list “WAN”, despite the fact that ip protocol will not go through it.



Do not forget that after the PPP interface has been raised on ether3, it will also need to be added to the “WAN” interface list



1.4. We hide the router from proximity detection and control from the networks of providers via MAC:

 /ip neighbor discovery-settings set discover-interface-list=!WAN /tool mac-server set allowed-interface-list=LAN /tool mac-server mac-winbox set allowed-interface-list=LAN
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

1.5. We create a minimum sufficient set of firewall filter rules to protect the router:

 /ip firewall filter add action=accept chain=input \ comment="Related Established Untracked Allow" \ connection-state=established,related,untracked
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

(the rule provides permission for established and related connections that are initiated both from connected networks and by the router itself)

 /ip firewall filter add action=accept chain=input \ comment="ICMP from ALL" protocol=icmp
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

(ping and not only ping. All icmp is allowed to enter. It is very useful for finding problems with MTU)

 /ip firewall filter add action=drop chain=input comment="All other WAN Drop" \ in-interface-list=WAN
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

(the rule closing the input chain forbids everything else that arrives from the Internet)

 /ip firewall filter add action=accept chain=forward \ comment="Established, Related, Untracked allow" \ connection-state=established,related,untracked
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

(the rule allows established and related connections that pass through the router)

 /ip firewall filter add action=drop chain=forward comment="Invalid drop" \ connection-state=invalid
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

(the rule drops connections, with connection-state = invalid, passing through the router. It is highly recommended by Mikrotik, but in some rare situations it can block useful traffic)

 /ip firewall filter add action=drop chain=forward \ comment="Drop all from WAN not DSTNATed" connection-nat-state=!dstnat \ connection-state=new in-interface-list=WAN
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

(the rule prohibits packets that go from the Internet and did not go through the dstnat procedure through the router. This will protect local networks from intruders who, being in the same broadcast domain with our external networks, will register our external IPs as a gateway and, thus, try “Explore” our local area networks.)



Comment. Let us assume that LAN1 and LAN2 are trusted networks and traffic between them and from them is not filtered.



1.6. Create a list with a list of non-routable networks:

 /ip firewall address-list add address=0.0.0.0/8 comment="\"This\" Network" list=BOGONS add address=10.0.0.0/8 comment="Private-Use Networks" list=BOGONS add address=100.64.0.0/10 comment="Shared Address Space. RFC 6598" list=BOGONS add address=127.0.0.0/8 comment=Loopback list=BOGONS add address=169.254.0.0/16 comment="Link Local" list=BOGONS add address=172.16.0.0/12 comment="Private-Use Networks" list=BOGONS add address=192.0.0.0/24 comment="IETF Protocol Assignments" list=BOGONS add address=192.0.2.0/24 comment=TEST-NET-1 list=BOGONS add address=192.168.0.0/16 comment="Private-Use Networks" list=BOGONS add address=198.18.0.0/15 comment="Network Interconnect Device Benchmark Testing"\ list=BOGONS add address=198.51.100.0/24 comment=TEST-NET-2 list=BOGONS add address=203.0.113.0/24 comment=TEST-NET-3 list=BOGONS add address=224.0.0.0/4 comment=Multicast list=BOGONS add address=192.88.99.0/24 comment="6to4 Relay Anycast" list=BOGONS add address=240.0.0.0/4 comment="Reserved for Future Use" list=BOGONS add address=255.255.255.255 comment="Limited Broadcast" list=BOGONS
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

(This is a list of addresses and networks that are not routed to the Internet and, accordingly, we will also follow this.)



Comment. The list may change, so I advise you to periodically check the relevance.



1.7. Configure DNS for the router itself:

 /ip dns set servers=1.1.1.1,8.8.8.8
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. In the current version of ROS, dynamic servers take precedence over statically defined ones. A name resolution request is sent to the first server in the order in the list. The transition to the next server is performed when the current one is unavailable. Big timeout - more than 5 sec. Returning back when the “fallen server” is resumed does not automatically happen. Given this algorithm and the presence of a multivan, the author recommends not using servers issued by providers.



1.8. Configure the local network.

1.8.1. Configure static IP addresses on local network interfaces:

 /ip address add interface=ether4 address=192.168.88.254/24 comment="LAN1 IP" /ip address add interface=ether5 address=172.16.1.0/23 comment="LAN2 IP"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

1.8.2. We set the rules for routes to our local networks through the main routing table:

 /ip route rule add dst-address=192.168.88.0/24 table=main comment=”to LAN1” /ip route rule add dst-address=172.16.0.0/23 table=main comment="to LAN2"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. This is one of the easiest and quickest ways to access local network addresses with external IP addresses of router interfaces through which the default route does not go.



1.8.3. Enable Hairpin NAT for LAN1 and LAN2:

 /ip firewall nat add action=src-nat chain=srcnat comment="Hairpin to LAN1" \ out-interface=ether4 src-address=192.168.88.0/24 to-addresses=192.168.88.254 /ip firewall nat add action=src-nat chain=srcnat comment="Hairpin to LAN2" \ out-interface=ether5 src-address=172.16.0.0/23 to-addresses=172.16.1.0
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. This allows you to access via external IP to your resources (dstnat), while inside the network.

2. Actually, the implementation of the correct multivans

To solve the problem of “answering from where they asked”, we will use two ROS tools: connection mark and routing mark . Connection mark allows you to mark the desired connection and later work with this label as a condition for applying the routing mark . And already with routing mark it is possible to work in ip route and route rules . We figured out the tools, now you have to decide which connections to label - one, where exactly to label - two.



With the first, everything is simple - we must mark all the connections that come to the router from the Internet through the appropriate channel. In our case, these will be three labels (according to the number of channels): “conn_isp1”, “conn_isp2” and “conn_isp3”.



The nuance with the second is that the incoming connections will be of two types: transit and those that are intended for the router itself. The connection mark mechanism works in the mangle table. Consider the movement of the package in a simplified diagram, kindly collected by the specialists of the resource mikrotik-trainings.com (not advertising):







Following the arrows, we see that the packet coming to the “ input interface ” passes through the “ Prerouting ” chain and only then it is divided into transit and local in the “ Routing Decision ” block. Therefore, to kill two birds with one stone, use the Connection Mark in the Mangle Prerouting table of the Prerouting chain.



Remark . In ROS, the “Routing mark” is indicated in the Ip / Routes / Rules section as “Table”, and in the remaining sections as “Routing Mark”. This can cause some confusion in understanding, but, in fact, it is one and the same, and is an analogue of rt_tables in iproute2 on linux.



2.1. We mark incoming connections from each of the providers:

 /ip firewall mangle add action=mark-connection chain=prerouting \ comment="Connmark in from ISP1" connection-mark=no-mark in-interface=ether1 \ new-connection-mark=conn_isp1 passthrough=no /ip firewall mangle add action=mark-connection chain=prerouting \ comment="Connmark in from ISP2" connection-mark=no-mark in-interface=ether2 \ new-connection-mark=conn_isp2 passthrough=no /ip firewall mangle add action=mark-connection chain=prerouting \ comment="Connmark in from ISP3" connection-mark=no-mark in-interface=pppoe-isp3 \ new-connection-mark=conn_isp3 passthrough=no
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. In order not to tag already marked connections, I use the condition connection-mark = no-mark instead of connection-state = new because I think this is more correct, as well as refusing drop invalid connections in the input filter.



passthrough = no - because in this implementation method, re-marking is excluded and to speed up it is possible to interrupt enumeration of rules after the first match.



It should be borne in mind that we still do not interfere with routing. Now there are only stages of preparation. The next stage of implementation will be the processing of transit traffic, which is returned via a steady connection from the addressee in the local network. Those. those packets that (see diagram) went through the router along the way:



“Input Interface” => ”Prerouting” => ”Routing Decision” => ”Forward” => ”Post Routing” => ”Output Interface” and got to their destination on the local network.



Important! In ROS there is no logical division into external and internal interfaces. If we trace the path of the response packet in the diagram below, it will follow the same logical path as the request:



“Input Interface” => ”Prerouting” => ”Routing Decision” => ”Forward” => ”Post Routing” => ”Output Interface” just for the request “ Input Interface ” there was an ISP interface, and for the answer - LAN



2.2. We direct the return transit traffic to the corresponding routing tables:

 /ip firewall mangle add action=mark-routing chain=prerouting \ comment="Routemark transit out via ISP1" connection-mark=conn_isp1 \ dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_isp1 passthrough=no /ip firewall mangle add action=mark-routing chain=prerouting \ comment="Routemark transit out via ISP2" connection-mark=conn_isp2 \ dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_isp2 passthrough=no /ip firewall mangle add action=mark-routing chain=prerouting \ comment="Routemark transit out via ISP3" connection-mark=conn_isp3 \ dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_isp3 passthrough=no
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. in-interface-list =! WAN - we only work with traffic from the local network and dst-address-type =! local without the destination address of the address of the interfaces of the router itself.





The same goes for local packets that came to the router along the way:



“Input Interface” => ”Prerouting” => ”Routing Decision” => ”Input” => ”Local Process”



Important! The answer will go along the following path:



”Local Process” => ”Routing Decision” => ”Output” => ”Post Routing” => ”Output Interface”



2.3. We direct the response local traffic to the corresponding routing tables:

 /ip firewall mangle add action=mark-routing chain=output \ comment="Routemark local out via ISP1" connection-mark=conn_isp1 \ dst-address-type=!local new-routing-mark=to_isp1 passthrough=no /ip firewall mangle add action=mark-routing chain=output \ comment="Routemark local out via ISP2" connection-mark=conn_isp2 \ dst-address-type=!local new-routing-mark=to_isp2 passthrough=no /ip firewall mangle add action=mark-routing chain=output \ comment="Routemark local out via ISP3" connection-mark=conn_isp3 \ dst-address-type=!local new-routing-mark=to_isp3 passthrough=no
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

At this stage, the task of preparing to send a response to the Internet channel from which the request came can be considered resolved. Everything is marked, marked and ready to route.

An excellent “side” effect of this setting is the ability to forward DSNAT ports from both (ISP2, ISP3) providers at the same time. Not at all, because on ISP1 we do not have a routable address. This effect is important, for example, for a mail server with two MXs that look at different Internet channels.



To eliminate the nuances of working local networks with external IP routers, we use the solutions from paragraphs. 1.8.2 and 3.1.2.6.



In addition, you can use the tool with markings and to solve paragraph 3 of the problem. We implement it like this:



2.4. We direct traffic from local clients from routing lists to the corresponding tables:

 /ip firewall mangle add action=mark-routing chain=prerouting \ comment="Address List via ISP1" dst-address-list=!BOGONS new-routing-mark=to_isp1 \ passthrough=no src-address-list=Via_ISP1 /ip firewall mangle add action=mark-routing chain=prerouting \ comment="Address List via ISP2" dst-address-list=!BOGONS new-routing-mark=to_isp2 \ passthrough=no src-address-list=Via_ISP2 /ip firewall mangle add action=mark-routing chain=prerouting \ comment="Address List via ISP3" dst-address-list=!BOGONS new-routing-mark=to_isp3 \ passthrough=no src-address-list=Via_ISP3
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

As a result, it looks something like this:

3. Configure ISP connectivity and enable brand-based routing

3.1. We configure connection to ISP1:

3.1.1. Configure a static IP address:

 /ip address add interface=ether1 address=100.66.66.2/30 comment="ISP1 IP"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

3.1.2. Configure static routing:

3.1.2.1. Add a default emergency route:

 /ip route add comment="Emergency route" distance=254 type=blackhole
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. This route allows traffic from local processes to go through the Route Decision stage regardless of the state of the channels of any of the providers. The nuance of outgoing local traffic is that for the packet to move at least somewhere, the active route to the default gateway must be present in the main routing table. If not, then the package will simply be destroyed.



As an extension of the check gateway tool for a deeper analysis of the channel state, I propose using the recursive route method. The essence of the method is that we tell the router to search for the path to its gateway not directly, but through an intermediate gateway. As such “test” gateways, 4.2.2.1, 4.2.2.2 and 4.2.2.3 will be selected for ISP1, ISP2 and ISP3, respectively.



3.1.2.2. Route to the “verification” address:

 /ip route add check-gateway=ping comment="For recursion via ISP1" \ distance=1 dst-address=4.2.2.1 gateway=100.66.66.1 scope=10
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. The scope value is reduced to default in the ROS target scope, in order to use further 4.2.2.1 as a recursive gateway. I emphasize: the scope of the route to the “verification” address must be less than or equal to the target scope of the route that will reference the verification.



3.1.2.3. The default recursive route for traffic without a routing mark:

 /ip route add comment="Unmarked via ISP1" distance=2 gateway=4.2.2.1
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. The distance = 2 value is used because ISP1 is declared as the first standby one under the conditions of the task.



3.1.2.4. The default recursive route for traffic with routing mark “to_isp1”:

 /ip route add comment="Marked via ISP1 Main" distance=1 gateway=4.2.2.1 \ routing-mark=to_isp1
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. Actually, here we are finally starting to use the fruits of the preparatory work that was carried out in paragraph 2.



On this route, all traffic that has mark route “to_isp1” will be directed to the gateway of the first provider, regardless of which the default gateway for the main table is currently active.



3.1.2.5. The first default fallback recursive route for labeled ISP2 and ISP3 traffic:

 /ip route add comment="Marked via ISP2 Backup1" distance=2 gateway=4.2.2.1 \ routing-mark=to_isp2 /ip route add comment="Marked via ISP3 Backup1" distance=2 gateway=4.2.2.1 \ routing-mark=to_isp3
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. These routes are also needed to reserve traffic from local networks, which are members of the address list “to_isp *” '



3.1.2.6. We prescribe the route for local router traffic to the Internet through ISP1:

 /ip route rule add comment="From ISP1 IP to Inet" src-address=100.66.66.2 table=to_isp1
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. In combination with the rules from clause 1.8.2, an exit to the desired channel with a given source is provided. This is critical for building tunnels in which the IP address of the local side (EoIP, IP-IP, GRE) is specified. Since the rules in ip route rules are executed from top to bottom, until the conditions match for the first time, this rule should be after the rules in paragraph 1.8.2.



3.1.3. We register the NAT rule for outgoing traffic:

 /ip firewall nat add action=src-nat chain=srcnat comment="NAT via ISP1" \ ipsec-policy=out,none out-interface=ether1 to-addresses=100.66.66.2
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. NAT is everything that goes, except that it falls into IPsec policies. I try not to use action = masquerade unless absolutely necessary. It is slower and more resource intensive than src-nat, because it calculates an address for NAT for each new connection.



3.1.4. We send clients from the list who are denied access through other providers directly to the ISP1 provider gateway.

 /ip firewall mangle add action=route chain=prerouting \ comment="Address List via ISP1 only" dst-address-list=!BOGONS passthrough=no \ route-dst=100.66.66.1 src-address-list=Via_only_ISP1 place-before=0
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Comment. action = route has a higher priority and is applied earlier than other routing rules.



place-before = 0 - puts our rule first in the list.



3.2. We configure connection to ISP2.



Since ISP2 provider gives us the settings via DHCP, it is reasonable to make the necessary changes with a script that starts when a DHCP client is triggered:

 /ip dhcp-client add add-default-route=no disabled=no interface=ether2 script=":if (\$bound=1) do={\r\ \n /ip route add check-gateway=ping comment=\"For recursion via ISP2\" \ distance=1 dst-address=4.2.2.2/32 gateway=\$\"gateway-address\" scope=10\r\ \n /ip route add comment=\"Unmarked via ISP2\" distance=1 gateway=4.2.2.2;\r\ \n /ip route add comment=\"Marked via ISP2 Main\" distance=1 gateway=4.2.2.2 \ routing-mark=to_isp2;\r\ \n /ip route add comment=\"Marked via ISP1 Backup1\" distance=2 \ gateway=4.2.2.2 routing-mark=to_isp1;\r\ \n /ip route add comment=\"Marked via ISP3 Backup2\" distance=3 \ gateway=4.2.2.2 routing-mark=to_isp3;\r\ \n /ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \ out-interface=\$\"interface\" to-addresses=\$\"lease-address\" \ comment=\"NAT via ISP2\" place-before=1;\r\ \n if ([/ip route rule find comment=\"From ISP2 IP to Inet\"] =\"\") do={\r\ \n /ip route rule add comment=\"From ISP2 IP to Inet\" \ src-address=\$\"lease-address\" table=to_isp2 \r\ \n } else={\r\ \n /ip route rule set [find comment=\"From ISP2 IP to Inet\"] disabled=no \ src-address=\$\"lease-address\"\r\ \n } \r\ \n} else={\r\ \n /ip firewall nat remove [find comment=\"NAT via ISP2\"];\r\ \n /ip route remove [find comment=\"For recursion via ISP2\"];\r\ \n /ip route remove [find comment=\"Unmarked via ISP2\"];\r\ \n /ip route remove [find comment=\"Marked via ISP2 Main\"];\r\ \n /ip route remove [find comment=\"Marked via ISP1 Backup1\"];\r\ \n /ip route remove [find comment=\"Marked via ISP3 Backup2\"];\r\ \n /ip route rule set [find comment=\"From ISP2 IP to Inet\"] disabled=yes\r\ \n}\r\ \n" use-peer-dns=no use-peer-ntp=no
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The script itself in the Winbox window:





Comment. The first part of the script is triggered when the lease is successfully received, the second - after the lease is released. ^{See note 2}



3.3. We configure the connection to the ISP3 provider.



Since the configuration provider gives us dynamic, it is reasonable to make the necessary changes with scripts that start after the rise and after the fall of the ppp interface.



Comment. The ppp interface can be specified as a gateway instead of an IP address. However, in this case, the recursive route cannot be activated. Therefore, we get the IP address of the provider side into the variable and use it further in the same way as for the other providers



3.3.1. First, configure the profile:

 /ppp profile add comment="for PPPoE to ISP3" interface-list=WAN name=isp3_client \ on-down="/ip firewall nat remove [find comment=\"NAT via ISP3\"];\r\ \n/ip route remove [find comment=\"For recursion via ISP3\"];\r\ \n/ip route remove [find comment=\"Unmarked via ISP3\"];\r\ \n/ip route remove [find comment=\"Marked via ISP3 Main\"];\r\ \n/ip route remove [find comment=\"Marked via ISP1 Backup2\"];\r\ \n/ip route remove [find comment=\"Marked via ISP2 Backup2\"];\r\ \n/ip route rule set [find comment=\"From ISP3 IP to Inet\"] disabled=yes;" \ on-up="/ip route add check-gateway=ping comment=\"For recursion via ISP3\" distance=1 \ dst-address=4.2.2.3/32 gateway=\$\"remote-address\" scope=10\r\ \n/ip route add comment=\"Unmarked via ISP3\" distance=3 gateway=4.2.2.3;\r\ \n/ip route add comment=\"Marked via ISP3 Main\" distance=1 gateway=4.2.2.3 \ routing-mark=to_isp3;\r\ \n/ip route add comment=\"Marked via ISP1 Backup2\" distance=3 gateway=4.2.2.3 \ routing-mark=to_isp1;\r\ \n/ip route add comment=\"Marked via ISP2 Backup2\" distance=3 gateway=4.2.2.3 \ routing-mark=to_isp2;\r\ \n/ip firewall mangle set [find comment=\"Connmark in from ISP3\"] \ in-interface=\$\"interface\";\r\ \n/ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \ out-interface=\$\"interface\" to-addresses=\$\"local-address\" \ comment=\"NAT via ISP3\" place-before=1;\r\ \nif ([/ip route rule find comment=\"From ISP3 IP to Inet\"] =\"\") do={\r\ \n /ip route rule add comment=\"From ISP3 IP to Inet\" \ src-address=\$\"local-address\" table=to_isp3 \r\ \n} else={\r\ \n /ip route rule set [find comment=\"From ISP3 IP to Inet\"] disabled=no \ src-address=\$\"local-address\"\r\ \n};\r\ \n"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The script itself in the Winbox window:





Comment. Line

/ ip firewall mangle set [find comment = "Connmark in from ISP3"] in-interface = $ "interface";

Allows you to correctly process the renaming of the interface, because it works with its code and not the display name.



3.3.2. Now, using the profile, create a ppp connection:

 /interface pppoe-client add allow=mschap2 comment="to ISP3" disabled=no \ interface=ether3 name=pppoe-isp3 password=isp3_pass profile=isp3_client \ user=isp3_client
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

As a final touch, set the clock:

 /system ntp client set enabled=yes \ server-dns-names=0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

For those who have read to the end

The proposed method for implementing multivans is the author’s personal preference and is not the only possible one. The ROS toolkit is extensive and flexible, which on the one hand causes difficulties for beginners, on the other - the reason for popularity. Explore, try, discover new tools and solutions. For example, as an application of the knowledge gained, it is possible in this implementation of the multivan to replace the check-gateway tool with recursive routes with Netwatch .

Notes

1. Check-gateway - a mechanism that allows you to deactivate a route after two consecutive unsuccessful checks of the gateway for availability. The check is carried out every 10 seconds, plus a response timeout. Total, the actual switching timing lies in the range of 20-30 seconds. If such switching timing is not sufficient, there is an option to use the Netwatch tool , where the check timer can be set manually.
   
   The Check-gateway mechanism does not work with periodic packet loss in the channel.
   
   
   
   Important! Deactivating the main route entails deactivating all other routes that refer to it. Therefore, there is no need to specify check-gateway = ping for them .
2. It happens that a failure occurs in the DHP operation mechanism, which looks like a client hanging in a renew state. In this case, the second part of the script will not work, but it will not hurt the traffic to walk correctly, since the state monitors the corresponding recursive route.
3. ECMP (Equal Cost Multi-Path) - ROS has the ability to specify a route with multiple gateways and the same distance. In this case, the connections will be distributed over the channels using the round robin algorithm, in proportion to the number of gateways specified.

For the impetus to writing the article, help in shaping its structure and placing emphasis - personal thanks to Eugene @jscar