Topic Thread

Expand all | Collapse all

Fortigate 5.4.9 Routing Issue

  • 1.  Fortigate 5.4.9 Routing Issue

     
    Posted 14 days ago
    Edited by Seongho Jeon 14 days ago
    Dear all,
     
    I have a question about the Routing Issue.
    Routing issues were detected yesterday afternoon in a firewall running on version 5.4.9 of the FortiOS.
    Due to current routing issues, there were no service issues, but management access was not possible with each firewall.
     
    [firewall Settings]
    - FortiGate3100D(5.4 Patch9) - HA : Standalone mode, Vdom : enable
    mgmt ip : 192.168.1.6
    - FortiGate101E(5.4 Patch9) - HA : Standalone mode, Vdom : enable
    mgmt ip : 192.168.1.250
     
    [Issue information]
    - management access was not possible with each firewall.
    - As a result of packet capture at the firewall, the SYN packet which tried GUI Access from the administrator's PC(192.168.120.15) was confirmed. 
    However, the firewall does not export SYN + ACK packets.
    - We tested ICMP / SSH / HTTPS etc, but the result was the same.
    - The target of the "exec traceroute" command of the firewall has been specified as the administrator PC. 
    However, the output value was identified as loopback ip with "127.0.0.1".
    e.g) exec trace route 
    exec trace route 192.168.120.15
    traceroute to 192.168.120.15 (192.168.120.15), 32 hops max, 3 probe packets per hot, 84 byte packes
    1 127.0.0.1 <localhost> 2991.668 ms !H 3000.442 ms !H^C *

    - The applied routing table is as follows.
    e.g) config router static
    edit 2
    set dst 192.168.120.0 255.255.255.0
    set gateway 192.168.1.253
    set deice mgmt1
    next
    - And as a result of adding host routing to the routing table, it became normal to communicate.
    e.g) config route static
    edit 3
    set dst 192.168.120.15 255.255.255.255
    set gateway 192.168.1.253
    set device mgmt1
    next
    - Also, after deleting the above "edit 3" routing, access was still possible.

    - At the time of the problem, "rtcache" had the IP "192.168.1.254" instead of the Default GW IP.
     
    ### diagnose ip rtcache list
    family=02 tab=254 vf=0 type=01 tos=0 flag=00040200
    0.0.0.0@0->192.168.120.15@4(mgmt1) gwy=192.168.1.254 prefsrc=192.168.1.6
    ci: ref=0 lastused=274 expire=0 err=00000000 used=2 br=0 pmtu=1500
     
    ### get router info routing-table database
    Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP 
    O - OSPF, IA - OSPF inter area 
    N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 
    E1 - OSPF external type 1, E2 - OSPF external type 2 
    i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area 
    > - selected route, * - FIB route, p - stale info 

    C *> 192.168.1.0/24 is directly connected, mgmt1 
    S *> 192.168.120.0/24 [10/0] via 192.168.1.253, mgmt1

    ### get router info routing-table all
    Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP 
    O - OSPF, IA - OSPF inter area 
    N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 
    E1 - OSPF external type 1, E2 - OSPF external type 2 
    i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area 
    * - candidate default 

    C 192.168.1.0/24 is directly connected, mgmt1 
    S 192.168.120.0/24 [10/0] via 192.168.1.253, mgmt1

    Is the current symptom a bug?


  • 2.  RE: Fortigate 5.4.9 Routing Issue

     
    Posted 14 days ago
    Hello,

    To avoid any confusion and resume :

      * You have two firewall with an interface configured to be used as management service (MGMT if i'm right). Right ?
      * Both firewalls are in same subnet for this management usage. Right ?
      * Issue concerned both firewall and you wasn't able to acces on it from LAN side. Right ?

    Moreover, during your troubleshooting, did you do :

      * diagnose firewall iprope flush / execute router restart ==> during issue and before adding a route ?
      * execute ping-options source [ip of your firewall] and then execute ping [gateway ip]
      * diagnose ip arp list ==> during issue to control if firewall had a mac address linked to your gateway. If you firewall wasn't able to resolve mac address linked to your gateway, trafic will not get out interface.
      * diagnose ip route list : do you have this output during issue and after it ?
      * a check of the other routes including policy routes ? Because i don't believe that firewall learn and use 192.168.1.254 as gateway for pleasure... And diagnose ip rtcache list show active sessions already "established", so there is a route linked to this. Just have to clarify "prefsrc=192.168.1.6"...

    When you add and when you delete a route, you force firewall to update his routing table. But, in case of add, only new session will be concerned by this new route. I think it's for that reason it was working after add. Moreover, in rtcache, we are able to see that it already exist a route to use 192.168.1.254 instead of 192.168.1.253 so all session will use it. And if firewall wasn't able to resolve mac address linked to 192.168.1.253, your route exist but it's not really "up" in the router.

    Currently, i'm not sure that it's a bug. But wait and see...

    Another clue : if you have an access on equipment behind your interface (probably a switch), please check logs too on it. Hope you will find something.

    ------------------------------
    Yohann [LastName] [Designation]
    Ing?nieur syst?me / r?seaux
    [CompanyName]
    [City] [State]
    [Phone]
    ------------------------------



  • 3.  RE: Fortigate 5.4.9 Routing Issue

     
    Posted 13 days ago
    Edited by Seongho Jeon 12 days ago
    Hello. Yohann,

    * You have two firewall with an interface configured to be used as management service (MGMT if i'm right). Right ?
    ===> That's right.
    * Both firewalls are in same subnet for this management usage. Right ?
    ===> That's right.
    * Issue concerned both firewall and you wasn't able to acces on it from LAN side. Right ?
    ===> Yes, the GW IP that we don't know remains in the routing cache. It also forwards the response to the request to 127.0.0.1.

    * diagnose firewall iprope flush / execute router restart
    ==> during issue and before adding a route ?
    ====> The same issue has not occurred so far. You did not perform the command you said when the problem occurred.
    * execute ping-options source [ip of your firewall] and then execute ping [gateway ip]
    ===>No, we tested using the "exec ping" command with no options.

    * diagnose ip arp list ==> during issue to control if firewall had a mac address linked to your gateway. If you firewall wasn't able to resolve mac address linked to your gateway, trafic will not get out interface.
    ===> The following command results were found when the problem occurred:
    ##diagnose ip arp list
    index=4 ifname=mgmt1 192.168.1.253 ac:1f:6b:25:4b:e5 state=00000002 use=1 confirm=1 update=1543 ref=3
    index=4 ifname=mgmt1 192.168.1.249 00:21:5a:ec:79:62 state=00000002 use=1 confirm=1654 update
    index=4 ifname=mgmt1 192.168.1.254 state=00000020 use=27422 confirm=34022 update=27122 ref=1

    * diagnose ip route list : do you have this output during issue and after it ?
    ===> Sadly, there is no "diagnose ip route list" check result in the tac report file.

    * a check of the other routes including policy routes ? Because i don't believe that firewall learn and use 192.168.1.254 as gateway for pleasure... And diagnose ip rtcache list show active sessions already "established", so there is a route linked to this. Just have to clarify "prefsrc=192.168.1.6"...
    ===> My question is the same. We have no routing overlapped with mgt in the static router configuration.
    It also does not use policy routing. But for some reason GW IP was assigned to "rtcahe" as 192.168.1.254?

    When you add and when you delete a route, you force firewall to update his routing table. But, in case of add, only new session will be concerned by this new route. I think it's for that reason it was working after add. Moreover, in rtcache, we are able to see that it already exist a route to use 192.168.1.254 instead of 192.168.1.253 so all session will use it. And if firewall wasn't able to resolve mac address linked to 192.168.1.253, your route exist but it's not really "up" in the router.
    ===> I have the same opinion as you.


    As a result of additional questions to the customer,
    The GW IP of the firewall using the "192.168.1.253" IP is verified to be an SSL-VPN appliance. (PulseSecure SSL-VPN)
    192.168.1.254 is also the GW IP value of the PulseSecure equipment.

    However, 192.168.1.254 is not the actual equipment ip address.
    Finally, 192.168.1.254 is the Dummy setting(Trash setting) of the internal interface of PulseSecure.

    these settings have already been used from the past. But so far no problems have been encountered.
    What is the impact of this setting on our equipment?
    I do not want to think of it as a problem with our firewall ... but I can not understand the "rtcahe" value.

    I need your help and advice.

    Thanks and Best Regards,