Debugging a WAAS Optimized HTTP Connection (301 Redirect Loop)

Sometimes optimized connection thorough a Cisco WAAS system do not work as expected. In this example an user cannot authenticate to a Joomla Web Application. User reported a loop of HTTP 301 (redirect). A couple of WAVE was deployed between client and server.

Analyzing the optimized traffic

Using TCPDUMP/Wireshark connection between client and server can be analyzed (only relevant information is showed).

The client initiates a connection to the server, requesting the login page:

GET /administrator/ HTTP/1.1
Host: 10.0.0.4

The server responds with the (compressed) login page:

HTTP/1.1 200 OK
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Wed, 22 Jan 2014 09:30:51 GMT
[LOGIN PAGE]

User provides username and password, and client sends them to the server:

POST /administrator/index.php HTTP/1.1
Host: 10.0.0.4
username=username&passwd=password

The Cisco WAVE responds with a redirect, no information is sent to the server:

HTTP/1.1 301 Moved Permanently
Location: http://10.0.0.4/administrator/index.php
Age: 148
X-Cisco: WAE

The client follows the redirection and request again the login page:

GET /administrator/index.php HTTP/1.1
Host: 10.0.0.4

Another redirect is generated by the Cisco WAVE, and the user falls in a redirection loop:

HTTP/1.1 301 Moved Permanently
Location: http://10.0.0.4/administrator/index.php
Age: 148
X-Cisco: WAE
[LOOP]

The Cisco WAVE shows that connection between client and server is optimized:

wave#show statistics connection server-ip 10.0.0.4 detail

Connection Id:            469263
    Peer Id:                  88:5a:92:f6:da:81
    Connection Type:          EXTERNAL CLIENT
    Start Time:               Wed Jan 22 10:35:05 2014
    Source IP Address:        10.1.0.3
    Source Port Number:       58793
    Destination IP Address:   10.0.0.4
    Destination Port Number:  80
    Application Name:         Web
    Classifier Name:          HTTP
    Map Name:                 WAAS-GLOBAL
    Directed Mode:            FALSE
    Preposition Flow:         FALSE
    Policy Details:
           Configured:        TCP_OPTIMIZE + DRE + LZ
              Derived:        TCP_OPTIMIZE + DRE + LZ
                 Peer:        TCP_OPTIMIZE + DRE + LZ
           Negotiated:        TCP_OPTIMIZE + DRE + LZ
              Applied:        TCP_OPTIMIZE + DRE + LZ
    Accelerator Details:
                Configured:   HTTP
                   Derived:   HTTP
                   Applied:   HTTP
                      Hist:   None

Moreover lot of pages are cached within the nearest Cisco WAVE:

wave#show cache http-metadatacache all | i 10.0.0.4
URL: http://10.0.0.4/administrator/index.php, Expiration (sec): 84573
URL: http://10.0.0.4/administrator/templates/khepri/images/j_corner_bl.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_login_lock.jpg, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_corner_br.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_bottom.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_crn_br_light.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_crn_bl_light.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_button1_next.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_button1_left.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_crn_tl_light.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_crn_tr_light.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/j_border.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/h_teal/j_header_left.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/h_teal/j_header_right.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/images/h_teal/j_header_middle.png, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/css/general.css, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/css/rounded.css, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/css/login.css, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/system/css/system.css, Expiration (sec): 84605
URL: http://10.0.0.4/administrator/templates/khepri/favicon.ico, Expiration (sec): 84605
URL: http://10.0.0.4/intranet/media/system/js/mootools.js, Expiration (sec): 84605

Disabling the WAAS and analyzing the real traffic

The next step requires to disable the traffic optimization between client and server on one of the Cisco WAVEs:

ip access-list extended Filter
 deny ip any host 10.0.0.4
 deny ip host 10.0.0.4 any
 permit ip any any
 exit
interception access-list Filter

Now traffic between client and server is excluded by optimization:

wave#show statistics connection server-ip 10.0.0.4

Current Active Optimized Flows:                      3776
   Current Active Optimized TCP Plus Flows:          2261
   Current Active Optimized TCP Only Flows:          1515
   Current Active Optimized TCP Preposition Flows:   0
Current Active Auto-Discovery Flows:                 89
Current Reserved Flows:                              102
Current Active Pass-Through Flows:                   490
Historical Flows:                                    582

Local IP:Port         Remote IP:Port        Peer ID           ConnType
10.1.0.3:62979     10.0.0.4:80        N/A               PT Interception ACL

User can now authenticate to the Web Application, and traffic analysis shows that the 301 Redirect is the key:

GET /administrator/ HTTP/1.1
Host: 10.0.0.4

HTTP/1.1 200 OK
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Wed, 22 Jan 2014 09:45:30 GMT
[LOGIN PAGE]

POST /administrator/index.php HTTP/1.1
Host: 10.0.0.4
username=username&passwd=password

HTTP/1.1 301 Moved Permanently
Location: http://10.0.0.4/administrator/index.php

GET /administrator/index.php HTTP/1.1
Host: 10.0.0.4

HTTP/1.1 200 OK
Expires: Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified: Wed, 22 Jan 2014 09:45:37 GMT
[USER LOGGED IN]

Server now can get authentication credentials and sends to the user the administration page.

Temporary fixing a 301 Redirect loop caused by Cisco WAAS

I suppose that Cisco WAAS shouldn’t intercept HTTP 301 after a POST, but currently the only solution is globally disable locally 301 generation messages by Cisco WAAS. Go to Devices > wave > Configure > Acceleration > HTTP/HTTPS Settings and unflag “Enable local HTTP 301 Redirect messages”:

Disable_301_on_WAAS

Posted on 22 Jan 2014 by Andrea.
  • Gmail icon
  • Twitter icon
  • Facebook icon
  • LinkedIN icon
  • Google+ icon