Detecting new URLs by disabling cookie support

An article for web application security scanner (WASS) developers.

You probably want to find every page in a web application. What if their smart web developer built a vulnerable, "cookie support in your browser is disabled" page for website users? Can your scanner find that page?


Let's think like a web developer. How can I detect if a user's browser accepts cookies? I think the best way[1] is to set a temporary cookie while redirecting them to a controller page. This page then should check if our previously set temporary cookie was sent during client's new request.

A WASS can find this page. Here is a way of doing this;

1. While crawling with cookie support, remember all pages which set cookies and redirect at the same time. We can create a NoCookieQueue for this operation. BTW, Cookies are set using "Set-Cookie" response header and redirection is made using a "Location" header during a 3xx HTTP response.
2. After crawling phase is complete, if NoCookieQueue is not empty, our scanner should disable cookie support in its crawler module and re-visit those pages in NoCookieQueue. This way we can see if those pages redirect to another location that don't exist in our scanners' complete or error queues.
   

Now your WASS can find new pages or test parameters which a cookieless client should see. While these cookie error pages are mostly static, you might find a vulnerable dynamic page, another directory on server, or an HTML comment with sensitive information etc.

_____
[1] A second method is to set a cookie, and then, using JavaScript check if that cookie was set (If doesn't, redirect client to cookie error page). But these "no cookie pages" can also be found with a JavaScript parser in our WASS.