Form based Authentication not setting cookies

[cyberop5]

Hi,
I've been struggling with this problem for days.  I had the crawler setup properly and started working on a front-end to replace the search page (I need security).  After working on that for a few days, I noticed my results returned from our intranet kept getting smaller and smaller.

The Web crawler seemed to have stopped accepting cookies from the application (phpbb).  I checked the log and the only cookies set when requesting a page are

Code:

Cookie: ASPSESSIONIDSSRDQDCB=FNJJKINDLMBLG

followed by basic authentication.

I use two levels of basic security for our site: basic authentication (which is working fine) and phpbb authentication to view the forums (phpBB).  I found that by setting both to crawl with 2 different urls (e.g. website.com and website.com/phpbb), it applies both authentication methods to everything: everything has a basic authorization header, and until this broke, had cookies for phpbb.

I replayed the headers from the log and the server is indeed sending several set-cookie requests (the log shows them too), but none of the subsequent requests are giving the server the cookie data.

Is there a way to clear cookies?  Is this a bug or configuration problem? Any insight at all?  I've already cleared the index to reset the spider configuration.  It was working at one point.

[kevbroch]

I believe I am having the same problem.  I provide the form-based authentication I need for the crawler and when it initially indexes the site it works fine.  But when it goes to re-index the site it doesn't reauthenticate and therefore all the previously indexed links get reindexed as the initial login page.

Unfortunately I have not come up with a solution to this problem.  Please let me know if you do.

Best Regards, Kevin

[NeilP]

I'm seeing the same thing as Kevin:  the initial login seems to work, and it crawls all of our protected pages with no problems....the first time.  However when it goes to recrawl, by that time the initial session has expired, and it doesn't seem to know that it needs to authenticate again.

Our site might have an additional wrinkle in that the login page you are redirected to if you request a secured resource doesn't have the form that is needed to actually log in (we've got 3 different classes of users on our site, and the type we are using for crawling isn't the default on the login page).  However, as I said, the first time through it knows what to do with the login with no problem, it's just the recrawl that isn't working.

Any updates on this issue?

Thanks!