[Server-devel] Reverse proxies on the XS

[Martin Langhoff]

2008/3/9 Ivan Krstić <krstic at solarsail.hcs.harvard.edu>:
> On Mar 9, 2008, at 1:04 AM, Martin Langhoff wrote:
>  > But you can't count on the upstream project to be producing good
>  > caching headers - they wouldn't be able to use it on the Internet.
>
>  This is far too strong a statement to stand without qualification.
>  It's one thing to say browsers might ignore cache control headers and
>  re-issue requests they shouldn't, causing unnecessary traffic. It's
>  another thing entirely to say that you believe applications should
>  intentionally omit cache control headers because it might break the web.

It stems from painful experiences, mostly around bugs with the
combination of MS Proxy + IE4/5/6. A particularly vivid one is where
they decide that caching headers are ignorable unless the url
contained a questionmark or was https. And this is with relatively
blunt cache directives and very widely deployed software. There are
plenty of odd proxies out there that we got reports for, and never
investigated fully.

For a quick check, I browsed the sources for WebKit, which I have on
my laptop, and the cache control logic seems to mostly make sense.
Doesn't seem to support all the variations, but seems to have safe
fallbacks.

And even then, perhaps I am wrong. I would be very glad to be shown to
be wrong - that IE and MS Proxy and odd browsers out there can be
trusted to do the right thing.

>  In a previous life, I built a non-trivial chunk of backend
>  architecture for one of the largest websites in existence. To my
>  knowledge, we were never burned by using cache control headers.

Perhaps it all runs over https or has questionmarks in all updatable
urls. Or perhaps the caching strategy didn't really trust the client
to obey them strictly.

>  hypothetically break is a bad idea, and doubly so when building a new
>  platform.

One thing that I am trying to keep in mind is that the XS is quite
behind, and the behaviour of our upstreams is important. Even if you
are right, the webapp maintainers mostly have been burned as I've
described and are wary. We can guide them towards the light and
improve things here and there - if there's one ;-) - but I don't have
any plans to rewrite their apps, at least not soon.

cheers,



m