[Http-crcsync] Handling of gzip encoded responses

Alex Wulms alex.wulms at scarlet.be
Fri Apr 24 12:30:32 EDT 2009


Hi,

I have updated the configuration of the upstream proxy so that it uses 
mod_deflate to inflate/unzip gzip-encoded content before it is handled to 
mod_crccache_server for the delta-encoding.

Now the delta-encoding also works in practice when surfing the web with a 
browser in stead of wget. Previously it did not work well because most 
websites respect the 'accept: gzip' header set by the browser in the request 
and respond with a 'gzip' encoded response.

Another thing that I want to do is to enhance the crccache module (encoder and 
decoder) so that the literal blocks get compressed with gzip compression 
(will use zlib for that, apache already depends on it anyway due to the fact 
that mod_deflate uses it). This to stay in line with the idea that we should 
minimize the data sent over the slow link between the encoder and the 
decoder. Currently we can end-up in the pervert situation that the full but 
gzipped origin server response is smaller then the non-compressed 
crccache-delta response when a significant part (e.g. more then 30%) of the 
page was changed. I realize that this problem is partially offset by the 
line-compression of dial-up modems but dial-up modems use the inferior LZW 
algorithm so doing it with gzip in the crccache module will still yield 
better results.

I'm also planning to add the global sha-1 checksum in the response so that the 
client can check the sha-1 of the reconstructed page against the servers 
sha-1 and send a failure to the client in case of mismatch. Does anybody know 
which standard library I can use for the sha-1 calculation?


Toby,

I have updated the readme's in the doc folder with a remark that mod_deflate 
must be added to the Apache configuration. Can you kindly finetune the 
instructions for Ubuntu? I don't have an ubuntu set-up here so I don't know 
how to detail those instructions.


Kind regards,
Alex




More information about the Http-crcsync mailing list