VDT mirroring

Mirroring a pacman based cache is important in case yoo are intrested in experiemnting the package install for and you know that it might take you many many tries (may be for a experienmtatiaion or providing local cache)

There are two ways of going about this.
1. Mirror 
2. Snapshot

only difference is mirrors can be updated not snapshots.

pacman will download and create mirror/snaps which can be readily used as local cache for fast access.

I recently did “pacman -mirror http://vdt.cs.wisc.edu/vdt_161_cache -o vdt-1.6″  and got a whooping 2.8 GB(@128kbps) mirror of VDT 1.6.1   which can be used to test with multiple distribution, as supported by VDT.

There is a catch for junta at low speed connections ,do following

[root@oscarhead vdt]# pacman -def
default: -v http tar
default: -setups csh sh
default: -retry 10 pause-30-seconds
default: -ask shell
No http proxy in use…

So when u are at slow speed 30 sec timeout might not be so good(in my case remote server reboots at some particular time once is a day) so there is a workaround available

pacman -retry 100 pause-10-minutes

or something which can do it reliably since mirroring is a transactional, i.e. either it is done or everything is reversed as before this command  so that means  if you did  99% of the download at it breaks all data is removed,that happened to me a lot of time until I found actual problem    (see this at line 44-46)

Now I had VDT mirror available with me I will require a repackaging of it in such a way that it could create platform specific separate subset of mirror. Which in turn should be accessible in same was as that of original but limited to that particular distro;

Advantage of doing this is simple, providing users with only what they are intrested into and not with 2.8G kind of thing and additionally it might be useful for people who are installaing multiple times for some reason or they don’t have net connectivity at the machine they are trying to run pacman commands, and most important for me OSCAR users should have everything available readily without dependency on anything else except what they have downloaded for their supported distro.

( Important thing to note here is VDT install globus/condor and it is highly unlikely that machines that people are installing don’t have net connectivity but still this is good for testing ,like for me)

Thanks to Alain,Saul Youssef from vdt-discuss mailing list, for providing details on  mirroring reliably at low speed, but still no recovery from local link failure/power failure  

Technorati Tags: vdt, oscar, globus, condor, pacman,

Powered by ScribeFire.