VDT mirroring

Mirroring a pacman based cache is important in case yoo are intrested in experiemnting the package install for and you know that it might take you many many tries (may be for a experienmtatiaion or providing local cache)

There are two ways of going about this.
1. Mirror 
2. Snapshot

only difference is mirrors can be updated not snapshots.

pacman will download and create mirror/snaps which can be readily used as local cache for fast access.

I recently did “pacman -mirror http://vdt.cs.wisc.edu/vdt_161_cache -o vdt-1.6″  and got a whooping 2.8 GB(@128kbps) mirror of VDT 1.6.1   which can be used to test with multiple distribution, as supported by VDT.

There is a catch for junta at low speed connections ,do following

[root@oscarhead vdt]# pacman -def
default: -v http tar
default: -setups csh sh
default: -retry 10 pause-30-seconds
default: -ask shell
No http proxy in use…

So when u are at slow speed 30 sec timeout might not be so good(in my case remote server reboots at some particular time once is a day) so there is a workaround available

pacman -retry 100 pause-10-minutes

or something which can do it reliably since mirroring is a transactional, i.e. either it is done or everything is reversed as before this command  so that means  if you did  99% of the download at it breaks all data is removed,that happened to me a lot of time until I found actual problem 😦   (see this at line 44-46)

Now I had VDT mirror available with me I will require a repackaging of it in such a way that it could create platform specific separate subset of mirror. Which in turn should be accessible in same was as that of original but limited to that particular distro;

Advantage of doing this is simple, providing users with only what they are intrested into and not with 2.8G kind of thing and additionally it might be useful for people who are installaing multiple times for some reason or they don’t have net connectivity at the machine they are trying to run pacman commands, and most important for me OSCAR users should have everything available readily without dependency on anything else except what they have downloaded for their supported distro.

( Important thing to note here is VDT install globus/condor and it is highly unlikely that machines that people are installing don’t have net connectivity but still this is good for testing ,like for me)

Thanks to Alain,Saul Youssef from vdt-discuss mailing list, for providing details on  mirroring reliably at low speed, but still no recovery from local link failure/power failure πŸ™‚ 

Technorati Tags: , , , , ,

Powered by ScribeFire.

Advertisements

1 comment so far

  1. […] Q. How to provide local cache for a distro so that machines without net connectivity can deploy globus to their cluster?Ans.This is actually not very important since anyone planning to install globus will surely have net connectivity, unless their is some reason like research projects. So I will try to do this some time later  after globus package is available for deployment.  So far my idea is to have a local mirror of VDT current version(1.6.1 costs around 2.8 GB) than figure out some script that can make a distro based tar file for download, I am sure it will be far less than the mirror itself πŸ™‚  So that OSCAR users can refer to the local package source rather than remote. more details on how do i plan it here. […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: