PS3 slim is now available in india at Sony Center I picked it from vashi,Mumbai on 27th nov 2k9 . Not able to get at croma or other retails still. So if are as desparate as me u know where to head to in your city.

Just completed a new Condor setup, a pool of 80 Processors and 40+ machines till now .
here are 1,2,3,4 for the same
1. Download Condor
2. Install condor on a machine tell it that it is a master so start a new pool , in case you already have a pool this post is not for you
3. Install condor on all machines that u have.
4 . Start submitting jobs

Setting up a condor pool for job scheduling and getting to use Idle time of hundreds of machines is not difficult I think every organization with computers should be having it as a resource. And  you can enjoy the real magic of Condor even without going deeper into it just literate your users on how to submit jobs. that’s it , Well I don’t recommend using condor if u don’t have a reason for using it, check with your system administrator too.

So what can u run on those Idle CPUS, well there are lot of things we will talk on these real soon.

I have done a lot of amusing things in my life & asked a lot of Questions to my parents , teachers mailing lists ,and to Myself but this one at Microsoft site is what I think I was looking for to prompt me to share that link and restart writing at my blog, hope I am regular with some good posts.

I am feeling good , so as my organization OSCAR and my mentor Donglnn, many thanks to him for support he has given me . Please read this page for more details, since I am running out of time, job search is a time consuming task

Powered by ScribeFire.

Technorati Tags: gsoc, oscar, google, job

From condor project

The goal of the Condor^® Project is to develop, implement, deploy, and evaluate mechanisms and policies that support High Throughput Computing (HTC) on large collections of distributively owned computing resources.
Guided by both the technological and sociological challenges of such a
computing environment, the Condor Team has been building software tools that enable scientists and engineers to increase their computing throughput.

Condor package  for Oscar is currently built around latest stable version 6.8.5. Condor installation is a straightforward way one can use .tar.gz or rpm and new install method is a perl script which has to be provided with many arguments and which configure Condor and make changes to condor_config file they way you want to see condor functioning.

1. I have tested Condor 6.8.5 with FC-5 and FC-6 so need to extract the correct rpm and pack it with package.

There are directly two ways of getting it configured .
Common : Head node as submit and manager & all client nodes in execute mode.

A. Head node in also in execute mode.
B. Head node able to send jobs to other manager. (Not more than one manager for now).
C. Head node able to receive jobs from other managers.

All the three ways above can be easily mixed, but common will always be there by default.

2. I am already done with basic scripts.

what remains for condor package to be released is
a. Convertion to new format (yet to understand how to do that)
b. Uninstaller(which should work in oscar_wizard manage mode, i.e for existing clusters)
c. Testing with other distros
d. test cases (currently performing)

By 18/July Condor package should be available for install with Oscar with all the above configuration modes. There are many more advance configuration of Condor on a cluster I am checking on it and if possible add it to this release but unless very important I won’t like to get them included.
A final release (  that might be first public release) will have to wait some more time which will include more distributions supported by Oscar than just Fedora code 5 & 6, and more testing done on the package.

There will be a issue if we not try to use VDT and configure WS-GRAM for condor scheduling since VDT itself installs Condor as a part luckily there is how to on how to avoid that.

So I am pretty much close with this. I shall make a post when this is over I shall again take over vdt-globus package there is some more learning I did with vdt so will jump directly to configuration since setuping up is almost complete with one issue left which can change the way VDT will install, more on that later.

Technorati Tags: condor, oscar, vdt, globus, gsoc

Powered by ScribeFire.

I have my Internship with Novell going on and will finish on 15th June so like always there is a semester end rush, something that I am *not* looking forward to encounter in GSoC project though, I have a final presentation and Report before 15th, I will be 100% on the GSoC work after that.

Updates so far..

I have analyzed some aspects for creating globus package for OSCAR, there are still some parts that need to be done so as to get it is done.

I was not frequently blogging but after 15th I shall be more active on providing updates frequently and documenting stuff in OSCAR Wiki.

My GSoC project involve two packages

1. Globus package.
2. Condor package.
( actually third one “gx-map” , but there is already some support for this in VDT more on it in other post)

I would talk about Globus package in this post.

Globus package must have following(I will provided full details soon.)

1. Installation
2. Configuration
3. Uninstallation
4. Support for all distro supported by OSCAR.

There were basically two ways  I was exploring (not explored other available )

1. Using a VDT backed installation. (vdt requires internet connectivity while installation,until there is something like this available)
2. Using created/available  globus rpm’s

We are using VDT which uses pacman for installations and configuration of all its components. We have to be flexible here to make it available all of the following features.

1. VDT’s current cache at (for 1.6.1) http://vdt.cs.wisc.edu/vdt_161_cache , requires internet connectivity.
2. Use locally downloaded vdt snapshot for particular distro.

Q. How to provide local cache for a distro so that machines without net connectivity can deploy globus to their cluster?
Ans.This is actually not very important since anyone planning to install globus will surely have net connectivity, unless their is some reason like research projects. So I will try to do this some time later  after globus package is available for deployment.  So far my idea is to have a local mirror of VDT current version(1.6.1 costs around 2.8 GB) than figure out some script that can make a distro based tar file for download, I am sure it will be far less than the mirror itself   So that OSCAR users can refer to the local package source rather than remote. more details on how do i plan it here.

I am currently using full mirror so with proper -pretend-platform flag it is working for me during testing on Fedora Core 5  therefore I am not stressing much on creating platform specific pacman VDT cache at this stage.

1. Installation sources : Because of above mentioned reasons I  have to ask user about installation sources for VDT whether they have downloaded it (so that I can ask them about where they did it)  or go to use online VDT.

2. Answering VDT questions : VDT asks several Questions and obviously they can’t be answered at regular prompt so we will try to preanswer them according to package that has been chosen  I will create set of common questions and package specific question so that user need to answer those which are relevant to their installation after dry running vdt based specific installation commands, this is in order to not to confuse user with so many question which might not be relevant to his/her setup.

3. Selecting what to Install : There are many packages which can provide different functionality accordingly like VDT-Gatekeeper, Globus-Server, Globus-WS-Server  there are some overlaps, I am testing vdt command same like in “2″ point above to make sure what I am installing.

4. Uninstaller : ./oscarwizard provide facility to removal of packages pacman files can be removed easily but to remove vdt packages installed there might be some additional steps not rpm base dso there is a need to create a uninstaller which will use pacman -remove before removing pacman installation from the Oscar.

5. Configuration of vdt installation : <* now here come real hard part* > configuration is somewhat tedious specially because we need to know what packages are selected for install and which configuration apply.

/**/
configurator.html will be ready by this weekend for sure, , I have already delayed this much. but this will have all questions to be asked/ installation sources/ and what specifically to install.

/**/
vdt-install.sh will be invoked with proper options set so as to go ahead with installation.  I hope to invoke it with different parameters.

/**/
here only after calling install script I wil run additional set of commands to do proper configuration.

/**/ I know how to go about uninstallaling vdt package but I just need to make sure that
 ”pacman -remove”  is called before pacman rpm uninstall, I couldn’t find any post_server_*uninstall script in trunk so have to test this.

hmm that was not so difficult Only stressing is configuration ( that involve lot of ways that grid can be configured so making me think more and more on how users may actually want to configure their resources)

This should complete Globus’s OSCAR package which does not involve any touching of clients since that’s all we will need, (not true with Condor package )

In addition to 5 points there is actually one more.

6. Additional interfaces :to globus for SGE/Condor/PBS   more on this when I will complete the above mentioned task.

and one more

7. Support for multiple distribution.( I am currently downloading CentOS 5, Fedora 7 ) to help in this.

Technorati Tags: oscar, cluster, grid, globus, condor, gsoc

Powered by ScribeFire.

Mirroring a pacman based cache is important in case yoo are intrested in experiemnting the package install for and you know that it might take you many many tries (may be for a experienmtatiaion or providing local cache)

There are two ways of going about this.
1. Mirror 
2. Snapshot

only difference is mirrors can be updated not snapshots.

pacman will download and create mirror/snaps which can be readily used as local cache for fast access.

I recently did “pacman -mirror http://vdt.cs.wisc.edu/vdt_161_cache -o vdt-1.6″  and got a whooping 2.8 GB(@128kbps) mirror of VDT 1.6.1   which can be used to test with multiple distribution, as supported by VDT.

There is a catch for junta at low speed connections ,do following

[root@oscarhead vdt]# pacman -def
default: -v http tar
default: -setups csh sh
default: -retry 10 pause-30-seconds
default: -ask shell
No http proxy in use…

So when u are at slow speed 30 sec timeout might not be so good(in my case remote server reboots at some particular time once is a day) so there is a workaround available

pacman -retry 100 pause-10-minutes

or something which can do it reliably since mirroring is a transactional, i.e. either it is done or everything is reversed as before this command  so that means  if you did  99% of the download at it breaks all data is removed,that happened to me a lot of time until I found actual problem    (see this at line 44-46)

Now I had VDT mirror available with me I will require a repackaging of it in such a way that it could create platform specific separate subset of mirror. Which in turn should be accessible in same was as that of original but limited to that particular distro;

Advantage of doing this is simple, providing users with only what they are intrested into and not with 2.8G kind of thing and additionally it might be useful for people who are installaing multiple times for some reason or they don’t have net connectivity at the machine they are trying to run pacman commands, and most important for me OSCAR users should have everything available readily without dependency on anything else except what they have downloaded for their supported distro.

( Important thing to note here is VDT install globus/condor and it is highly unlikely that machines that people are installing don’t have net connectivity but still this is good for testing ,like for me)

Thanks to Alain,Saul Youssef from vdt-discuss mailing list, for providing details on  mirroring reliably at low speed, but still no recovery from local link failure/power failure  

Technorati Tags: vdt, oscar, globus, condor, pacman,

Powered by ScribeFire.

Amit,

With a quick overview of the VDT website, it looks like that VDT is a kind of clustering toolkit like OSCAR and it made me think of the several questions.
(setup == install/configure)

1. Why/how did we start to look at VDT?
2. If VDT has a complete solution to setup globus/condor, can we use it (not VDT itself but its solution) in OSCAR?
3. Is it difficult to install globus/condor without VDT?
4. Can we just start to install globus/condor without VDT by studying how VDT works?

Basically, I want OSCAR to take over eventually what VDT does for the globus/condor installation now.

I am not sure if this is a right approach is or not but I would like to spend some times to dig out how VDT setup globus/condor in a cluster.

Anyway, if VDT is the only way that we have to take, as I mentioned at IM on May 17 we have to consider the following extra things((A),(B), and (C) should be necessary whether we go with VDT or not).

• VDT should be ready for any distros that OSCAR supports(i.e., if there are any rpm dependencies on VDT in a certain distro, VDT rpm may need to be recompiled on the corresponding distros)
• The distros that VDT supports should be the same as OSCAR or more
• VDT should be able to setup globus/condor with local repositories(i.e., without any internet connections)
• The automatic installation should be implemented by referring to this link
• All the necessary pre-configuration should be done at Step 2 “Configure Selected OSCAR Packages..” in oscar_wizard (A)
• During the installation, we can configure with $OSCAR_HOME/packages/*/setup where * can be globus or any appropriate package name (B)
• All the necessary post-configuration should be done at Step 7 “Complete Cluster Setup” in oscar_wizard (C)

But I think that OSCAR can take over all the necessary features of VDT to setup globus/condor.
What do you think?

For a short term, I think using VDT to setup globus/condor on OSCAR is a good idea because I believe that we can figure out how VDT set these things up.
Once we figure it out, let’s do some scripts to take over all the necessary features.

- DongInn

As soon as I will receive Dell^TM Inspiron^TM 6400 Notebook I will start setup for Oscar development environment . Just praying it to get delivered soon.

My old monitor not at appropriate place, It died with sparks

My system served me for 5 years and still alive, available just as a remote window PIII, 256 MB,8MB ATI Graphics. Running Fedora Core 3. It let me submit my GSoC application thanks dear, Oh Yeah thanks to my monitor as well

Powered by ScribeFire.

My proposal for GSoC 2007 for Oscar is Accepted and for this summer I would be working on it. Since I had a good exposure to Grid Technologies while I work with Gridone project at BITS Pilani CSD,

Abstract :

Grid Computing — Globus
Cluster — OSCAR
High Throughput Computing — Condor

Clusters,Grid and high throughput computing projects(condor) are
projects that help setting up computing infrastructures which are of
importance. Clusters ,Beowulf, used to be isolated from other emerging
technologies like Grid because of many reasons importantly ease of
deployment and seamless integration.

Grid software like globus have inherent support for clusters and can
be used easily with same grid interface. Today many deployments
specially in University there exists mixed environments.

Globus/Condor package will provide easy installation and configuration of those two software of top of clusters resources.

I will be mentored by an active Oscar developer DongInn Kim.

Oscar is a simple software which can be used fairly easily to depoly a working beowulf class supercomputers (Clusters) on supported linux distribution like Fedora,RedHat,SUSE and many

All updates regarding the project will be available at tag/oscar

I would like to thanks whole Oscar Community for such a great software ,Google for provide Summer of Code platform and to my teachers who guided me throughout my learning.

Powered by ScribeFire.

Technorati Tags: beowulf, oscar, gsoc, grid computing, grid, globus, condor, cluster, gx-map