Archive for the ‘GSoC’ Category

Google Summer of code 2007, Project Complete

I am feeling good 🙂 , so as my organization OSCAR and my mentor Donglnn, many thanks to him for support he has given me . Please read this page for more details, since I am running out of time, job search is a time consuming task 😦

Powered by ScribeFire.

Technorati Tags: , , ,

Condor package for Oscar!

From condor project

The goal of the Condor® Project is to develop, implement, deploy, and evaluate mechanisms and policies that support High Throughput Computing (HTC) on large collections of distributively owned computing resources.
Guided by both the technological and sociological challenges of such a
computing environment, the Condor Team has been building software tools that enable scientists and engineers to increase their computing throughput.

Condor package  for Oscar is currently built around latest stable version 6.8.5. Condor installation is a straightforward way one can use .tar.gz or rpm and new install method is a perl script which has to be provided with many arguments and which configure Condor and make changes to condor_config file they way you want to see condor functioning.

1. I have tested Condor 6.8.5 with FC-5 and FC-6 so need to extract the correct rpm and pack it with package.

There are directly two ways of getting it configured .
Common : Head node as submit and manager & all client nodes in execute mode.

A. Head node in also in execute mode.
B. Head node able to send jobs to other manager. (Not more than one manager for now).
C. Head node able to receive jobs from other managers.

All the three ways above can be easily mixed, but common will always be there by default.

2. I am already done with basic scripts.

what remains for condor package to be released is
a. Convertion to new format (yet to understand how to do that)
b. Uninstaller(which should work in oscar_wizard manage mode, i.e for existing clusters)
c. Testing with other distros
d. test cases (currently performing)

By 18/July Condor package should be available for install with Oscar with all the above configuration modes. There are many more advance configuration of Condor on a cluster I am checking on it and if possible add it to this release but unless very important I won’t like to get them included.
A final release (  that might be first public release) will have to wait some more time which will include more distributions supported by Oscar than just Fedora code 5 & 6, and more testing done on the package.

There will be a issue if we not try to use VDT and configure WS-GRAM for condor scheduling since VDT itself installs Condor as a part luckily there is how to on how to avoid that.

So I am pretty much close with this. I shall make a post when this is over I shall again take over vdt-globus package there is some more learning I did with vdt so will jump directly to configuration since setuping up is almost complete with one issue left which can change the way VDT will install, more on that later.

Technorati Tags: , , , ,

Powered by ScribeFire.

Vdt based globus package for oscar!

I have my Internship with Novell going on and will finish on 15th June so like always there is a semester end rush, something that I am *not* looking forward to encounter in GSoC project though, I have a final presentation and Report before 15th, I will be 100% on the GSoC work after that.

Updates so far..

I have analyzed some aspects for creating globus package for OSCAR, there are still some parts that need to be done so as to get it is done.

I was not frequently blogging but after 15th I shall be more active on providing updates frequently and documenting stuff in OSCAR Wiki.

My GSoC project involve two packages

1. Globus package.
2. Condor package.
( actually third one “gx-map” , but there is already some support for this in VDT more on it in other post)

I would talk about Globus package in this post.

Globus package must have following(I will provided full details soon.)

1. Installation
2. Configuration
3. Uninstallation
4. Support for all distro supported by OSCAR.

There were basically two ways  I was exploring (not explored other available )

1. Using a VDT backed installation. (vdt requires internet connectivity while installation,until there is something like this available)
2. Using created/available  globus rpm’s

We are using VDT which uses pacman for installations and configuration of all its components. We have to be flexible here to make it available all of the following features.

1. VDT’s current cache at (for 1.6.1) http://vdt.cs.wisc.edu/vdt_161_cache , requires internet connectivity.
2. Use locally downloaded vdt snapshot for particular distro.

Q. How to provide local cache for a distro so that machines without net connectivity can deploy globus to their cluster?
Ans.This is actually not very important since anyone planning to install globus will surely have net connectivity, unless their is some reason like research projects. So I will try to do this some time later  after globus package is available for deployment.  So far my idea is to have a local mirror of VDT current version(1.6.1 costs around 2.8 GB) than figure out some script that can make a distro based tar file for download, I am sure it will be far less than the mirror itself 🙂  So that OSCAR users can refer to the local package source rather than remote. more details on how do i plan it here.

I am currently using full mirror so with proper -pretend-platform flag it is working for me during testing on Fedora Core 5  therefore I am not stressing much on creating platform specific pacman VDT cache at this stage.

1. Installation sources : Because of above mentioned reasons I  have to ask user about installation sources for VDT whether they have downloaded it (so that I can ask them about where they did it)  or go to use online VDT.

2. Answering VDT questions : VDT asks several Questions and obviously they can’t be answered at regular prompt so we will try to preanswer them according to package that has been chosen  I will create set of common questions and package specific question so that user need to answer those which are relevant to their installation after dry running vdt based specific installation commands, this is in order to not to confuse user with so many question which might not be relevant to his/her setup.

3. Selecting what to Install : There are many packages which can provide different functionality accordingly like VDT-Gatekeeper, Globus-Server, Globus-WS-Server  there are some overlaps, I am testing vdt command same like in “2” point above to make sure what I am installing.

4. Uninstaller : ./oscarwizard provide facility to removal of packages pacman files can be removed easily but to remove vdt packages installed there might be some additional steps not rpm base dso there is a need to create a uninstaller which will use pacman -remove before removing pacman installation from the Oscar.

5. Configuration of vdt installation : <* now here come real hard part* > configuration is somewhat tedious specially because we need to know what packages are selected for install and which configuration apply.

/**/
configurator.html will be ready by this weekend for sure, 😦 , I have already delayed this much. but this will have all questions to be asked/ installation sources/ and what specifically to install.

/**/
vdt-install.sh will be invoked with proper options set so as to go ahead with installation.  I hope to invoke it with different parameters.

/**/
here only after calling install script I wil run additional set of commands to do proper configuration.

/**/ I know how to go about uninstallaling vdt package but I just need to make sure that
 “pacman -remove”  is called before pacman rpm uninstall, I couldn’t find any post_server_*uninstall script in trunk so have to test this.

hmm that was not so difficult Only stressing is configuration ( that involve lot of ways that grid can be configured so making me think more and more on how users may actually want to configure their resources)

This should complete Globus’s OSCAR package which does not involve any touching of clients since that’s all we will need, (not true with Condor package )

In addition to 5 points there is actually one more.

6. Additional interfaces :to globus for SGE/Condor/PBS   more on this when I will complete the above mentioned task.

and one more

7. Support for multiple distribution.( I am currently downloading CentOS 5, Fedora 7 ) to help in this.

Technorati Tags: , , , , ,

Powered by ScribeFire.

Why do we use VDT?

Amit,

With a quick overview of the VDT website, it looks like that VDT is a kind of clustering toolkit like OSCAR and it made me think of the several questions.
(setup == install/configure)

  1. Why/how did we start to look at VDT?
  2. If VDT has a complete solution to setup globus/condor, can we use it (not VDT itself but its solution) in OSCAR?
  3. Is it difficult to install globus/condor without VDT?
  4. Can we just start to install globus/condor without VDT by studying how VDT works?

Basically, I want OSCAR to take over eventually what VDT does for the globus/condor installation now.

I am not sure if this is a right approach is or not but I would like to spend some times to dig out how VDT setup globus/condor in a cluster.

Anyway, if VDT is the only way that we have to take, as I mentioned at IM on May 17 we have to consider the following extra things((A),(B), and (C) should be necessary whether we go with VDT or not).

  • VDT should be ready for any distros that OSCAR supports(i.e., if there are any rpm dependencies on VDT in a certain distro, VDT rpm may need to be recompiled on the corresponding distros)
  • The distros that VDT supports should be the same as OSCAR or more
  • VDT should be able to setup globus/condor with local repositories(i.e., without any internet connections)
  • The automatic installation should be implemented by referring to this link
  • All the necessary pre-configuration should be done at Step 2 “Configure Selected OSCAR Packages..” in oscar_wizard (A)
  • During the installation, we can configure with $OSCAR_HOME/packages/*/setup where * can be globus or any appropriate package name (B)
  • All the necessary post-configuration should be done at Step 7 “Complete Cluster Setup” in oscar_wizard (C)

But I think that OSCAR can take over all the necessary features of VDT to setup globus/condor.
What do you think?

For a short term, I think using VDT to setup globus/condor on OSCAR is a good idea because I believe that we can figure out how VDT set these things up.
Once we figure it out, let’s do some scripts to take over all the necessary features.

– DongInn

Thank you My “Old is gold” System !!

As soon as I will receive DellTM InspironTM 6400 Notebook I will start setup for Oscar development environment . Just praying it to get delivered soon.

My old monitor not at appropriate place, It died with sparks 😦

monitor.jpg

My system served me for 5 years and still alive, available just as a remote window PIII, 256 MB,8MB ATI Graphics. Running Fedora Core 3. It let me submit my GSoC application 🙂 thanks dear, Oh Yeah thanks to my monitor as well 🙂

system.jpg

Powered by ScribeFire.

GSoC 2007 : Globus/Condor package Application accepted.

My proposal for GSoC 2007 for Oscar is Accepted and for this summer I would be working on it. Since I had a good exposure to Grid Technologies while I work with Gridone project at BITS Pilani CSD,

Abstract :

Grid Computing — Globus
Cluster — OSCAR
High Throughput Computing — Condor

Clusters,Grid and high throughput computing projects(condor) are
projects that help setting up computing infrastructures which are of
importance. Clusters ,Beowulf, used to be isolated from other emerging
technologies like Grid because of many reasons importantly ease of
deployment and seamless integration.

Grid software like globus have inherent support for clusters and can
be used easily with same grid interface. Today many deployments
specially in University there exists mixed environments.

Globus/Condor package will provide easy installation and configuration of those two software of top of clusters resources.

I will be mentored by an active Oscar developer DongInn Kim.

Oscar is a simple software which can be used fairly easily to depoly a working beowulf class supercomputers (Clusters) on supported linux distribution like Fedora,RedHat,SUSE and many

All updates regarding the project will be available at tag/oscar

I would like to thanks whole Oscar Community for such a great software ,Google for provide Summer of Code platform and to my teachers who guided me throughout my learning.

Powered by ScribeFire.

Technorati Tags: , , , , , , , ,

Timeline for Condor/globus/gx-map package

Project implemetation schedule

Events

Dates

Installing
Oscar/arranging software/setting environment

11-20
April

Refreshing
Experimentation with Oscar

11
– 20 April

Getting
familiar with Oscar Pkg creation

(learning
from other packages)

20
– 30 April

Globus
package testing (poking around in other packages)

1
– 3 May

Globus
package creation outline

3
– 5 May

Globus
package creation start

6


-basic
installation

6
– 13 May


-main scripts

13
– 22 May


– more
scripts for further customization

22
29 May


-uninstaller

30
– 31 May


-Testing
Globus package with fresh install, fix problems

31
May – 5 June


-Final
testing

6-9
June

Globus
package complete

9
June

—-Break——

10June
-15 June

Condor
package creation outline

16-18
June

condor
package creation start

19
June

condor
package (1 month )

20
June – 20 July

*Uploading
code to
http://code.google.com/hosting

as
soon as it opens after 9th June , will be done for
Globus package fully implemented and partial condor package. for
Mid -term

Testing

20
– 30 July

Condor
package completed

31
July

Cross
environment job submission testing/scripts

between
Globus & condor

1-5
August

gx-map
package outline

5
August onwards

gx-map
package start

gx-map
package complete

gx-map
package testing with running Globus instance

Final
submission for packages.

Before
20 August

Technorati Tags: , , , , , , ,

Porting Fast Fourier Transform Tools

Opensolaris list of ideas were great. I found one of my interest.  There will lot of learning I am looking forward to implement things I promised 🙂

Powered by ScribeFire.

SSHv2 Client server

* Learn new OS
* Learn SSH from its root.
* Implement it
* Enjoy it
* Let world enjoy Plan9 so that it may scale well.

powered by performancing firefox

Java API Compatibility Validator

Just waiting to start coding… cheers..

This project will describe any java package as well as compare the two

packages. It will populate all the classes in the package, describe the

classes like what all public functions as well as data members that are

declared in a specific class.

Apart from this this project will compare two packages in order to see what

all the changes happened in two packages.

The changes would be categorized

[A] at package level depicting following information
1. What all classes have been changed.
2. What all new classes have been added to the package
3. What all classes have been removed from the package.

[B] at class level depiciting following information
1. What changes have been made in the class
2. What all new member functions have been added in a class
3. What all member function have been deleted in a class
4. What all data members have been added in a class
5. What all data members have been deleted in a class
6. What all changes happened at function level as well as data members with

respect to accessibility (from the point of view of access modifiers)

[C] Member Function level
1. What all parameters are added into a function declaration
2. What all parameters are deleted from a function declaration

This tool will expose some interfaces so that it can be integrated with the any java application.

powered by performancing firefox