tag:blogger.com,1999:blog-48900397220031273412011-08-15T22:07:46.969-04:00Reflections on distributed computing in clusters, clouds, and grids.Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.comBlogger291100tag:blogger.com,1999:blog-4890039722003127341.post-51597515023238737542010-11-16T13:00:00.003-05:002010-11-16T14:02:38.174-05:00

Today, in my graduate operating systems class, we discussed what I believe is the most important result in computer science ever to be persistently ignored:Popek and Goldberg, Formal Requirements for Virtualizible Third Generation Architectures, Communications of the ACM, Volume 17, Issue 7, July 1974.This paper puts forth a very simple principle that must be observed in order for a CPU to be

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com1tag:blogger.com,1999:blog-4890039722003127341.post-4115206100553539752010-11-08T11:00:00.004-05:002010-11-08T11:00:03.381-05:00

Most days, software engineering involves compromises and imperfect solutions. It's rare for two pieces of software to mesh perfectly -- you always have to work to overcome the limitations or assumptions present in different modules. But, every once in a while, the pieces just come together in a satisfying way.A few weeks back, we ran into a problem with BXGrid, our system for managing biometric

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-72959282712298780942010-11-01T11:00:00.005-04:002010-11-01T11:00:01.570-04:00

Over the last year, our Makeflow system has become quite popular here at Notre Dame. Briefly, Makeflow takes a workload expressed in the plain old Make format, and executes it in a distributed system, using the dependency information to set up the appropriate remote execution environment. It does not require a distributed filesystem, so it's easy to get your applications going on hundreds to

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-63924731681518199322010-10-27T11:00:00.002-04:002010-10-27T11:14:40.106-04:00

Hoang Bui is leading the development of ROARS: a Rich Object Archival System, which is our generalization many of the ideas expressed in the Biometrics Research Grid. Hoang presented a paper on ROARS at the workshop on Data Intensive Distributed Computing earlier this year.What makes ROARS particularly interesting is that it combines elements of both relational databases and file systems, and

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-76014535970219988552010-10-18T12:04:00.017-04:002010-10-19T15:12:36.338-04:00

In recent months, we have been working on the problem of building elastic parallel applications that can adapt to the available resources at run-time. Much has been written about elastic internet services, but scientific applications have a ways to catch up.Traditional parallel applications are rigid: the user chooses how many nodes (or cores or CPUs) to use when the program starts. If more

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-39740630464369125252010-04-05T12:45:00.000-04:002010-04-05T12:47:14.415-04:00

As I have noted in this column before, a perennial challenge of distributed computing in the real world is dealing with the multiplicity of operating systems and related environments. If you are dealing with an uncontrolled environment like a large university or an 'at home' computing environment, there is no telling what you are going to get. If you have a piece of software that depends

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-85650972351584605882010-01-21T13:37:00.001-05:002010-01-21T13:39:06.906-05:00

We invite outstanding undergraduates to apply for summer researchpositions in scientific and cloud computing at the University of Notre Dame.Students will build and operate systems that harness hundreds ofmachines at once to attack large problems in science and engineering.Research topics include:Green Cloud Computing Portals for Scientific Research Languages for Distributed Computing More

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-74893424117170637662010-01-11T11:07:00.006-05:002010-01-11T12:29:09.141-05:00

The Green Cloud is now online!The Green Cloud is the invention of Dr. Paul Brenner at the ND Center for Research Computing. It is a containerized data center located at the South Bend city greenhouse, stocked with used servers kindly donated by Ebay, Inc. The first batch of machines was installed in December, and will eventually reach about 400 cores once everything is turned on.What makes the

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com1tag:blogger.com,1999:blog-4890039722003127341.post-72199086661500460942009-10-08T09:00:00.000-04:002009-10-08T09:00:04.815-04:00

One of the biggest challenges in building computer systems is finding a way to make things simpler. Any propeller-head can make a piece of software more complicated. Unfortunately, our industry seems to have a way of gravitating toward the complex. Let's look at the current state of the web browsers -- pick any one -- which seem to insist upon reimplementing or otherwise abusing the operating

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com7tag:blogger.com,1999:blog-4890039722003127341.post-12038568536429045172009-10-01T23:00:00.001-04:002009-10-01T23:08:47.991-04:00

We have been thinking about cloud computing quite a bit over the last month. As I noted earlier, cloud computing is hardly a new idea, but it does add a few new twists on some old concepts in distributed systems. So, we are spending some time to understand how we can take our existing big applications and make them work with cloud systems and software. It should come as no surprise that there

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-4930699428662770402009-08-03T09:00:00.001-04:002009-08-03T09:00:05.393-04:00

This post continues last week's subject of summer REU projects.Rachel Witty and Kameron Srimoungchanh worked on BXGrid, our web portal and computing system for biometrics research. This project is a collaboration between the Cooperative Computing Lab and the Computer Vision Research Lab at Notre Dame. Hoang Bui is the lead graduate student on the project. Rachel and Kameron added a bunch of new

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-37764190802695203382009-07-28T12:01:00.011-04:002009-07-29T13:21:52.091-04:00

This summer, we hosted four REU students who contributed to two web portals for distributed computing: Biocompute and BXGrid. I'll write about one this week and the other next week.REU students Ryan Jansen and Joey Rich worked with recent grad Rory Carmichael on Biocompute, our web portal and computing system for bioinformatics research. Biocompute was originally created by Patrick Braga-Henebry

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com2tag:blogger.com,1999:blog-4890039722003127341.post-7986411163681871782009-07-03T12:55:00.006-04:002009-07-03T14:32:54.923-04:00

In previous articles, I have introduced the idea of abstractions for distributed computing. An abstraction is a way of specifying a large amount of work in a way that makes it possible to be distributed across a large computing system. All of the abstractions I have discussed so far have a compact, regular structure. However, many people have large workloads that do not have a regular structure.

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com3tag:blogger.com,1999:blog-4890039722003127341.post-86777370116221191412009-06-02T11:33:00.002-04:002009-06-02T11:38:14.576-04:00

Dr. Paul Brenner, a research scientist in the Computing Research Center at the University Notre Dame, has been advocating a novel idea called grid heating. He recently won a "Green IT Award" from the Uptime Institute for his work. Here is a short introduction to the idea:Around the world, large data centers consume enormous amounts of power. In addition to the energy needed to spin disks and

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com2tag:blogger.com,1999:blog-4890039722003127341.post-75348527859673994822009-05-29T16:00:00.000-04:002009-05-29T16:35:20.683-04:00

Dynamic linking is one of the more frustrating aspects of distributed computing in the real world. It's is the sort of technology that is meant to optimize the computer's happiness at the expense of the end user's sanity. Dynamic linking should really be avoided, except in a few very specific cases outlined below.For those of you who don't remember, here is a brief primer on linking:Back in the

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com10tag:blogger.com,1999:blog-4890039722003127341.post-39731384147781765022009-04-14T10:16:00.013-04:002009-04-14T12:19:13.872-04:00

Lately, my research group has been collaborating with Prof. Scott Emrich on several problems in bioinformatics. Our students Chris Moretti and Mike Olson have been building a system for carrying out whole-genome assembly problems on campus grids. They recently got it scaled up to run on nearly 1000 nodes spread across Notre Dame, Purdue, and Wisconsin, making the problem complete in a few hours

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-79302972548194199602009-02-25T09:54:00.006-05:002009-03-13T11:13:13.146-04:00

About once a week, a well-meaning person stops by my office to ask a question like this:I need to run about 1000 simulations that take about an hour each. I can't wait a thousand hours for the results, so I need to parallelize my simulation. So, should I re-write my application using threads, MPI, or something else?For some reason, they are always disappointed by my response:Just run multiple

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-47675484603986778802009-02-21T13:41:00.011-05:002009-02-24T16:53:50.833-05:00

In response to my previous article, a commenter asked:Why exponential backoff? To put a finer point on the question, How should I choose the parameters for my exponential backoff algorithm? I think many people choose parameters that back off too much, too fast.The idea of exponential backoff in distributed systems goes back quite a few years. An early example can be found in the Ethernet network.

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com1tag:blogger.com,1999:blog-4890039722003127341.post-29708496523513949242009-02-08T13:32:00.006-05:002009-02-21T13:40:44.085-05:00

A common misconception among programmers is that software should always attempt to hide failures in distributed systems. This idea seems sensible at first, because distributed systems are full of failures of all kinds: machines crash, software fails, and networks go down, just to name a few. If I am writing a function called transfer_file() which copies a file from one place to another, then I

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com4tag:blogger.com,1999:blog-4890039722003127341.post-72001344890536862492009-01-14T09:19:00.013-05:002009-01-14T16:33:36.656-05:00

Kim Zetter at Wired magazine recently wrote about the use of log files in electronic voting machines. (It actually shows snippets of the relevant data, which is a refreshing use of primary evidence in journalism.) The article illustrates an often overlooked rule of software engineering:A DEBUG FILE IS NOT THE SAME THING AS AN AUDIT TRAIL.Here is my rough guess at what happened: Political forces

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-46417252324599315992008-12-29T11:55:00.015-05:002009-01-05T09:34:52.820-05:00

One of our graduate students, Hoang Bui, presented a poster on the Biometrics Research Grid (BXGrid) at the IEEE e-Science conference in Indianapolis a few weeks ago. BXGrid is a large data repository that we have built to support both production research in biometrics and to provide a platform for research in large scale data intensive computing. It provides another example of the idea of

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-51064828421305444802008-12-11T13:49:00.010-05:002008-12-12T10:30:05.780-05:00

I just attended the IEEE conference on e-Science in Indianapolis, and gave this talk on harnessing distributed systems with high level abstractions.Another highlight of the conference was Rich Wolski's talk on Eucalyptus, an open source toolkit for cloud computing. Like Nimbus, it is API compatible with Amazon's EC2. That is, if your code works with Amazon, you can install Nimbus or Eucalyptus

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com1tag:blogger.com,1999:blog-4890039722003127341.post-31323795460305468112008-12-09T12:13:00.009-05:002008-12-09T15:08:24.508-05:00

The end of the semester is nearing, so activity in our distributed system really shoots up as undergraduates finish their semester projects and graduate students hurry to generate those last few research results. You can see this activity reflected in a number of visual displays that we have created to track activity in the system.For example, the following is a snapshot of an applet that

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-4093738847482529152008-11-30T17:00:00.000-05:002008-12-02T14:41:43.417-05:00

Two students at Notre Dame, Qi Liao and Andrew Blaich, recently received the Best Paper award at USENIX LISA for their work on Enavis, a tool that gives a visual display of network traffic collected by the Lockdown network administration tool. Enavis gives the administrator of a large network a way to browse all of the users, programs, hosts, and network connections in a system of hundreds or

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-7336983864684388712008-11-13T09:18:00.013-05:002008-11-17T15:06:42.101-05:00

This is the third in a series of posts on the idea of abstractions for distributed computing on clusters, clouds, and grids. An abstraction is a simple interface that allows you to scale up well-structured problems to run on hundreds or thousands of computers at once.The Wavefront abstraction came up in a discussion with several economists. You want to compute a recurrence relation where each

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-57838556522886612112008-10-31T16:58:00.007-04:002008-10-31T17:47:16.832-04:00

In the last post, I presented the idea of abstractions for distributed computing, and explained the All-Pairs abstraction, which represents a very large Cartesian product. Of course, a single abstraction is meant to address a very focused kind of workload. If you have a different category of problem, then you need another abstraction.We discovered another abstraction while working with Nitesh

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-3187934409500813282008-10-22T12:37:00.013-04:002008-10-23T12:04:00.792-04:00

My current research revolves around the idea of abstractions for distributed computing. An abstraction is a way of simplifying a workload that runs on thousands of machines, in much the same way that a high level language simplifies the tiresome process of programming in assembly language. Let me explain a little more.Real assembly language has operations like this:MOV memory to registerPUSH

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-8705413400925393282008-10-06T10:24:00.001-04:002008-10-06T16:34:23.144-04:00

One of our students, David Cieslak, just presented this paper on troubleshooting large distributed systems at the IEEE Grid Computing Conference in Japan. Here's the situation:You have a million jobs to run.You submit them to a system of thousands of CPUs.Half of them complete, and half of them fail.Now what do you do? It's hopeless to debug any single failure, because you don't know if it

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0tag:blogger.com,1999:blog-4890039722003127341.post-2771894655915811822008-10-01T16:44:00.000-04:002008-10-02T21:20:32.434-04:00

In this blog, I'll discuss open problems and new developments in the field of distributed systems.A distributed system is a set of independent computers that accomplish a meaningful task by working together. The field covers a huge range of interesting systems, from sensor networks to multi-tier web server farms. For the most part, I will be writing about the sort of large computing systems used

Douglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.com0