The World’s Largest Supercomputing Cloud

I had no idea the Hubble telescope could see only 12 billion years into the past.



Frankly, I’d never really thought about telescopes looking into the past until Dr. Michael Norman, a researcher from UCSD gave me a basic education in astronomy – and explained the Hubble looks at celestial bodies whose light is just now reaching us. But it can “only” see 12 billion years into the past – and that was a veil he’d like to pierce. (I asked him what he did for a living, he said, “I simulate the universe.” Trump that job description.)


The question he was interested in answering was, “what about the prior 1.7 billion years?” The universe is roughly 13.7 billion years old, and given the Hubble’s limits, he was using the world’s largest supercomputer, the Ranger platform at the University of Texas at Austin’s TACC (Texas Advanced Computer Center) to simulate the prior 1.7 billion years. (He later confided he was most interested in the prior 1.5 billion years, the first 200-300 million were characterized by lots of hydrogen fog, yet to clump into the wells that enable stars to be born.)


I was asked to give a keynote to celebrate Ranger’s opening, and this was only one example of the flood of basic research and science that will now be performed on the world’s largest open computing platform. Open? The facility was funded by the National Science Foundation, and is committed to providing large scale supercomputing as a service to any researcher or scientist within the US (submit proposals here). Ranger is built entirely on Sun – to dip into geekspeak for a moment, here are the stats:


  • In around 6,000 square feet datacenter space, consuming less than 3 Megawatts…
  • More than 4000 quad core Sun/Opteron blades, 120+ Tb of DRAM, running CentOS
  • Delivering more than 500 teraflops computing capacity
  • Jobs scheduled by Sun’s Grid Engine
  • Interconnected by two, 100 terabit non-blocking Magnum switches (horns optional)
  • Data managed by the Lustre file system, on Thumpers
  • More than 2 petabytes of storage
  • Managed by our hierarchical data management SAM-FS product, archived to Sun tape platforms
  • With overall systems managed and monitored by xVM OpsCenter (the world’s largest installation).


An enormous amount of engineering went into the construction of the facility and the technology behind it, which Sun can now replicate across the world in smaller (and larger, of course) installations (public and commercial). Beyond governments and research facilities, Industries across the planet are turning to high performance computing for business advantage, not simply scientific endeavor. This system consumes a fraction of the power budget required just a few years ago – making it among the greenest supercomputing facilities on earth, too.


To give you a sense of how significant Ranger actually is, check out this chart (click for live version):





Ranger’s capacity exceeds that of all other National Science Foundation granted supercomputing facilities combined. When they say big in Texas, they mean big.


As the director of Cyberinfrastructure at the NSF pointed out during his congratulatory speech, computational simulation is now considered a legitimate field of scientific exploration. From drug discovery to climate modeling, fluid dynamics to simulating the universe, epidemiology to materials science – a facility of this size will revolutionize science, in the US and across the world. To date, there are already more than 500 research projects using Ranger – it’s already changing the world. And because it’s part of the NSF Teragrid, output from the studies will be shared throughout the world. Open means open. Jay Boisseau, TACC’s director, let me know they’re dangerously close to receiving more applications for time on Ranger (they have about 500 million cpu hours to allocate each year, or 125m/quarter) than they have available. For folks like Jay and Dr.Norman, increasing capacity increases appetite – unlike much of enterprise computing, where surpluses are often consolidated away (the heart of Greg’s redshift theory).


How did Ranger come together? It resulted from a commitment to basic science from the National Science Foundation, a passionate set of people at the University of Texas, inspired by a driven technical leader in Jay, commitments from an exceptional (truly exceptional) team of TACC, Sun and AMD employees, with all three groups in a mad scramble to stand up the facility in record time – as the world’s largest open supercomputing facility. The world’s largest, by a factor of 4.


Ranger will transform academia, industry and ultimately society. Why do I believe that?



As I pointed out during my speech, there was a point at which the Niagara Falls power plant in the United States supplied fully 30% of America’s electrical requirements. The engineering and basic science that went into that work parallels the work required to build Ranger. It was truly fundamental research.


Did electricity transform society? Unquestionably. Will knowing what happened in the first 1.7 billion years of the universe transform our lives? We don’t know yet. That’s what Dr. Norman is figuring out. A question Sun, AMD and the University of Texas researcher will now be able to help him answer. With a platform Sun will now be making generally available to the commercial market. (I was going to write something like “parting the clouds of cloud computing,” but even I winced when I read that.)


__________________


(For those interested, this is a great summary of Dr. Norman’s approach to computational astrophysics – notably, like pretty much all the work I’m seeing in high performance computing across the world…


…predicated upon free and open source software.)

41 Comments

Filed under General

41 responses to “The World’s Largest Supercomputing Cloud

  1. Tobias

    One day a part of the worlds global computing cloud will be running in our living rooms … to heat them. Mark my words.

  2. First of all I want to say that maybe this computer with those enormous configuration can define purpose of our lives. All things on this planet can be prooved on some way (like Darvin’s theory that is not complete but will be one day when scientists find lost piece) but when we came to talk about universe and big bang whole prooves stops and after that all became just theories. I hope that Dr. Norman will discover the most wanted secrets. I wish him good luck and I keep my fingers crossed.

  3. A Sun CEO praising a supercomputing project using Sun hardware but a non-Sun OS? I would take this as proof of the positive paradigm shift at Sun. In the contemporary corporate landscape, simply giving the customer what they want seems increasingly rare – except at Sun. Bravo.

  4. javathreads

    NASA is running the Solaris operating environment on SUN Microsystems servers, this as a "Java developer" makes me very proud to see such a company being everywhere guided by a singular vision — "The Network is the Computer".

  5. Nice job Jonathan. I think it’s important to note that the deployment of Ranger was the result of incredibly hard work by the Sun team.
    You can view the complete video of Jonathan’s speech at the TACC Ranger dedication ceremony at hpc.sun.com.
    Cheers,
    -Rich

  6. What is the programming model like, in particular coordinating access to all the data ?
    Distributed persistent memory would provide a simple programming model, by presenting all the data in local address space, with each process seeing the same address space.

  7. Matt

    Very nice! I just hope they’re affordable, I’m expecting I’ll need one of these for the next release of Windows, in 2020.

  8. Wow. you guys paid too much! Everyone knows the universe is only 6000 years old!
    … which I can simulate just fine on my EC2 box over on Amazon😛
    (Just to clarify – I’m joking)

  9. Anonymous

    This is fantastic news !
    I’m a student that is looking to follow that path of Astrophysics.
    I dev OSS. I wonder if Sun could do something soon that could allow me to get my hands on a Sun Ultra 25 Workstation for my studies ( A89-XHZB1-9AT-1GDU ), I been a good boy all last year *hint*😦
    Dear Jonathan,
    Please bring SPARC to a value where students can once again code around it and bring value back to the SPARC platform. Also, please support XVR-300 Graphics Accelerator in Xorg.
    Thanks for your time,
    Edward. (Hope he reads his own blog ?)

  10. …and I MAKE the universe.🙂 Ha ha, just kidding. But, seriously, much respect and admiration. That’s one pretty girl(Ranger). Propers to TACC. Good job! Open means open. Reach beyond the limits of today and into the possibility of tomorrow.

  11. suntransformer

    Jon,
    You got it right — Sun transforms the world, as is shown by your leadership in almost everything (design, performance, cost, maintainability). It is clear that Sun has a better solution (I dare say much better than US politics!) for changing the world, and making it a better place. The world would never be able to repay Sun for all the donations it has given to the open source community. It is a blessing for individuals to use Sun’s far superior hardware and software. It is my expectation that as the world learns more about Sun, and your truly revolutionary ideas, they will switch exclusively to Sun’s products for all their computing needs!

  12. obligfanboy

    Awesome!
    Now that’s what we call nice bright Sunshine!!
    (Sorry, couldn’t resist.😛 )
    Fwiw, I did wish upon a couple of shooting stars on my recent vacation… (first time I saw them, to be honest.)
    And I found that the bright glitzy city lights hide a lot more really beautiful and serene stars than we spoilt city brats care to think.
    Planets shouldn’t even think of outshining these stars. It just needs a trip back to the clear, open skies to see things in perpsective.
    @Tobias: +1

  13. Dave

    This is a fine example of how the private market can build a solution which is far superior to projects managed through government red tape.
    It is a pitty that a proprietary firmware and a proprietary CPU architecture were required in each one of those nodes deployed – instead of being a completely open solution based upon an open CPU architecture like OpenSPARC ( http://www.opensparc.net/ ) and Open Firmware ( http://en.wikipedia.org/wiki/Open_Firmware )
    It is nice to see SUN succeeding in the proprietary commercial space, however! Kodos to SUN for competing in this marketplace and winning!
    Perhaps the need for proprietary nodes in a supercomputer solution can be fixed with the next commercial engagement by SUN with a complete soup-to-nuts open solution!

  14. Bullfrog

    Why do you want something that sees into the past. Me, personally I would rather be able to see into the future rather than the past as it were.

  15. Software Guy

    It’s just another dying solution like RFID in Sun.
    (NOTE: The whole RFID team disappeared quitely)
    It’s also a proof of "Hardware Company" strategy. Solaris can be ignored as long as Sun many boxes sold. And.. is Java(the word in your stock symbol) relative here?
    WHAT IS YOUR SOFTWARE STRATEGY ????

  16. Peter

    Time is relative, don’t you just love the Universe? In a way, even though our liftimes are so short, our presence is immortal in the Universe, where our energy, although miniscule, will remain, as light and radio waves, long after our passing. I think it’s great that Sun is making such a dominant impact on hpc with commodity x86, I do however, like Edward’s comment, capture the minds of the future and your sparc will shine long after your passing.

  17. seer

    To Bullfrog:
    Kinda obvious…the past informs the present and the future….

  18. sunrocks

    At last, a company that creates supercomputers for the good of mankind, not only to play chess😀

  19. anne marie

    Sun transforms the world, as is shown by your leadership in almost everything (design, performance, cost, maintainability). Is sun working on or interested in a data flow architecture that manages the super computer or applications that anyone can develop and design using the super computer? This is definitely an area that has not been perfected and is it available now?

  20. I believe seeing in past is very important to see a better future…and appreciating what our ancestors did..

  21. guyfrog

    (Ignore if already implemented)
    Have a look at
    * http://www.masswerk.at/jsuix/
    * "eyeOS"
    IMO, those ideas could be useful in these places:
    http://opensolaris.org/os/
    https://sungrid.dev.java.net/
    umm….. solaris simulator on facebook, a bit like "scrabulous" ?
    (ducks, runs for cover…)
    Of course, applets, though slower, are technically better.
    And better != "popular"😉
    –guy-frog

  22. Kevin Hutchinson

    Clicking the ads on the right side of the "S" on your main page gives this error:
    http://ads.sun.com/ads/www/delivery/adclick.php?bannerid=2495&zoneid=399&source=&dest=http%3A%2F%2Fwww.sun.com%2Fsoftware%2Fproducts%2Fmysql%2Fgetit.jsp&ismap=
    Not Found
    The requested URL /ads/www/delivery/adclick.php was not found on this server
    You’re a $13.3bn company that doesn’t run tests on it’s web site? Wow. I’m concerned coz I hold JAVA shares. How can you sell if your ad clicks don’t work?
    Kevin

  23. Kishore

    1. how do they compare to graphon goglobal, hummingbird exceed on demand, citirx (or does Sun has to find out who they are?), nx (nomachine)? – performance wise.
    Goglobal performance for layout is pretty impressive. Did Sun do any comparative study and has a roadmap to better their performance on LAN/WAN for layout. Goglobal works pretty well with dial-up too.
    2. How do we or Sun overcome the NFS security issue when we deploy workstation? Do they have a plan for security?
    3. Not all of Sun uses Sunray. The EDA area is still dependent on workstations. What is Sun plan to unify this? Do they have a roadmap for "EDA layout capable" sunray?
    4. Is the performance a limitation of the Sunray, its protocol OR the backend server supporting it? If we an make a good size server serve 1 or 2 sunray; is the performance acceptable for layout?
    5. What is the Sun answer for PC over IP technology?. Do they have plans to incorporate some of those technologies?
    6. Is serving vmvare vdi images to sunray over LAN/WAN has an acceptable performance?
    7. The sun global desktop is hardly a workable solution for EDA.
    We need to really get answers and commitments from Sun to try to deploy; otherwise all EDA will put the sun ray on the backburner. There is hardly anything available when doing a search on sunray on internet. Does sun ray has a life or is it a dying technology?

  24. Truly commendable ! Keep up the pace…

  25. Interesting find. some job description Dr. Norman has.

  26. Mike

    How about this job description: "I am providing the means for the guy who is simulating the universe."🙂

  27. hernae3

    Interesting.

  28. That config is huge!I wonder if the horns are scaled according to the size of config you order!!!

  29. Different Mike

    If you’re magnifying the distance you can view light from, aren’t you looking at the future of old light?

  30. What is the programming model like, in particular coordinating access to all the data ?

  31. Excellent work Jonathan. I am a very old investor and until recently had given up hopes on my $20+ shares🙂 Sun looks all set to change the world !
    I have a small compliant though: There is no Adobe Acrobat reader for opensolaris on x86. How do you even dare to think that solaris is going to be on the desktops of university students and then on the data centers if this rudimentary piece of software is unavailable ? If Adobe can give a sparc version, it should be a matter of minutes to produce an x86 version. Does Sun even care enough to go talk about this to Adobe ?
    All the best for your upcoming quarter results !

  32. Key

    We would love to simulate the universe now wouldn’t we? Now this is great! Seeing the past would be awesome and explain some unanswerable questions.

  33. I really wish I could get a mug like in your previous post… Or better 5 of them🙂

  34. Tobias

    Hi! Just wanted to let you know, that I have been subscribed to one SUN newsletter. To one only. In the past days I got so many emails through that newsletter that I consider it as SPAM an unsubscribed completely. Have a nice day, Tobias

  35. Jan Schultheiss

    Interesting story about amazing computer power being unleashed. I hope, however, that ultimately these scientists will also be able determine what was there before the universe, i.e. before those roughly 13.7 billion years …

  36. Raghunath L

    Rival for T1/Rock, it should be wake up call for SUN.
    http://www.pcworld.com/article/id,143581-c,cpus/article.html

  37. Cloud computing is such a bad term to describe Utility Computing. You don’t get anything out of a cloud but rain and it blocks the sun.

  38. It’s amazing how far we’ve come with technology (thanks to companies like Sun) and yet how little we know. It’d be interested to learn what Dr. Norman discovers about those prior 1.5 billion years over the next decade.

  39. What is the programming model like, in particular coordinating access to all the data ?

  40. Vincente

    IBM has been doing buyback after buyback. HP revenues and profit keep beating expectations quarter after quarter. Dell CEO just talked about the overall high growth prospect the company is now experiencing. I think Sun needs to do something too. Another $2 bn stock buyback will definitely show the management’s confidence in the company’s well being.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s