When performing HPC in the cloud, one of the workflow tasks engineers and scientists have to deal with is the handling of large amounts of data. The question the engineer often has to ask is: “What data is useful to me and how can I present it in a functional form?” When all the data is stored in the cloud, it is essential to minimize the time between generating the data and being able to practically use the data. Simply downloading gigabytes of data is not the preferred or most efficient way.

Of course, different applications strive towards different goals. Sometimes the engineer wants all the output data stored locally and readily accessible. Other times, the cloud proves to be a convenient way to store data for projects and the engineer can, at any time, download date a subsets of jobs run months ago; however, turnaround time for an HPC job is essential. There are several approaches to minimize the turnaround time for data-intensive jobs. Below are some real ways Rescale customers minimize the time by optimizing how they handle data.

LS-DYNA pre/post
LS-DYNA jobs can generate large data output. It’s up to the user what data their job generates. How often the run writes to the hard drive and what data is written is left to the user to configure. However, the average engineer is not 100% sure what data he’ll need, but thinks he has a good idea of what he’ll end up using. The engineer makes the decision to save as much data as possible  to binary files and knows how to use Rescale’s convenient wrapper script to run his LS-DYNA jobs:

The engineer, however, saves himself download bandwidth by using Rescale’s built-in command-line post-process option and using the ‘-d’ flag to post-process his input files into a single .db file:

Because he knows the information he will likely need (displacement, plastic strain, Von Mises stresses and thickness), his create_postdb.inp file may look something like this:

The engineer now only has to download output.db with all his desired information in it. So, instead of 35GB of data he is downloading a single 1GB file. Instead of waiting an hour and a half to download his file, the time is reduced to just a couple minutes to download all the data he needs.

On-cluster post-processing
Another engineer is running Converge CFD. Her job just completed and has generated 40 GB of data. Fortunately, she has set up her ssh credentials on the Rescale settings page:


She’s entered her public key–which Rescale has put on all her Rescale nodes–and set up a CIDR Rule so her nodes are only accessible from her network. The job logs have notified her how to connect to the cluster:


She can now easily ssh into her cluster by copying and pasting the command in her terminal or using putty on Windows. When logged into her node, she simply runs:

This launches Converge CFD’s convenient post-process tool which allows her to convert select sets of data into Tecplot, Ensight or other plot-able formats. She can now use scp or sftp to download these small plot files to her local hard-drive.

I don’t want to ‘sync’ about downloading all my data
The first engineer runs many jobs on Rescale, but when he comes in to the office each morning, he realizes that many of his jobs completed overnight. Fortunately, instead of having to go and manually download all his output files, he ran Rescale’s convenient java-based command-line utility for downloading files to his hard-drive whenever a job finishes.

After requesting an API key from Rescale, he launched the command-line utility from his work computer using the simple command:

The command uses Rescale’s API to check every 10 minutes whether there are any newly completed jobs; and if there are, Rescale will download the files automatically to a local drive. The command-line utility efficiently downloads and decrypts (multi-threaded) the output data. So, while our engineer was sleeping, his company network was hard at work downloading all finished jobs to his work computer so he could spend the next day being more productive.

Coming soon: GUI based post-processing
Although the second engineer can already use X11 forwarding over ssh to do some basic GUI based post-processing in the cloud, she thinks it would be very useful if the performance was better. Fortunately, she won’t have to wait long as Rescale is actively working on a remote desktop solution for doing GUI based post-processing

Being smart with data
The bottom line, when it comes to data in the cloud, is to use the tools at your disposal as efficiently as possible. LS-DYNA and Converge CFD are not the only software providing easy to use command-line post-processing tools. Many other tools available on Rescale provide this same functionality.

The question of “What is done in the cloud and what is done locally?” is only answered by deciding how Rescale fits best into your existing engineering processes. Sometimes it is convenient to have all data on premise while doing all post-processing on-premise as well. But often, more of this engineering work can be moved to the cloud as demonstrated in both engineering cases.

For this reason, our goal at Rescale is to provide all the necessary tools to make the issue of dealing with data as minimal as possible.

This article was written by Mulyanto Poort.


In the new user interface (UI), we faced a common UI challenge of letting users select a large set of files and perform a group action on those items. The first thing that jumped to our minds was to show all files with a checkbox next to each one. That proved to be problematic for three reasons: 1) loading data for 100,000+ items up front can take a long time, 2) rendering out the DOM for 100,000+ items can potentially crash a browser, and 3) toggling “Select/Deselect All” or ordering a list of 100,000+ items can be a painfully slow experience.

First, we looked for an existing solution in a frequently used application, and found that Gmail has a similar problem except with emails. In the Gmail app, you cannot “Select All” from your inbox, but once you’ve narrowed down the list of emails using a filter and selected all conversations on the page, you get this notification:


Clicking “Select all conversations that match this search” would select every email that is filtered by the search keyword, but if you tried deselecting an email in the list, the select all toggle would get deactivated. In other words, performing an action on all items that match the filter except a few is not possible in Gmail’s UI. In such cases, you would have to set up your filter carefully to include only the emails that you’re interested in performing a group action.

What Gmail did was definitely better than potentially crashing a browser, but it would not work for our users’ needs. On Rescale’s platform, people actually wanted to delete or download all of their files with the exception of a few. So we came up with what we call an inclusion/exclusion rule, and here’s how it works.

Screen Shot 2014-09-24 at 12.34.20 PM

When a user first visits the results page, he/she only sees first 10 files (unless specified otherwise). The list is paginated from the server side, so the browser is only aware of the 10 files at once.

Screen Shot 2014-09-24 at 12.35.44 PM

If the user starts selecting files using a checkbox, the files that get selected will go into the inclusion set.

Screen Shot 2014-09-24 at 12.35.56 PM

But what’s different here is the “Select all N files”. If the user clicks on this button, the inclusion set gets emptied out and the flag for selecting all files gets set to true. And from that point, the files that the user deselects will go into an exclusion set.

Screen Shot 2014-09-24 at 12.37.49 PM

Finally, when the user performs a group action, Rescale send the server either an inclusion set or an exclusion set to be used for the action.

As users in the past have requested a better way of managing their output files, we worked to make the new UI more helpful–especially for those who run simulations that generate a large set of files.

This article was written by Rescale.


Stephen Jones is a lead software engineer at SpaceX and former CUDA architect at NVIDIA

For this post, Stephen Jones gives his perspective on aspects of technology of great interest to Rescale’s users. As a major contributor to CUDA, many of Rescale’s users have run analyses taking advantage of his work on GPGPU technology, and others have created their own codes using CUDA libraries. From a personal perspective, having returned to work with engineering software after a number of years in general software engineering, I decided it would be interesting to ask Stephen some questions on technological advances of recent years, and how he sees future trends.

To get the ball rolling, I started with the evolution of parallel computing:

Stephen Jones: Around 10 years ago, all computing became parallel computing. Processors stopped getting individually faster, and Moore’s Law was continued through adding more processors (consumer processors all have at least 4 cores, and servers can have 16 or more – any non-parallel program is therefore using no more than 25% of the machine). I think a key effect arises because parallel programming is different to serial programming in several fundamental ways which make it much more difficult. I expect in future (sic) to see parallel programming be done by tools (compilers, libraries, automated programming languages) not by humans, and that this will mark a sea-change in the way all programming is approached.

At Rescale, we have a diverse customer base. Many of our partners in industry or academia work with High Performance Computing (HPC) clusters on-premise. I asked Stephen what HPC means to him:

Stephen Jones: I see this as a general term covering the cutting edge of computing technology. The universe is complex, and we are very far from being able to model it precisely; therefore, we can always find tasks for ever-larger computers. What happens is that, as technology advances what a computer can do, it unlocks new problems which we couldn’t attack before. HPC will always exist, and will always be a niche market. On the other hand, it won’t soon be supplanted by “powerful-enough” desktops because of the aforementioned complexity of the universe. It does act as a very interesting technology incubator, in terms of both hardware and software.

Engineers and scientists worldwide are using the Rescale platform to complement and extend the resources their organization can provide. I asked Stephen about how he sees the utilization of cloud technologies in HPC:

Stephen Jones: This is one of the great enablers of the last 10 years. Amazon’s cloud, in particular, is game-changing and (to my mind) one of the most amazing things in the computing world. It is now acceptable, when trying to get your startup funded, to say “my infrastructure is AWS so we’re not spending money on hardware”. What’s particularly interesting is that it enables HPC as well as basic IT infrastructure, so you begin to see startups doing new things with massive-scale computing which previously were out of reach of such small players. In effect, we can have 100x as many people developing for HPC as ever in the past, and that has to produce new interesting applications.

In my role as an application engineer here at Rescale, I recently built and installed some open source packages that take advantage of CUDA and NVIDIA hardware on the Rescale platform. I asked Stephen about his personal experience working on CUDA:

Stephen Jones: It’s obviously hard for me to be objective about this, because I was the architect of CUDA for a number of years. One thing you see in parallel and HPC programming languages is a very “lowest common denominator” approach – nobody wants to invest in highly complex code which will only run on a single machine and will prevent them from buying a different one. The result is the “designed by a committee” problem, coupled with a “master of none” result. CUDA is interesting because it intentionally goes to the other extreme – it works only on a single type of machine but is therefore able to push the boundaries because it is not limited by trying to please everyone. It is doing well right now because the NVIDIA GPU is widely prevalent in HPC; the result is that CUDA investigates a lot of new programming model aspects first. Many CUDA features find their way into other languages in one form or another (OpenCL, OpenMP, Renderscript) because they’re found to be useful. The thing which really enables CUDA is that the language designers can get hardware support from the chip designers (because they’re in the same company), and so push new limits which just are not open to independent language designers. There’s a lot of innovative stuff in CUDA, and even more coming in the future, as a result of this close association.

One thing I’ve noticed with new codes is the productivity enhancement researchers are achieving by the judicious use of Python in scientific projects. I asked Stephen what combination of tools (that may include languages) he would recommend to scientists or engineers hoping to achieve maximum productivity so they can do their actual work:

Stephen Jones:  I’m a believer in always writing software in the highest-level language and the fewest number of lines possible. Even in high-performance code, often just 10% of the lines of code are consuming 90% of the cycles. If this 10% implements a well-known algorithm (linear algebra, fourier transform, etc.) then you can often use an off-the-shelf implementation which someone else has fine-tuned.

This means you can often get away with doing *everything* in a high-level language, such as SciPy or NumPy, and use the underlying libraries to do the heavy lifting. I’d say you’d be crazy not to – it leads to more maintainable code, with fewer lines and fewer bugs, and most importantly for industry: you can much more easily hire people who know python, than people who know arcane low-level parallel languages. Your code base then has much better longevity.

Where things get interesting is when the CPU-intensive operations are not standard algorithms. You then need to actually implement them yourself, and once again I’d recommend the highest-level language you can get away with. This involves thinking about the maintainability<->performance balance: you might want to accept not reaching peak performance in exchange for writing more maintainable, future-proof code. NumPy, for example, lets me do some CUDA-related parallelisation (sic) in a Python framework, with a very respectable speedup for very little effort.

At SpaceX, I target about 75% of peak performance on the grounds that the last 25% will make the code much harder to develop and – most importantly – much less future-proof. It’s simply not worth that extra 25% if I have to recode it for new architectures every few years. Admittedly, we don’t use Python for our performance code, but we do use it for a host of things which would be way too complicated in C++.

As for development tools – I’m always looking at what language and/or platform has the best diagnostic tools. Debuggers, profilers, and analysis can save me literally 50% of my development time. So a combination of that and of a high-level language is the first thing I go to. As you can tell, I’m a fan of Python, but it really will depend on the use-case. Matlab also has excellent tools, though it comes with framework lock-in.

A word on other types of language. Haskell, in particular, is getting a lot of traction these days. As a functional programming language, it parallelises (sic) much more naturally than an imperative language and it’s looking interesting. Academia is very interested in it, but I think that the talent-pool is too small for me to use it in industry: hiring good people is hard enough for Python, let alone for C.

I followed up by asking what he saw as the future of FORTRAN, a language still frequently used in engineering analyses.

Stephen Jones: It’ll go away as computers become less able to run it. It is not intrinsically parallel, so you need to wrap a framework such as MPI around it. I think that larger (exascale) computers will be increasingly difficult to program with MPI, and that this will mark the end of FORTRAN. Unlike C, FORTRAN has no presence in low-level programming so it’ll fade as the dusty decks are finally rewritten.

A final word, looking into my crystal ball: I think the future of programming lies with software-writing-software. That is to say, I’ll write a program which writes the programs for me. Compilers already do a lot for us; high-level frameworks like Ruby manage all sorts of complexity for web programming; auto-parallelisation (sic) is a big buzzword right now (though it’s not here yet). Computers are going to become increasingly hard to program, and we’ll only be able to do it with the help of other computers…

Moving on to the topic of working with what I like to call “real engineers” (disclosure, my first engineering job was in a ship-repair yard), Stephen was equally forthcoming.

Stephen Jones: It is really interesting, to work as a software guy in a pure mechanical-engineering environment. A computer is a tool like any piece of lab equipment, and nobody could imagine doing engineering without one; however, non-software-engineers at all engineering companies I’ve seen use them like tools (which is to say in a fairly software-unsophisticated way).

Finally, software engineering and its relationship with engineering is a topic that really interests me. I asked Stephen for his thoughts on this.

Stephen Jones: This is another huge item. I live it every day at SpaceX, trying to bridge that gap. I went to a conference a few months ago on engineering simulation and met a dozen people who all had surprisingly similar stories to tell (as did I) – the lack of understanding about software is widespread, and a real handicap for engineers. We need to educate engineers far better than we do in use and programming of computers – it’s not like a drill or a saw that you can just learn to use easily on the job. Those companies that spend the time to provide this education gain clear benefits in engineering innovation and productivity. This goes for all sciences, by the way, not just engineering. Working in HPC for NVIDIA, I met countless brilliant physicists, chemists, and biologists who were struggling to do their cutting-edge science with these massively complex supercomputers. The most successful ones were the ones who had actively learned to program, as against just muddled through (sic). In many cases, the less experienced physicist does better work because they can use the computer more effectively. There is undeniably an attitude – in both science and engineering – that software is “less pure”, but I suspect this will prove to be generational as people now grow up with computers at their fingertips from childhood.

Many thanks to Stephen for his unique perspective. At Rescale, we will continue work to enable scientists and engineers to achieve their goals by providing the scalable computing resources they need.

This article was written by Rescale.