Within the life sciences industry, one of the most important simulation methods for developing a new drug is free energy perturbation (FEP), which is a particular method in the class of free energy calculations. In simplified terms, the objective of free energy calculations is to compute the free energy difference between two different chemical states A and B by alchemically transforming state A into state B over the course of several intermediate non-physical chemical states, denoted by lambda. There are several methods available to calculate free energies, including slow growth, thermodynamic integration (TI), and free energy perturbation (FEP); the reason FEP has become a popular method for computing free energies is because of it’s inherent scaling properties which makes it particularly amenable to running in a high performance environment. There are excellent online resources which cover the theory of free energy calculations, so I will not go into more details here, other than to say that the fact that the lambda windows are independent from each other allows us to run multiple simulations in parallel. In practical terms, this means that we can use the Rescale platform to create a compute cluster and divide the work of calculating the free energy into M independent simulations, each with a given value of lambda. To increase the sampling efficiency, we can also couple these independent simulations using a method called Hamiltonian Replica Exchange if the software package we choose to use supports this method.

To demonstrate how easy it is to run these simulations on Rescale, I will take an example from Alchemistry.org for the absolute solvation free energy of Ethanol calculated using GROMACS. In this example, the model has already been built and equilibrated for us so we don’t need to do anything further in regards to model building. The topology file Ethanol.top contains the definitions of the molecules while the coordinate file Ethanol.gro contains the equilibrated 3-dimensional coordinates of the atoms involved in the system. We will use both of these files as they are, without further changes. Moreover, the run input configuration file Ethanol.mdp includes a section for the settings necessary to calculate the free energy using FEP.

According to the settings given in this file, we see that nine intermediate states have been defined using the ‘coul-lambdas’ and ‘vdw-lambdas’ keywords. A given lamdba intermediate state is referenced as a (coul-lambdasi, vdw_lambdasi) pair; therefore, the lengths of each of these arrays must be the same, otherwise, an error will occur. We will run nine separate simulations, one for each intermediate state defined above. The specific lambda value for a given simulation is specified with the ‘init-lambda-state’ keyword and is an integer between 0 and 8. The only work we need to do is write a simple script that will generate the input configuration file for each of the lambda values; this can be done directly within Rescale when setting up a new job, which is discussed further below.

A few comments may be helpful on the other keywords given in the configuration file. First, the ‘free-energy = yes’ keyword tells the simulation engine that we are doing a free energy calculation, and the ‘couple-moltype = Ethanol’ keyword specifies that the ethanol molecule is the only object that will be transformed. In this case, because of the way the ethanol molecule is defined in the topology file, the whole molecule is transformed from a fully interacting molecule to a ghost particle which no longer interacts with the rest of the system. Secondly, the ‘calc-lambda-neighbors = -1’ keyword tells GROMACS to calculate the energy difference between the reference intermediate state and all other intermediate states. This keyword needs to be set in this way in order to do the Multi-state Bennett Acceptance Ratio analysis method.

With this background, let’s set up and run this example calculation on Rescale. First, upload the three input files. I have tarred them together with all three files in the top level directory for simplicity.

Screen Shot 2015-09-21 at 9.28.57 AM

Next, click Software Settings and select Gromacs. Choose version 5.0 (MPICH, Single Precision, AVX2) and write the command script as shown in the screenshot below to run the calculations. This script loops over the lambda values from 0 to 8, generating a new run input configuration file for each lambda value. The sed command replaces the ‘X’ in the Ethanol.mdp template file with the corresponding value for lambda and saves the new input file with the lambda value included in the filename, e.g. Ethanol.4.mdp. Then we continue the normal GROMACS workflow by calling grompp_mpi to generate the input structure file Ethanol.4.tpr. Finally, we give the command to run the Hamiltonian Replica Exchange simulation:

Screen Shot 2015-09-21 at 9.27.39 AM

Here we specify that we will be running 9 MPI processes, one for each simulation at different lambda values, by giving mpirun the -np 9 option. The options after mdrun_mpi configure the GROMACS simulation engine to run Hamiltonian Replica Exchange on nine trajectories (-multi 9) with an exchange frequency every 1000 time steps (-replex 1000). The -ntomp 2 option tells GROMACS to attach 2 openmp threads to each mpi process, so we will be running a total of 18 threads for this calculation. This maps well to the Onyx core types which offer 18 cores per node. With this setup, we will be mapping one openmp thread to each physical core which is the optimal use of the computing hardware. One note on the -ntomp option to mdrun_mpi, if we don’t explicitly tell GROMACS how many openmp threads we want, it will probe the processor to find out how many threads are available. When hyperthreading is enabled on the processor, we will have two virtual threads per physical core and GROMACS will subsequently assign two openmp threads per physical core. This will greatly degrade performance since GROMACS is already highly optimized to run with one thread per physical core. Hence, with -ntomp 2, we explicitly tell GROMACS we want to run a total of 18 openmp threads for this job (divided between the 9 MPI processes).

This brings us to the next step, where we click on Hardware Settings and select the Onyx core type and choose 18 cores. Now, we are ready to run the simulations; once we have selected the number of cores, we click Submit to run the job. For this post, we will not go into the details of how to do the analysis to actually calculate the free energy, we will save this topic for a future post. Suffice to say that the files that we will need to do the final analysis are contained in the .dhdl.xvg output files.

Screen Shot 2015-09-21 at 9.29.41 AM

The creators of this Ethanol solvation free energy example recommend running for 6 ns of simulation time, which took me 3 hours, 21 minutes for a final performance of 42 ns/day. This completes the example for setting up and running a free energy perturbation calculation using GROMACS on the Rescale platform. We encourage researchers in the pharmaceutical industry to run their free energy calculations on Rescale. I would be happy to assist you in becoming familiar with Rescale and look forward to helping contribute to great science and advancing the use and impact of free energy calculations in drug development and beyond.

If you would like to run this job yourself, click this link (you will need to create an account, if you do not already have one). 

To create an account, you can go to www.rescale.com/signup.

If you have any questions or would like more information, please contact info@rescale.com.

This article was written by Sidney Elmer.


Consolidated simulation environment allows Japanese enterprise companies to drastically reduce analysis time and access 100+ simulation software

ITOCHU Techno-Solutions Corporation (headquartered in Chiyoda-ku, Tokyo; Satoshi Kikuchi, President & CEO; hereinafter “CTC”) has formed a strategic partnership Rescale, Inc. (headquartered in California, United States; Founder & CEO; Joris Poort) to become a reseller of Rescale’s cloud simulation platforms. Rescale provides a unified environment that allows companies to accelerate their engineering and science simulations using customizable high performance computing (HPC) resources. CTC now provides this customizable HPC, simulation platform, covering everything from the design and construction of the HPC environment to operation and support. CTC’s current focus for this service is on companies in the manufacturing industry, centering on automobiles and precision machines that use large-scale computer simulations, as well as the construction and pharmaceutical industries.

Computer simulations are very compute intensive and a critical part of research and development teams across many industries, including, aerospace, automotive, construction, pharmaceutical, and energy, among others. However, in order to run these compute-intensive simulations, high-performance servers are required to achieve timely results. Building large HPC clusters internally is very costly; therefore, many companies do not have the full capacity they need to run all their analyses. Insufficient simulation resources can lead to missed deadlines, less-than-optimized product designs, missed profits.

CTC now provides the industry-leading HPC, simulation cloud platforms developed and maintained by Rescale. Across Rescale’s platforms, CTC can offer a complete simulation service that includes everything from the design and construction of a cloud HPC system to integrating with companies’ existing HPC environments, as well as operation and analysis support.

Behind Rescale’s powerful and robust platforms is an infrastructure network of more than 8 million servers comprised of the latest in hardware technology. The platforms are available on pay-per-use service based on the compute cluster size and duration of time needed. In addition to IT resources such as servers and storage, it supports more than 100 simulation software, including structural analysis, fluid dynamics, and quantum chemical calculations. All of Rescale’s data centers maintain the highest security standards in the industry and are available in more than 30 locations around the world. It is also possible to use the platforms by restricting locations, including data centers in Japan, according to certain business security levels.

With these highly secure, customizable simulation resources, Rescale enables companies to drastically reduce HPC procurement timelines, analyses runtimes, and product release schedules. Rescale’s platforms are accessible through the use of an intuitive web-based interface or an Application Programming Interface (API) function for integrating with a company’s existing HPC systems. Companies can now focus attention, budgets, and resources towards main research and development businesses. Many of Rescale’s customers have reduced the overall simulation runtime by over 40%.

CTC has offered CAD*1 and CAE*2 software and systems, accessing simulation technology for many years. It has also worked on analysis support in fields related to weather, construction, civil engineering, pharmaceutical, and biomedical applications. This new strategic partnership will allow CTC to combine it’s years of experience in the analysis and simulation fields with Rescale’s industry leading, simulation-optimized HPC platforms to provide the most complete HPC service to their customers.

Rescale Features:

  1. Scalable HPC environment to improve performance and reduce simulation runtime
    Rescale provides HPC cloud platforms accessing more than 8 million servers from over 30 data centers around the world. It possesses an overall computing capacity of 1,400 peta FLOPS*3 or greater, making it the platform with the largest global infrastructure network and the ability to drastically reduce analysis time and accelerate the speed of research and development.
  2. Extensive list of integrated analysis software
    More than 100 simulation software are available across different industries and different types of analyses, including, structural analysis, fluid dynamics, and quantum chemical calculation. Independently developed software can also be used.
  3. Computing resources optimized for HPC
    Specialized HPC are also available on Rescale’s platforms, including GPU*4 that has been widely used for scientific technology computing in recent years, and InfiniBand*5 for forming high-speed networks.

Originally published by CTC here: http://www.ctc-g.co.jp/en/corporate/press/2015/0911a_e.html

About Rescale, Inc.
Rescale is the world’s leading cloud platform provider of simulation software and high performance computing (HPC) solutions. Rescale’s platform solutions are deployed securely and seamlessly to enterprises via a web-based application environment powered by preeminent simulation software providers and backed by the largest commercially available HPC infrastructure. Headquartered in San Francisco, CA, Rescale’s customers include global Fortune 500 companies in the aerospace, automotive, life sciences, marine, consumer products, and energy sectors.

*1 CAD: Refers to the use of computers for designing architectures and industrial products. CAD also refers to software and systems used for these purposes.
*2 CAE: Refers to a computer system that supports the design and development process of industrial products. CAE is a product design support system and an analysis system used to compute characteristics, such as strength and heat resistance, through the use of a designed product model.
*3 FLOPS: One of the units used to indicate a computer’s processing speed. This represents the number of times floating-point arithmetic is carried out per second. A 1 PFLOPS computer can carry out floating-point arithmetic one quadrillion times per second.
*4 GPU: Graphics Processing Unit, a specialized electronic circuit for handling graphics. GPGPU(General-purpose computing on a GPU), a technology for applying GPU arithmetic resources, has attracted attentions recently. These resources can process a large volume of data in multiple processors simultaneously and in parallel for graphics processing.
*5 InfiniBand: Refers to high-speed I/O bus architecture and interconnection for backbone-type and HPC-type servers and clusters that provides a high level of RAS (reliability, availability and serviceability).

This article was written by Rescale.

Rescale’s ScaleX Enterprise Portal allows administrators to have fine-grained control over the their members’ spending on the ScaleX platform by setting up Groups and Projects. In this post we’ll explain what these concepts are and how to use them.

Groups on the ScaleX platform are groups of members – they are useful mainly for their relationship to other entities like projects. Projects can be limited to participating groups, which allows administrators to control which members participate in which projects.

Projects on the ScaleX Enterprise platform are labels that members can assign to jobs. For instance, a company may decide to use a ScaleX project for each real-world product they are developing, or a consultancy may use a ScaleX project for each client project they have. Administrators can control the spending on projects by setting a budget for each project.

Project budgets, coupled with the ability to require jobs to be assigned to projects, allow administrators in-depth regulation over the spending of their members.

Let’s explain how to use these features with an example, Example Co. Example Co.is developing several products – an electric car engine, an electric car battery, and an airplane wing. Example Co. administrators decide to set up projects for each of these so they control budgets and member participation. First, the administrators go to the project page on the ScaleX Enterprise Portal, and creates projects for each of these products:


Now, the administrators would like to determine which members can participate in these projects. The internal Example Co. groups are Car R&D, Car Engineering, Airplane R&D, and Airplane Engineering. The administrators would like Car R&D members to be able to participate in both the car jobs, Car Engineering to be able to participate in only the Electric Car Engine project, and both of the airplane groups to be able to participate in the Airplane Wing project.
First, the administrators go to the Groups page to create groups:


Then, back on the Projects page, the administrators click on the settings for each project to assign groups:



After the administrators add the desired groups to all the projects, they can see the group membership on each of the project’s settings pages:




Now, the administrators would like to assign budgets to each of these projects. They can do this either on the settings page:


or directly on the Project page:


Members will not be allowed to exceed the budget on jobs assigned to project. Whenever a member creates a job under a project, the project budget will be checked and if it would be exceeded, the job will remain queued.

Now the administrators assign their members to appropriate groups. They do this on each group’s page, which can be accessed from the main Groups list:


Then the member search box on the right can be used to add members to the group:


Members can be added to more than one group.

Finally, the Example Co. administrators would like to require their members to assign a project in order to create jobs. This will allow them to track spending and budget carefully. On the company settings page, select this option:


To recap, all of the Example Co. members will be required to select a project when creating jobs on the ScaleX Enterprise platform, and they will have to select a project that is available to one of the groups to which they belong. Their spending will be restricted based on the project selected.

To begin leveraging the ScaleX Enterprise platform to scale simulations and manage projects, you can contact info@rescale.com or signup at www.rescale.com/signup.

This article was written by Alex Kudlick.