Share this:

Big Data and HPCBig Data is becoming much more than just widespread distribution of cheap storage and cheap computation on commodity hardware.  Could Big Data Analytics become the new “killer app” for HPC?

IDC calls it “High Performance Data Analysis”.  The speed of computation matters just as much as the scale.  In order to unlock the full potential of Big Data, we have to pair it with “Big Compute” (High Performance Computing, or HPC).  Here are 3 ways Big Data and HPC are converging, and how you can take advantage of the phenomenon right now:

Hadoop Meets Infiniband

Many consider Infiniband performance just as basic of a requirement for HPC as bare metal processing.  If you can’t move information back and forth between nodes quickly, it really limits the horizontal scalability you can achieve.  RDMA for Apache Hadoop provides an excellent high speed, low latency interconnect option for Big Data platforms.  You can even provision a Hadoop cluster in the cloud that leverages RDMA in no time.  Consider that 56Gbps FDR Infiniband can be over 100 times faster than even 10Gbps Ethernet due to its superior bandwidth and latency advantage.  Short of custom (very expensive) bus fabrics, this is the fastest way to distribute data and processing across computational nodes.  Finally, you can scale that Big Data platform to the size it deserves without worrying nearly as much about bottlenecks.

Hadoop Meets Accelerators

Another key feature of HPC is the use of popular coprocessors and accelerators, such as passively cooled NVIDIA Tesla and Kepler GPUs.  Just as these technologies greatly assist technical computing solutions, they can also help Big Data and Analytics.  Hadoop leveraging GPU technologies such as CUDA and OpenCL can boost Big Data performance by a significant factor.  All other things being equal, higher performance Big Data platforms such as Hadoop and MapReduce will lead to much faster results for complex analytics.  In fact, the only way to keep up with the growing amount of data we are collecting is to increase computation speed at the same time.  Big Data leveraging coprocessors and accelerators is an important way for HPC to make a big impact in this space.

Big Data and HPC Converge in the Cloud

As Big Data fuels public cloud growth faster than any other application, HPC on demand is an emerging force ready to help.  The more data we collect, the more computational capacity we will need to analyze it.  Simply stated, Big Data and HPC growth in the cloud go hand in hand.  The only way to provide enough scale to keep up with demand is to deploy HPC class assets to increase processing performance and density.  Thanks to the marriage of Big Data platforms with high speed interconnects and coprocessors, HPC on demand services such as JARVICE are positioned to help enable the next major wave of analytics innovation.  We can’t wait to see what you will build, solve, and even revolutionize with this type of power at your fingertips!