Tachyon Nexus: The World’s First Memory-Centric Virtual Distributed Storage System

The Silicon Review

“Based out of San Mateo, CA, USA, Tachyon bridges applications and underlying storage systems providing unified data access orders of magnitudes faster than existing solutions.”

Open source software is critical to the modern enterprise software landscape. Tachyon Nexus is an open source, memory-centric, distributed storage system. The company is one of the fastest growing open source projects in the Big Data ecosystem. With two years of open source history, Tachyon has attracted more than 150 contributors from over 50 organizations. Tachyon Nexus collaborates closely with the community in furthering this progress.

Tachyon began as a research project when its founder and CEO, Haoyuan Li, was a Ph.D. student at UC Berkeley’s AMPLab in 2012. At the time, Spark and Mesos were taking off. He saw what Spark and Mesos could do for compute and resource management respectively, while the storage piece of this story was missing. Together with his research group, he started investigating how to enable memory speed data sharing across different applications. The first version of Tachyon was built during Christmas of 2012, and it was open sourced in April 2013. Two years later, Tachyon, Inc. was founded, receiving a $7.5 million investment from Andreessen Horowitz, to realize the vision of Tachyon becoming the de facto storage unification layer for big data and other scale out application environments and to provide a commercial backer for the project.

Today, the company is very excited to announce the 1.0 release of Tachyon, the world’s first memory-centric virtual distributed storage system, which unifies data access and bridges computation frameworks and underlying storage systems. Applications only need to connect with Tachyon to access data stored in any underlying storage systems. Additionally, Tachyon’s memory-centric architecture enables data access orders of magnitude faster than existing solutions.

Now, organizations can run any computation framework (Apache Spark, Apache MapReduce, Apache Flink, etc.) with any storage system (Alibaba OSS, Amazon S3, OpenStack Swift, GlusterFS, Ceph, etc.), leveraging any storage media (DRAM, SSD, HDD, etc.).

Solving Critical Problems in Enterprises
Beyond the numbers, Tachyon is solving critical problems in enterprises across different industries around the world. For example, search giant Baidu has been running Tachyon in their production for more than a year. It brings them 30X performance improvement. Barclays, a world leading bank, uses Tachyon to make the impossible possible, by reducing the end-to-end latency from hours to seconds. Public cloud providers such as Alibaba and RackSpace have also shown how Tachyon virtualizes their object storage systems. Intel has published articles to showcase several ways to leverage Tachyon in their customers’ environments. IBM presented how Tachyon can abstract OpenStack storage to enable fast data analytics. It has been exciting to see the project adoption grow from zero companies to many, including various industry leaders. The achievements thus far validate the tremendous potential of Tachyon and demonstrate the industry and community’s great excitement around it.

Accomplishments
Over the past three years, Tachyon has evolved from a small codebase for research prototyping into a stable and reliable system, with a vibrant community, deployed by companies around the world. Since the first open source release, the community has grown from 1 contributor to more than 200 contributors from over 50 companies. There are production deployments of Tachyon with hundreds of machines. Its meet up group has grown to more than 800 people, and the most recent meet up had over 300 registrants. The number of commits has grown from 200 to more than 12,000.

What to look forward to
Tachyon and its community have grown tremendously in many aspects over the past three years. With the increase in adoption of Tachyon and the growing community, a non-profit organization, Alluxio Open Foundation was established, to provide a better venue for the project. The project has been rebranded from Tachyon to Alluxio to protect it from potential trademark litigation and to preserve the intellectual property of the open source software community’s contributions internationally. Furthermore, in response to growing demand for a forum to communicate and learn about the company, a conference is scheduled to take place later this year in the San Francisco Bay Area.

Knowing the CEO – Haoyuan Li

Prior to founding Tachyon, Li worked in Google as a Research Intern for 3 years after which he worked as a Software Engineer at Conviva for a year. He has been a Spark Committer and PMC Member at The Apache Software Foundation since 2011. Li is currently pursuing his Ph.D. from UC Berkeley.