====== IPADS Data Analytics Stack (IDAS) ====== ===== News ===== *[Paper] August, 2020. Our paper “Fast RDMA-based Ordered Key-Value Store using Remote Learned Cache” was accepted by 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2020). *[Paper] July, 2020. Our paper “Unifying Timestamp with Transaction Ordering for MVCC with Decentralized Scalar Timestamp” was accepted by 2021 USENIX Symposium on Networked Systems Design and Implementation (NSDI 2021). *[Paper] July, 2019. Our paper “Performance and Protection in the ZoFS User-space NVM File System” was accepted by 27th ACM Symposium on Operating System Principles (SOSP 2019). *[Paper] April, 2019. Two papers “Pragh: Locality-preserving Graph Traversal with Split Live Migration” and “Pisces: A Scalable and Efficient Persistent Transactional Memory” were accepted by 2019 USENIX Annual Technical Conference (USENIX ATC 2019). *[Award] September, 2018. Congratulations to Xingda Wei on winning the 2018 Microsoft Research Asia Fellowship Award! Only 11 Ph.D students got this prestigious fellows among 102 distinguished Ph.D. candidates from 40 leading research universities/institutions. ===== Overview ===== IDAS, the IPADS Data Analytics Stack, is an open source software stack that targets new heterogenous hardware features (e.g., RDMA, HTM, NVM, and GPU) and integrates software components being built by the IPADS to make sense of distributed in-memory transaction and analytical processing over big data. As shown in our architecture figure below, IDAS consists of a storage layer (key-value store and file system), an engine layer (transaction engine and analytics engine), and a batch of optimizations (from availability, durability, and concurrency to streaming processing, reconfiguration, and load balance). {{:projects:idas-arch.png?500|Figure 1: The architecture of IDAS.}} ===== Source Code ===== You can use git clone or just download zip archive to get the codes The source code of **IDAS** is available at IPADS's [[https://ipads.se.sjtu.edu.cn:opensource/idas|gitlab]] git clone git@ipads.se.sjtu.edu.cn:opensource/idas.git ===== People ===== === Faculties === *[[pub:members:Rong Chen]] *[[pub:members:Haibo Chen]] *[[pub:members:Zhaoguo Wang]] === Students === *Xingda Wei, Minkai Dong, Xiaoniu Song, Wenhao Zhang, Xuehan Ke, Xiating Xie, Zhenhan Gong === Alumni === *Jiaxin Shi (Software Engineer at Baidu) *[[http://dolobyte.net/|Yunhao Zhang]] (Ph.D. Student at Cornell) *[[https://www.cs.jhu.edu/~chlou/|Chang Lou]] (Ph.D. Student at John Hopkins) *Youyang Yao (Software Engineer at Alibaba) *Ning Wang (Software Engineer at Alibaba) *Yaozeng Zeng (Software Engineer at PayPal) ===== Publication ===== *[**NSDI 2021**] [Unifying Timestamp with Transaction Ordering for MVCC with Decentralized Scalar Timestamp](docs/papers/nsdi-2021.pdf). Xingda Wei, Rong Chen, Haibo Chen, Zhaoguo Wang, Zhenhan Gong, and Binyu Zang. Proceedings of 18th USENIX Symposium on Networked Systems Design and Implementation, Boston, MA, US, April 2021. *[**OSDI 2020**] [Fast RDMA-based Ordered Key-Value Store using Remote Learned Cache](docs/papers/osdi-2020.pdf). Xingda Wei, Rong Chen, and Haibo Chen. 14th USENIX Symposium on Operating Systems Design and Implementation, Banff, Alberta, Canada, November 2020. *[**SOSP 2019**] [Performance and Protection in the ZoFS User-space NVM File System](docs/papers/sosp-2019.pdf). Mingkai Dong, Heng Bu, Jifei Yi, Benchao Dong, Haibo Chen. The 27th ACM Symposium on Operating System Principles. Deerhurst Resort, Huntsville, Ontario, Canada, October 27-30, 2019. *[**USENIX ATC 2019**] Pragh: Locality-preserving Graph Traversal with Split Live Migration. Xiating Xie, Xingda Wei, Rong Chen, and Haibo Chen. Proceedings of 2019 USENIX Annual Technical Conference, Renton, WA, US, July 2019.[{{:publications:wukong+m-atc19.pdf|paper}}] *[**USENIX ATC 2019**] [Pisces: A Scalable and Efficient Persistent Transactional Memory](docs/papers/uatc-2019-2.pdf). Jinyu Gu, Qianqian Yu, Xiayang Wang, Zhaoguo Wang, Binyu Zang, Haibing Guan, Haibo Chen. 2019 USENIX Annual Technical Conference, Renton, WA, USA, July 2019. *[**OSDI 2018**] [Deconstructing RDMA-enabled Transaction Processing: Hybrid is Better!](docs/papers/osdi-2018.pdf). Xingda Wei, Zhiyuan Dong, Rong Chen, and Haibo Chen. Proceedings of 13th USENIX Symposium on Operating Systems Design and Implementation, Carlsbad, CA, US, October 2018.[{{:publications:drtm+h-osdi18.pdf|paper}}] [[https://github.com/SJTU-IPADS/drtmh|github]] *[**USENIX ATC 2018**] Fast and Concurrent RDF Queries using RDMA-assisted GPU Graph Exploration. Siyuan Wang, Chang Lou, Rong Chen, and Haibo Chen. Proceedings of 2018 USENIX Annual Technical Conference, Boston, MA, US, July 2018. [{{:publications:wukong+g-atc18.pdf|paper}}] [{{:publications:wukong+g-atc18-slides.pptx|slides}}] *[**APSys 2018**] Analysis and Improvement of Optimizer for Query Processing on Graph Store. Youyang Yao, Jiaqi Li and Rong Chen. Proceedings of the 9th ACM SIGOPS Asia-Pacific Workshop on Systems, Jeju Island, South Korea. August 2018. [{{:publications:wukong+o-apsys18.pdf|paper}}] *[**SOSP 2017**] Sub-millisecond Stateful Stream Querying over Fast-evolving Linked Data. Yunhao Zhang, Rong Chen, and Haibo Chen. Proceedings of the 26th ACM Symposium on Operating Systems Principles, Shanghai, China, October 2017. [{{:publications:wukong-s-sosp17.pdf|updated paper}}] [{{:publications:wukong_s-sosp17posters.pdf|poster}}] [{{:publications:wukong_s-sosp17-slides.pptx|slides}}] [[http://dl.acm.org/authorize?N48222|ACM DL]] *[**USENIX ATC 2017**] [Replication-driven Live Reconfiguration for Fast Distributed Transaction Processing](docs/papers/uatc-2017.pdf). Xingda Wei, Sijie Shen, Rong Chen, and Haibo Chen. Proceedings of 2017 USENIX Annual Technical Conference, Santa Clara, CA, US, July 2017. [{{:publications:drtm3-atc17.pdf|paper}}] *[**OSDI 2016**] Fast and Concurrent RDF Queries with RDMA-based Distributed Graph Exploration. Jiaxin Shi, Youyang Yao, Rong Chen, Haibo Chen, and Feifei Li. Proceedings of 12th USENIX Symposium on Operating Systems Design and Implementation, Savannah, GA, US, November 2016. [{{:publications:wukong-osdi16.pdf|paper}}] [{{:publications:wukong-osdi16-slides.pptx|slides}}] [{{:publications:wukong-osdi16-poster.pdf|poster}}] [[http://ipads.se.sjtu.edu.cn/projects/wukong|homepage]] [[https://github.com/SJTU-IPADS/wukong|github]] *[**EuroSys 2016**] Fast and General Distributed Transactions Using RDMA and HTM. Yanzhe Chen, Xingda Wei, Jiaxin Shi, Rong Chen and Haibo Chen. Proceedings of 11th ACM European Conference on Computer Systems, London, UK, April 2016. [{{:publications:drtm2-eurosys16.pdf|paper}}] [[http://dl.acm.org/authorize?N34632 |ACM DL]] *[**SOSP 2015**] Fast In-memory Transaction Processing using RDMA and HTM. Xingda Wei, Jiaxin Shi, Yanzhe Chen, Rong Chen and Haibo Chen. Proceedings of the 25th ACM Symposium on Operating Systems Principles, Monterey, CA, USA, October 2015. [{{:publications:drtm-sosp15.pdf|paper}}] [{{:publications:drtm-sosp15-slides.pptx|slides}}] [{{:publications:drtm-sosp15-poster.pdf|poster}}] [[http://dl.acm.org/authorize?N07024|ACM DL]] \\ [[https://blog.acolyer.org/2015/10/26/fast-in-memory-transaction-processing-using-rdma-and-rtm/|Featured on "The Morning Paper"]] [[https://github.com/SJTU-IPADS/drtm|github]] ===== Acknowledgements ===== The project is supported in part by China National Natural Science Foundation (61402284, 61772335, 61572314, 61525204), the National Key Research & Development Program (No. 2016YFB1000502), the Program for New Century Excellent Talents in University of Ministry of Education of China (No.ZXZY037003), a foundation for the Author of National Excellent Doctoral Dissertation of PR China (No. TS0220103006), and research grants from Huawei Corporation (No.FA2018091021-201906).