Tsinghua University: distributed storage helps China Brain Project

Background:

Brain science, the apex of human cognition, is of great significance for advancing human cognitive ability, diagnosis and treatment of encephalopathy, and AI technology. The brain is an active area of research; and the United States and EU have launched large-scale research projects. China Brain Project is initiated in 2016 and is considered as the top research project in China.

Brain science aims to build a brain circuit diagram to reveal the mapping of connections between neurons. However, there are tens of billions of neurons in the cerebral cortex, and each neuron contains more than a thousand information sending and receiving branches. Due to the data throughput constraint of the current microscope, high-resolution whole-brain dynamic imaging is impossible; and to develop a precise brain circuit diagram is recognized as a global challenge.

In 2018, RUSH (real-time ultra-large-scale imaging at high-resolution) is developed by the team led by Dai Qionghai from Tsinghua University, which is a critical breakthrough. The RUSH platform supports multi-dimensional continuous shooting and the specifications are FoV (1 × 1.2cm), resolution (800nm), frame rate (30FPS), spatial and temporal resolution (169MP). The data throughput is increased from 10-megapixel per second to 5-gigapixel per second, enabling multi-scale observations that involve details of the entire brain of living animals. The FoV is 120 times of commercial microscopes with the same resolution and 3.3 times of Mesolens. The data throughput is more than 500 times of 2p-RAM which is the world’s second largest high-throughput microscope. This is of strategic importance to push forward life and medical sciences and strengthen the research and application capacity of the China Brain Project.

Challenge:

Reliability: high sensitivity, high resolution camera needs continuous shooting, can not lose frame

Bandwidth: the minimum bandwidth of rush must reach 20.16GB/s

Large capacity: the data amount of continuous tens of hours observation reaches PB

Solution:

Concurrency and scalability of the storage system are two key elements to support the RUSH platform. Through public bidding and rigorous testing, Inspur distributed storage AS13000 stood out and won the bid. A 40-node storage unit is applied to control 28 cameras for concurrent collection and storage of high-throughput data. Continuous shooting (12MP photos, 30FPS, 16-bit) is enabled. Storage bandwidth is stabilized at 20.16GB/s, maximum shooting duration is 48hours, and the storage capacity is around 5PB.

Benefit:

In response to the application characteristics of concurrent collection and storage of high-throughput data, as well as the requirements for the performance, bandwidth and scalability of the storage system, Inspur customized the AS13000 solution and conducted a simulation to measure the data traffic and operation time. It is noted that data traffic was increased in the simulation to guarantee that the system can handle future business expansion, and to test the load capacity and performance changes of the system and platform. Finally, the AS13000 solution was verified to have stable read bandwidth of 45GB/s and stable write bandwidth of 35GB/s, which fully meets the storage needs of the RUSH platform. The solution helps safeguard the China Brain Project and shows Inspur’s power in distributed storage.