MongoDB as an In-memory Sharded Database: Difference between revisions
(More structure) |
(Pari's suggestions) |
||
| Line 2: | Line 2: | ||
= Description = |
= Description = |
||
The Data Life-Cycle Lab Earth and Environment at KIT manages data from Earth-observing climate satellites such as Envisat MIPAS |
The Data Life-Cycle Lab Earth and Environment at KIT manages data from instruments set up on Earth-observing climate satellites, such as Envisat MIPAS. The corresponding data is stored in the NoSQL database MongoDB. This large dataset should be handled in a distributed database cluster, using sharding - a horizontal-partitioning solution available in MongoDB. |
||
A typical MongoDB instance makes heavy use of available system memory but ultimately relies on underlying persistent storage. At the same time, our database cluster offers only low-performance persistent storage which is unsuitable for sustained load. It is therefore required to have MongoDB on our cluster operate as an *in-memory database*. |
A typical MongoDB instance makes heavy use of available system memory but ultimately relies on underlying persistent storage. At the same time, our database cluster offers only low-performance persistent storage which is unsuitable for sustained load. It is therefore required to have MongoDB on our cluster operate as an *in-memory database*. |
||
Revision as of 15:26, 9 February 2016
Description
The Data Life-Cycle Lab Earth and Environment at KIT manages data from instruments set up on Earth-observing climate satellites, such as Envisat MIPAS. The corresponding data is stored in the NoSQL database MongoDB. This large dataset should be handled in a distributed database cluster, using sharding - a horizontal-partitioning solution available in MongoDB.
A typical MongoDB instance makes heavy use of available system memory but ultimately relies on underlying persistent storage. At the same time, our database cluster offers only low-performance persistent storage which is unsuitable for sustained load. It is therefore required to have MongoDB on our cluster operate as an *in-memory database*.
Task
Your task will be to to research optimal configuration of MongoDB for in-memory operation, develop tools for the initial population of cluster nodes as well as periodic commits of their data to persistent storage, and finally to evaluate the performance of the system.
Requirements
- basic administration of Linux/Unix systems
- good working knowledge of Python, Node.js JavaScript or other scripting language capable of interfacing with MongoDB
- familiarity with MongoDB and/or sharding would be a plus
Contact
Marek.Szuba@kit.edu - 29178
