Benchmarking Deep learning platforms by means of Tensorflow and Docker: Difference between revisions
From Lsdf
Jump to navigationJump to search
No edit summary |
No edit summary |
||
(4 intermediate revisions by 2 users not shown) | |||
Line 2: | Line 2: | ||
The goal of the DEEP Hybrid DataCloud project [1] is to prepare a new generation of e-Infrastructures that harness latest generation technologies, supporting e.g. deep learning. In order to assess our solutions for new e-Infrastructures, a set of benchmark tools is needed. |
The goal of the DEEP Hybrid DataCloud project [1] is to prepare a new generation of e-Infrastructures that harness latest generation technologies, supporting e.g. deep learning. In order to assess our solutions for new e-Infrastructures, a set of benchmark tools is needed. |
||
Aim of this |
Aim of this project is to prepare a set of benchmarks in form of Docker image(s) based on Tensorflow [2] and a number of pre-existing Tensorflow scripts (e.g. [3]), run them by means of various container tools and in different environments, compare performance. |
||
=== Tasks === |
=== Tasks === |
||
* get acquanted with Docker containers and best practices [4, 5]. |
* get acquanted with Docker containers and best practices [4, 5]. |
||
* adapt existing Convolutional Neural Network (CNN) benchmarks [3] and pack them in Docker image(s) such that they can be run on DEEP testbeds. |
* adapt existing Convolutional Neural Network (CNN) benchmarks [3] and pack them in Docker image(s) such that they can be run on GPUs in the DEEP testbeds. |
||
* run these benchmarks in HPC-like environment by means of udocker and singularity and on DEEP testbeds, use syntetic and real data, compare performance with bare-metal case. |
* run these benchmarks in HPC-like environment by means of udocker and singularity and on DEEP testbeds, use syntetic and real data, compare performance with bare-metal case. |
||
* (optional) extend the benchmarks for another neural network type (neural network code will be provided). Run same experiments as for CNN scripts. |
* (optional) extend the benchmarks for another neural network type (neural network code will be provided). Run same experiments as for CNN scripts. |
||
Line 15: | Line 15: | ||
* knowledge on how to install a program from source in Linux |
* knowledge on how to install a program from source in Linux |
||
* ideally you have knowledge of containers and neural networks |
* ideally you have knowledge of containers and neural networks |
||
=== Contact === |
|||
[mailto:valentin.kozlov@kit.edu valentin.kozlov@kit.edu] |
|||
=== References === |
=== References === |
||
[1] https://deep-hybrid-datacloud.eu |
[1] https://deep-hybrid-datacloud.eu |
||
[2] https://www.tensorflow.org |
[2] https://www.tensorflow.org |
||
[3] https://www.tensorflow.org/performance/benchmarks |
[3] https://www.tensorflow.org/performance/benchmarks |
||
[4] https://www.docker.com |
[4] https://www.docker.com |
||
[5] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/ |
[5] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/ |
Latest revision as of 11:25, 9 October 2018
Description
The goal of the DEEP Hybrid DataCloud project [1] is to prepare a new generation of e-Infrastructures that harness latest generation technologies, supporting e.g. deep learning. In order to assess our solutions for new e-Infrastructures, a set of benchmark tools is needed.
Aim of this project is to prepare a set of benchmarks in form of Docker image(s) based on Tensorflow [2] and a number of pre-existing Tensorflow scripts (e.g. [3]), run them by means of various container tools and in different environments, compare performance.
Tasks
- get acquanted with Docker containers and best practices [4, 5].
- adapt existing Convolutional Neural Network (CNN) benchmarks [3] and pack them in Docker image(s) such that they can be run on GPUs in the DEEP testbeds.
- run these benchmarks in HPC-like environment by means of udocker and singularity and on DEEP testbeds, use syntetic and real data, compare performance with bare-metal case.
- (optional) extend the benchmarks for another neural network type (neural network code will be provided). Run same experiments as for CNN scripts.
- at the end of Praktikum you write a short report with results of the performance tests (similar to [3]).
Requirements
- programming experience in python
- knowledge on how to install a program from source in Linux
- ideally you have knowledge of containers and neural networks
Contact
References
[1] https://deep-hybrid-datacloud.eu
[2] https://www.tensorflow.org
[3] https://www.tensorflow.org/performance/benchmarks
[5] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/