Benchmarking Deep learning platforms by means of Tensorflow and Docker: Difference between revisions

From Lsdf
Jump to navigationJump to search
(Created page with "=== Description === The goal of the DEEP Hybrid DataCloud project [1] is to prepare a new generation of e-Infrastructures that harness latest generation technologies, supporti...")
 
No edit summary
 
(6 intermediate revisions by 2 users not shown)
Line 2: Line 2:
The goal of the DEEP Hybrid DataCloud project [1] is to prepare a new generation of e-Infrastructures that harness latest generation technologies, supporting e.g. deep learning. In order to assess our solutions for new e-Infrastructures, a set of benchmark tools is needed.
The goal of the DEEP Hybrid DataCloud project [1] is to prepare a new generation of e-Infrastructures that harness latest generation technologies, supporting e.g. deep learning. In order to assess our solutions for new e-Infrastructures, a set of benchmark tools is needed.


Aim of this student project is to prepare a set of benchmarks in form of Docker image(s) based on Tensorflow [2] and a number of pre-existing Tensorflow scripts (e.g. [3]), run them by means of various container tools and in different environments, compare performance.
Aim of this project is to prepare a set of benchmarks in form of Docker image(s) based on Tensorflow [2] and a number of pre-existing Tensorflow scripts (e.g. [3]), run them by means of various container tools and in different environments, compare performance.


=== Tasks ===
=== Tasks ===
- get acquanted with Docker containers and best practices [4, 5].
* get acquanted with Docker containers and best practices [4, 5].
- adapt existing Convolutional Neural Network (CNN) benchmarks [3] and pack them in Docker image(s) such that they can be run on DEEP testbeds.
* adapt existing Convolutional Neural Network (CNN) benchmarks [3] and pack them in Docker image(s) such that they can be run on GPUs in the DEEP testbeds.
- run these benchmarks in HPC-like environment by means of udocker and singularity and on DEEP testbeds, use syntetic and real data, compare performance with bare-metal case.
* run these benchmarks in HPC-like environment by means of udocker and singularity and on DEEP testbeds, use syntetic and real data, compare performance with bare-metal case.
- (optional) extend the benchmarks for another neural network type (neural network code will be provided). Run same experiments as for CNN scripts.
* (optional) extend the benchmarks for another neural network type (neural network code will be provided). Run same experiments as for CNN scripts.
- at the end of Praktikum you write a short report with results of the performance tests (similar to [3]).
* at the end of Praktikum you write a short report with results of the performance tests (similar to [3]).


=== Requirements ===
=== Requirements ===
- programming experience in python
* programming experience in python
- knowledge on how to install a program from source in Linux
* knowledge on how to install a program from source in Linux
- ideally you have knowledge of containers and neural networks
* ideally you have knowledge of containers and neural networks

=== Contact ===
[mailto:valentin.kozlov@kit.edu valentin.kozlov@kit.edu]


=== References ===
=== References ===
[1] https://deep-hybrid-datacloud.eu
[1] https://deep-hybrid-datacloud.eu

[2] https://www.tensorflow.org
[2] https://www.tensorflow.org

[3] https://www.tensorflow.org/performance/benchmarks
[3] https://www.tensorflow.org/performance/benchmarks

[4] https://www.docker.com
[4] https://www.docker.com

[5] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
[5] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

Latest revision as of 11:25, 9 October 2018

Description

The goal of the DEEP Hybrid DataCloud project [1] is to prepare a new generation of e-Infrastructures that harness latest generation technologies, supporting e.g. deep learning. In order to assess our solutions for new e-Infrastructures, a set of benchmark tools is needed.

Aim of this project is to prepare a set of benchmarks in form of Docker image(s) based on Tensorflow [2] and a number of pre-existing Tensorflow scripts (e.g. [3]), run them by means of various container tools and in different environments, compare performance.

Tasks

  • get acquanted with Docker containers and best practices [4, 5].
  • adapt existing Convolutional Neural Network (CNN) benchmarks [3] and pack them in Docker image(s) such that they can be run on GPUs in the DEEP testbeds.
  • run these benchmarks in HPC-like environment by means of udocker and singularity and on DEEP testbeds, use syntetic and real data, compare performance with bare-metal case.
  • (optional) extend the benchmarks for another neural network type (neural network code will be provided). Run same experiments as for CNN scripts.
  • at the end of Praktikum you write a short report with results of the performance tests (similar to [3]).

Requirements

  • programming experience in python
  • knowledge on how to install a program from source in Linux
  • ideally you have knowledge of containers and neural networks

Contact

valentin.kozlov@kit.edu

References

[1] https://deep-hybrid-datacloud.eu

[2] https://www.tensorflow.org

[3] https://www.tensorflow.org/performance/benchmarks

[4] https://www.docker.com

[5] https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

Retrieved from "https://wiki.scc.kit.edu/lsdf/index.php?title=Benchmarking_Deep_learning_platforms_by_means_of_Tensorflow_and_Docker&oldid=5224"