Difference between revisions of "Large-scale visualisation/analysis platform for climate data"

From Lsdf
(Created page with "= Description = Many analysis tasks in climate research are mean and/or variance calculations over small (but many) netcdf files. Currently these analysis tasks are done in Py...")
 
Line 1: Line 1:
 
= Description =
  
= Description =
−
Many analysis tasks in climate research are mean and/or variance calculations over small (but many) netcdf files. Currently these analysis tasks are done in Python using [http://www.numpy.org/ numpy] and [http://xarray.pydata.org/en/stable/ xarray].
 +
Many analysis tasks in climate research are mean and/or variance calculations over small (but many) netcdf files. Currently these
  +
analysis tasks are done in Python using [http://www.numpy.org/ numpy] and [http://xarray.pydata.org/en/stable/ xarray]. The overarching goal of this project is to build an interactive web-framework for data analysis and discovery using the Python Visualization library [http://pyviz.org/ PyViz]. The groundwork for such a framework is already laid out in a former [https://github.com/ucyo/praktClimaAnalyse Praktikum].
   
  +
= Task =
−
The goal of this project is to develop an interactive web-framework/interface to these basic analysis frameworks using [https://plot.ly/python/ plotly], [https://mpld3.github.io/examples/index.html#example-gallery mpl3d]
  
  +
The task for this semester is to integrate data streaming technology for visualization. There are several techniques already supported by the underlying technology e.g. AJAX data sources.
  +
Further it is possible to use RPC on different data hosting machines to move the computation to the data.
   
 
= Tasks =
  
= Tasks =
−
* getting familiar with the firmware update procedures of different hardware vendors like e.g. IBM, HP, Dell, SuperMicro
 +
* getting familiar with PyViz and the underlying technologies e.g. Bokeh
−
* implement scripts that use the firmware update tools provided by the vendors to check for available updates
 +
* implement scripts that enhance the current framwork with streaming technologies such as AJAX
  +
* look into possibilties to use RPC for moving the computation to the data
−
* integrate the scripts into the SCC monitoring system
  
   
 
= Requirements =
  
= Requirements =
−
* Programming experience in Python and HTML as well as Javascript
 +
* Programming experience in Python and some Javascript
   
 
= Contact =
  
= Contact =

Revision as of 09:51, 8 April 2019

Description

Many analysis tasks in climate research are mean and/or variance calculations over small (but many) netcdf files. Currently these analysis tasks are done in Python using numpy and xarray. The overarching goal of this project is to build an interactive web-framework for data analysis and discovery using the Python Visualization library PyViz. The groundwork for such a framework is already laid out in a former Praktikum.

Task

The task for this semester is to integrate data streaming technology for visualization. There are several techniques already supported by the underlying technology e.g. AJAX data sources. Further it is possible to use RPC on different data hosting machines to move the computation to the data.

Tasks

  • getting familiar with PyViz and the underlying technologies e.g. Bokeh
  • implement scripts that enhance the current framwork with streaming technologies such as AJAX
  • look into possibilties to use RPC for moving the computation to the data

Requirements

  • Programming experience in Python and some Javascript

Contact

Cayoglu@kit.edu

Retrieved from "https://wiki.scc.kit.edu/lsdf/index.php?title=Large-scale_visualisation/analysis_platform_for_climate_data&oldid=5292"