Topic IV

In this video lecture, the presenter introduces the Desk Python module, which is an open-source Python library for parallel computing. The module allows scaling Python code from a local machine to large clusters. The presenter explains that Desk arrays are structures composed of chunks of non-Py arrays, while Desk data frames are composed of smaller pandas data frames. The presenter demonstrates the benefits of using Desk by showing an example of creating a large NumPy array and encountering a memory allocation error due to insufficient RAM. Is is subsequently show how Desk can solve this issue by setting up a Desk client with specific configurations for processes, threads, and memory usage. By using Desk arrays, computations on the large data arrays can be performed.