IPython: The awesome python shell
0 likes572 views
The document discusses IPython, an enhanced interactive Python shell with features like custom magic commands and tools for parallel computing. It describes its architecture, including engines, controllers, and clients, and highlights functionalities like task management, data scattering and gathering, and dynamic load balancing. It also mentions challenges in setting up IPython clusters over the internet due to network configurations and I/O overhead.
IPython: The awesome python shell
- 1. The AWESOME Shell http://ipython.scipy.org/
- 2. IPython is... Released under the revised BSD license
- 3. A enhanced interactive python shell Lovely introspection tools
- 4. Demo ?, ??, tab
- 5. %magic %lsmagic (edit, run, pdb)
- 6. Custom magic A architecture for interactive parallel computing
- 7. // Architecture Engines ( does the work, run any where )
- 8. Controller ( does job mangement and coordination ) MultiEngine interface
- 9. Task interface (Ideal for task farming) Clients ( people with things to compute. ) Has corresponding MultiEngineClient
- 10. TaskClient
- 11. Quick Cluster Demo Start Controller
- 12. Start Engine
- 13. Start Client
- 14. Easy Cluster Demo Ipcluster
- 15. Various cluster types Local
- 16. Ssh
- 17. Mpi
- 18. Pbs (Portiable Batch Sytem)
- 19. MultiEngine Interface Much more suited for interactive work
- 20. All engines are exposed to us
- 21. Execute funtion runs on all engines
- 22. Scatter & gather (Demo)
- 23. Map function, allows a function to be applied to a sequence. (Demo)
- 24. Task Interface Ideal base for Job farming system
- 25. Does dynamic load balancing between engines
- 26. Has a the same map function as the Multiengine, but also has the parallel decorator (Demo)
- 27. Can construct more complex Task, with StringTask
- 28. // Problems Not so easy setup when you are setting up on the Internet Lots of creating tunnels and firewall pins so your client and engines can talk to the controller IO Overhead Really sucks when you end up doing more IO than tasking the cpu.
- 29. // The hard part Asyncclient Forget all concept of linear time
- 30. All calls are non blocking
- 31. Defereds
- 32. Callbacks
- 33. errbacks