Building custom kernels for IPython
3 likes1,629 views
This document discusses building custom kernels for IPython. It begins by explaining what an IPython kernel is and how kernels work. It describes the benefits of building custom kernels for other languages that lack interactive development tools. The architecture of IPython is explained, with the kernel and clients communicating over ZeroMQ. The two types of kernels are discussed. The messaging specification and channels are covered at a high level. Finally, the document focuses on building wrapper kernels by extending the Kernel base class and implementing specific methods, using examples like a Bash and Redis kernel.
Building custom kernels for IPython
- 1. Building a custom kernel for IPython Hari Allamraju [email protected] https://github.com/hari-allamraju remitplan.com rememberthebudget.in
- 2. What this talk is about • What is an IPython kernel • How does an IPython kernel work • How can you build a simple wrapper kernel • Some pointers on how to build a full fledged native kernel
- 3. Why build a kernel? • IPython does a lot of neat magic which makes it invaluable for Python development • There are no such tools for other languages, or even applications which could benefit from having a nice interactive front end • We are developers, we can build such tools and share them • It’s fun to take things apart and see how they work and could be the first step to you contributing to IPython itself
- 4. IPython architecture Distributed client and kernel components talking over zeromq; separation of client and kernel means you can add any client to the same kernel
- 5. Notebook is just another client This can be extended to any client in any language or interface
- 6. The two types of kernels Changes after the recent Jupyter project
- 7. IPython messaging zeromq sockets; all clients connected to a kernel receive all messages and share the kernel context
- 8. Messaging spec • You can find more details here - https://ipython.org/ipython- doc/dev/development/messaging.html • 4 types of channels • Shell: request reply channel where all execution happens • IOPub: this is where the kernel broadcasts the side effects of any request to all clients • Stdin: this is where the kernel requests input from the clients • Control: for shutdown, abort etc
- 9. { # The message header contains a pair of unique identifiers for the # originating session and the actual message id, in addition to the # username for the process that generated the message. This is useful in # collaborative settings where multiple users may be interacting with the # same kernel simultaneously, so that frontends can label the various # messages in a meaningful way. 'header' : { 'msg_id' : uuid, 'username' : str, 'session' : uuid, # All recognized message type strings are listed below. 'msg_type' : str, # the message protocol version 'version' : '5.0', }, # In a chain of messages, the header from the parent is copied so that # clients can track where messages come from. 'parent_header' : dict, # Any metadata associated with the message. 'metadata' : dict, # The actual content of the message must be a dict, whose structure # depends on the message type. 'content' : dict, }
- 10. Full native kernel • You have to handle all the zeromq sockets and communication mechanism • You have to ensure that the IPython messages are created and parsed correctly for each type of request - if you don’t implement something then you need to handle that gracefully • You have the freedom to structure your code as you see fit • https://github.com/dsblank/simple_kernel
- 11. Wrapper kernels • Use IPython machinery to start the kernel and the various zeromq channels which will be used to communicate with the clients • Provide the ability to create and parse the messages as per the IPython messaging spec • Give you specific end points or methods to implement without having to worry about how it will all fit together • https://github.com/takluyver/bash_kernel
- 12. Our focus - Wrapper kernels
- 13. The base class • from IPython.kernel.zmq.kernelbase import Kernel • Available from IPython 3 • Provides you all the scaffolding needed to build the kernel
- 14. The main properties implementation implementation_version language language_version banner Information for Kernel info replies. ‘Implementation’ refers to the kernel (e.g. IPython), and ‘language’ refers to the language it interprets (e.g. Python). The ‘banner’ is displayed to the user in console UIs before the first prompt. All of these values are strings. language_info Language information for Kernel info replies, in a dictionary. This should contain the key mimetype with the mimetype of code in the target language (e.g. 'text/x-python'), and file_extension (e.g. 'py'). It may also contain keys codemirror_mode and pygments_lexer if they need to differ from language. Other keys may be added to this later. https://ipython.org/ipython-doc/dev/development/messaging.html#msging-kernel-info
- 15. The main method do_execute(code, silent, store_history=True, user_expressions=None, allow_stdin=False) Execute user code. Parameters: code (str) – The code to be executed. silent (bool) – Whether to display output. store_history (bool) – Whether to record this code in history and increase the execution count. If silent is True, this is implicitly False. user_expressions (dict) – Mapping of names to expressions to evaluate after the code has run. You can ignore this if you need to. allow_stdin (bool) – Whether the frontend can provide input on request (e.g. for Python’s raw_input()). Your method should return a dict containing the fields described in Execution results. To display output, it can send messages using send_response(). See Messaging in IPython for details of the different message types.
- 16. Launching the kernel if __name__ == '__main__': from IPython.kernel.zmq.kernelapp import IPKernelApp IPKernelApp.launch_instance(kernel_class=MyKernel)
- 17. Let’s see some code! • Some simple examples - echo and nodejs (with pexpect) • https://github.com/supercoderz/redis_kernel • This is a wrapper kernel, connects to redis and executes redis commands • Almost complete in terms of functionality
- 18. Let’s see some code! • Code overview of redis kernel • More methods - https://ipython.org/ipython-doc/ dev/development/wrapperkernels.html • This might change with Jupyter, still have to test redis_kernel with Jupyter
- 19. Thank you! Any questions? Or you can reach me at [email protected]