Identify notebook file being run

[aggFTW]

Hi,

I've seen this type of question a lot:
http://stackoverflow.com/questions/20050927/how-to-get-the-ipython-notebook-title-associated-with-the-currently-running-ipyt?rq=1

It makes sense to me that the kernel should not know what it's talking to from a design perspective.

However, I'm currently in the process of working through a Jupyter High Availability scenario. Our goal is to have two Jupyter instances running in two different VMs and switch them if one of those two VMs go down for some reason without losing the kernel state.

We have control over the kernels we are running (see https://github.com/jupyter-incubator/sparkmagic/blob/master/remotespark/wrapperkernel/sparkkernelbase.py), and we'd like to be able to tie some state (a session number) to a particular kernel instance.

It seems to me like I'd need some things to achieve this, but maybe you have better ideas:

• Fire some piece of code automatically every time a notebook starts: this could be the __init__ method in my kernel or some other piece of code that is triggered every time a kernel gets started (some Javascript code in the notebook maybe? I know this wouldn't apply for other clients but it's a start).
• This previous bit of code that gets fired would need to always be run with the same ID to be able to identify the state it needs to reconstruct (i.e. it would need to know that for this particular kernel we had X particular state).
• Some persistent storage that both Jupyter instances could have access to.

I thought of a concrete implementation and I'd like to hear some feedback on it if possible:
There is a Notebook extension that reads some ID in the notebook's page DOM (I need help knowing what ID this would be: e.g. notebook name with relative paths from root folder included or a GUID in some hidden cell in the notebook file), which would then issue a request to the kernel with this ID to restore its state. The kernel would then take this ID and get the session ID from cloud storage. If the ID is embedded in Javascript, both Jupyter servers would need to trust the notebook from the get go.

Thanks for any help or pointers you may have!
(cc. @msftristew, @MohamedElKamhawy, @ellisonbg)

[aggFTW]

cc @Carreau and @jdfreder

[minrk]

A custom KernelManager could add an environment variable when a kernel is started, though the KernelManager doesn't have access to the notebook path. A SessionManager could pass that down, though it wouldn't be updated when the notebook is renamed, so a filename is probably not the best key to use.

[jdfreder]

You can put a GUID in the notebook-level metadata. I think you can do it without JS, at the web server level, on new or existing notebook load.

[jdfreder]

--- oh, this is issue #1000 ! 🍰 🎉

[Carreau]

:-P

[Carreau]

Wouldn't a custom MappingKernelManager that store the various kernel-models in a shared DB we enough ? (or I miss something about the notebook name).

It is highly unlikely that the notebook would be renamed during the swap of VMs.

There might need some extra logic for clean startup/exit/restart, but that should be able to resume connections.

[msftristew]

So, I've picked up this work where @aggFTW left off. I think this is how we're thinking about doing this:

1. Use a custom SessionManager that passes down the notebook name as an argument to the MappingKernelManager.
2. Use a custom KernelManager that communicates the notebook name to the new kernel process on startup (through an environment variable or some other method).
3. Our custom kernels will take the notebook name as a key and will update their metadata as appropriate in the way that @aggFTW described above.
4. Use a custom ContentsManager to update the metadata necessary for resuming stale sessions when a method is renamed.

Item (4) will certainly be an internal extension to Jupyter for us, but we were wondering whether items (1) and (2) would have any chance of being accepted upstream. I understand that the kernel not knowing what's talking to it is part of the design, but it seems like it would be generally useful (not just for this scenario) if kernels could be made aware what the name of their notebook is either through an environment variable, a command-line argument, or a 0mq message. Do you suppose there would be any interest in that PR?

[minrk]

I think it is generally useful, and we should probably do it. An environment variable is the way to go, I think. The only disadvantage of that is that you cannot update the file location on rename after the kernel has started, but a zmq message updating the file doesn't seem like the right thing to do, to me.

[olgabot]

Was this ever resolved? I'm making output and figure folders based off of the name of the notebooks and this code works in the notebooks, but when I

from IPython.core.display import Javascript
from IPython.display import display


def get_notebook_name():
    """Returns the name of the current notebook as a string
    
    From From https://mail.scipy.org/pipermail/ipython-dev/2014-June/014096.html
    """
    display(Javascript('IPython.notebook.kernel.execute("theNotebook = " + \
    "\'"+IPython.notebook.notebook_name+"\'");'))
    return theNotebook

But when I move it into a common.py file so it can be accessed across all notebooks, I get a NameError:

Is this because the .py file has no notebook? Is there a way to get the .py file to recognize the notebook it is being called from?

[Carreau]

display(Javascript('IPython.notebook.kernel.execute("theNotebook = " + \
"\'"+IPython.notebook.notebook_name+"\'");'))
## Here are dragons. 
return theNotebook

Handwaving:

The display javascript will take some time to reach the browser, and it will take some time execute the JS and get back to the kernel.

During this time IPython have have to continue executing code, so try to "return theNotebook" which is undefined. So it raise. even if you could "Wait for the JS to execute" you could not set the name of the notebook before returning the function .

Does that make some sens ?

[takluyver]

The JS sets the name in the main user namespace. When the function is moved into a module, it's looking in the module namespace, so it never sees that name. But that function is a hack, and I wouldn't rely on it in any case.

[natbusa]

ok, maybe this would sound silly, but would it be enough to add the ipynb filename in the metadata section of the notebook data structure when it's read? the field should not be stored in file but only updated once read in memory. - a sort of ephemeral metadata info

[natbusa]

I see it looks like the kernel is completely agnostic to the concept of file and it just processes cells data. I would say that the only options are indeed env variables or passing the filename during the creation of the kernel if any filename is available at that point.