博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
在Jupyter Notebook服务器中使用远程内核
阅读量:2518 次
发布时间:2019-05-11

本文共 12740 字,大约阅读时间需要 42 分钟。

uses kernels to execute code interactively. The Jupyter runs kernels as separate processes on the same host by default. However, there are scenarios where it would be necessary or beneficial to have the Notebook server use kernels that run remotely.

使用内核以交互方式执行代码。 默认情况下,Jupyter 在同一主机上将内核作为单独的进程运行。 但是,在某些情况下,让Notebook服务器使用远程运行的内核将是必要或有益的。

A good example is when you want to use notebooks to explore and analyze large data sets on an cluster that runs in the cloud. In this case, the notebook kernel becomes the Spark driver program, and the Spark architecture dictates that drivers run as close as possible to the cluster, preferably on the same local area network [1].

一个很好的例子是,当您想使用笔记本浏览和分析在云中运行的集群上的大数据集时。 在这种情况下,笔记本内核成为Spark驱动程序,Spark架构要求驱动程序尽可能靠近群集运行,最好在同一局域网上运行[1]。

To run Jupyter Notebook with remote kernels, first you need a kernel server that exposes an API to manage and communicate with kernels. Second, you need to modify the default behavior of the Notebook server, which is to spawn kernels as local processes on the same host.

要使用远程内核运行Jupyter Notebook,首先您需要一个内核服务器,该服务器公开API以管理内核并与之通信。 其次,您需要修改Notebook服务器的默认行为,即在同一主机上将内核作为本地进程生成。

The satisifies the first need. It allows clients to provision and communicate with kernels using HTTP and web socket protocols.

第一个需求。 它允许客户端使用HTTP和Web套接字协议配置内核并与内核通信。

Jupyter Notebook 4.2 introduced to make it possible to extend (or modify) the Notebook server behavior. I leveraged this new capabilitiy and created a demo server extension to modify the Notebook server to use remote kernels hosted by the Kernel Gateway. For lack of a better name, I called it .

Jupyter Notebook 4.2引入了 ,从而可以扩展(或修改)Notebook服务器行为。 我利用了这一新功能,并创建了一个演示服务器扩展,以修改Notebook服务器以使用由内核网关托管的远程内核。 由于缺少更好的名称,我将其称为 。

We can visualize the Jupyter Notebook components in the diagram below, which was taken from the .

我们可以在下图中可视化Jupyter Notebook组件,该组件取自 。

The key points are:

关键点是:

  • The Notebook web UI (browser) and the Notebook server communicate using HTTP and web socket protocols.
  • By default, the Notebook server spawns kernels on the same host as the server.
  • The server and kernel processes communicate using .
  • Notebook Web UI(浏览器)和Notebook服务器使用HTTP和Web套接字协议进行通信。
  • 默认情况下,Notebook服务器在与服务器相同的主机上生成内核。
  • 服务器和内核进程使用进行通信。

The extension essentially proxies all kernel requests and web socket communication from the notebook web UI to a Kernel Gateway. Using the extension, the browser to server to kernel communication looks like this:

扩展实质上代理了从笔记本Web UI到内核网关的所有内核请求和Web套接字通信。 使用扩展名,浏览器到服务器到内核的通信如下所示:

试试吧 (Try It)

This section provides an example of running a Jupyter Kernel Gateway and the Jupyter Notebook server with the nb2kg extension in a environment. (The also includes Dockerfiles and a docker-compose recipe if you care to try it using Docker).

本节提供了运行一个内核Jupyter网关并将其与以所述nb2kg扩展Jupyter笔记本服务器的示例环境。 ( 还包括Dockerfile和docker docker-compose配方,如果您想尝试使用Docker的话)。

First, open a terminal and create and activate a new conda environment. Install Jupyter Notebook version 4.2 or later.

首先,打开终端并创建并激活新的conda环境。 安装Jupyter Notebook 4.2或更高版本。

conda create yes name nb2kg notebook==4.2

conda create — yes — name nb2kg notebook == 4.2

source activate nb2kg

source activate nb2kg

Use pip to install the Kernel Gateway, and start it on port 8889.

使用pip安装内核网关,并在端口8889上启动它。

pip install jupyter_kernel_gateway==0.6.0

pip install jupyter_kernel_gateway == 0.6.0

jupyter kernelgateway KernelGatewayApp.port=8889

jupyter kernelgateway — KernelGatewayApp . port = 8889

Open a second terminal and attach to the nb2kg conda environment.

打开第二个终端,并连接到nb2kg环境。

1

1

source activate nb2kg

source activate nb2kg

Install and enable the nb2kg Jupyter Notebook server extension.

安装并启用nb2kg Jupyter Notebook服务器扩展。

pip install “git+https://github.com/jupyter/kernel_gateway_demos.git#egg=nb2kg&subdirectory=nb2kg”

pip install “git+https://github.com/jupyter/kernel_gateway_demos.git#egg=nb2kg&subdirectory=nb2kg”

jupyter serverextension enable py nb2kg sysprefix

jupyter serverextension enable — py nb2kg — sys – prefix

You can ensure that the extension is enabled using:

您可以使用以下方法确保启用扩展:

config dir: /Users/jtyberg/anaconda3/envs/nb2kg/etc/jupyter nb2kg enabled – Validating… nb2kg OK

config dir: /Users/jtyberg/anaconda3/envs/nb2kg/etc/jupyter nb2kg enabled – Validating… nb2kg OK

1

1

2

2

3

3

4

4

5

5

6

6

jupyter serverextension list

jupyter serverextension list

config dir: /Users/jtyberg/anaconda3/envs/nb2kg/etc/jupyter

config dir : / Users / jtyberg / anaconda3 / envs / nb2kg / etc / jupyter

    nb2kg  enabled

     nb2kg   enabled

     Validating...

     – Validating . . .

      nb2kg  OK

       nb2kg   OK

Finally, start the notebook server. The nb2kg extension requires that you set the KG_URL environment variable to the URL of the Kernel Gateway. It also requires that you override the default session, kernel, and kernel spec managers when you start the notebook.

最后,启动笔记本服务器。 nb2kg扩展名要求您将KG_URL环境变量设置为Kernel Gateway的URL。 它还要求您在启动笔记本计算机时覆盖默认的会话,内核和内核规范管理器。

export KG_URL=http://127.0.0.1:8889

export KG_URL = http : //127.0.0.1:8889

jupyter notebook

jupyter notebook

  NotebookApp.session_manager_class=nb2kg.managers.SessionManager

   — NotebookApp . session_manager_class = nb2kg . managers . SessionManager

  NotebookApp.kernel_manager_class=nb2kg.managers.RemoteKernelManager

   — NotebookApp . kernel_manager_class = nb2kg . managers . RemoteKernelManager

  NotebookApp.kernel_spec_manager_class=nb2kg.managers.RemoteKernelSpecManager

   — NotebookApp . kernel_spec_manager_class = nb2kg . managers . RemoteKernelSpecManager

If things are working properly, you should see output similar to:

如果一切正常,您应该看到类似以下的输出:

1

1

2

2

3

3

4

4

5

5

6

6

7

7

8

8

9

9

10

10

11

11

12

12

13

13

[W 20:09:18.462 NotebookApp] Unrecognized JSON config file version, assuming version 1

[ W 20 : 09 : 18.462 NotebookApp ] Unrecognized JSON config file version , assuming version 1

[W 20:09:18.463 NotebookApp] Unrecognized JSON config file version, assuming version 1

[ W 20 : 09 : 18.463 NotebookApp ] Unrecognized JSON config file version , assuming version 1

[I 20:09:18.897 NotebookApp] Loaded nb2kg extension

[ I 20 : 09 : 18.897 NotebookApp ] Loaded nb2kg extension

[I 20:09:18.898 NotebookApp] Overriding handler URLSpec(‘/api/kernelspecs/(?P[w.-]+)$’, , kwargs={

}, name=None)

[ I 20 : 09 : 18.898 NotebookApp ] Overriding handler URLSpec ( ‘/api/kernelspecs/(?P[w.-]+)$’ , , kwargs = { } , name = None )

[I 20:09:18.898 NotebookApp] Overriding handler URLSpec(‘/api/kernelspecs$’, , kwargs={

}, name=None)

[ I 20 : 09 : 18.898 NotebookApp ] Overriding handler URLSpec ( ‘/api/kernelspecs$’ , , kwargs = { } , name = None )

[I 20:09:18.898 NotebookApp] Overriding handler URLSpec(‘/api/kernels/(?Pw+-w+-w+-w+-w+)/channels$’, , kwargs={

}, name=None)

[ I 20 : 09 : 18.898 NotebookApp ] Overriding handler URLSpec ( ‘/api/kernels/(?Pw+-w+-w+-w+-w+)/channels$’ , , kwargs = { } , name = None )

[I 20:09:18.898 NotebookApp] Overriding handler URLSpec(‘/api/kernels/(?Pw+-w+-w+-w+-w+)/(?Prestart|interrupt)$’, , kwargs={

}, name=None)

[ I 20 : 09 : 18.898 NotebookApp ] Overriding handler URLSpec ( ‘/api/kernels/(?Pw+-w+-w+-w+-w+)/(?Prestart|interrupt)$’ , , kwargs = { } , name = None )

[I 20:09:18.898 NotebookApp] Overriding handler URLSpec(‘/api/kernels/(?Pw+-w+-w+-w+-w+)$’, , kwargs={

}, name=None)

[ I 20 : 09 : 18.898 NotebookApp ] Overriding handler URLSpec ( ‘/api/kernels/(?Pw+-w+-w+-w+-w+)$’ , , kwargs = { } , name = None )

[I 20:09:18.898 NotebookApp] Overriding handler URLSpec(‘/api/kernels$’, , kwargs={

}, name=None)

[ I 20 : 09 : 18.898 NotebookApp ] Overriding handler URLSpec ( ‘/api/kernels$’ , , kwargs = { } , name = None )

[I 20:09:18.902 NotebookApp] Serving notebooks from local directory: /Users/jtyberg

[ I 20 : 09 : 18.902 NotebookApp ] Serving notebooks from local directory : / Users / jtyberg

[I 20:09:18.902 NotebookApp] 0 active kernels

[ I 20 : 09 : 18.902 NotebookApp ] 0 active kernels

[I 20:09:18.902 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/

[ I 20 : 09 : 18.902 NotebookApp ] The Jupyter Notebook is running at : http : //localhost:8888/

[I 20:09:18.902 NotebookApp] Use ControlC to stop this server and shut down all kernels (twice to skip confirmation).

[ I 20 : 09 : 18.902 NotebookApp ] Use Control – C to stop this server and shut down all kernels ( twice to skip confirmation ) .

A web browser window should open to the notebook server home page (in this case http://localhost:8888/tree). You should be able to create and use notebooks as normal, with the following caveats:

Web浏览器窗口应打开到笔记本服务器的主页(在本例中为http://localhost:8888/tree )。 您应该能够正常创建和使用笔记本,但要注意以下几点:

  • When you enable the nb2kg extension, all kernels run on the configured Kernel Gateway, instead of on the Notebook server host. (The extension does not support local kernels).
  • Keep in mind that notebooks and other files reside on the Notebook server, and remote kernels may not be able to access them.
  • 启用nb2kg扩展名后,所有内核都在配置的Kernel Gateway上运行,而不是在Notebook服务器主机上运行。 (该扩展不支持本地内核)。
  • 请记住,笔记本和其他文件位于笔记本服务器上,并且远程内核可能无法访问它们。

翻译自:

转载地址:http://keqwd.baihongyu.com/

你可能感兴趣的文章
installing the numpy via pip in the enviroment dos
查看>>
ASP.NET MVC中如何在当前页面上弹出另外一个页面
查看>>
VS2015 MSVC编译FFMPEG
查看>>
红外图像盲元补偿matlab实现源码与效果验证
查看>>
Linux环境下部署SpringBoot项目
查看>>
C# 通过反射来动态创建泛型类型
查看>>
git bash中不能显示中文
查看>>
常用算法
查看>>
SQL算术数字的默认类型
查看>>
正则表达式
查看>>
TCP/IP Socket
查看>>
How do I debug a published XBAP file in VS2010?
查看>>
单工、半双工和全双工的定义
查看>>
Hdu【线段树】基础题.cpp
查看>>
时钟系统
查看>>
BiTree
查看>>
5个基于HTML5的加载动画推荐
查看>>
水平权限漏洞的修复方案
查看>>
静态链接与动态链接的区别
查看>>
Android 关于悬浮窗权限的问题
查看>>