Meta-ticket: Improve live documentation in the Jupyter notebook

[videlec]

Since #20690 we have access to documentation (with live execution)
within the Jupyter notebook. It was decided to keep the code simple.
There are two ongoing projects within Jupyter for having this working.
It might be wiser to wait for them...

Some of the improvements mentioned in #20690 were:

• [ ] Configure Sphinx to add a small header to our html page with
  Thebe configuration: use the Jupyter instance serving the page.
  We currently use window.location.origin; is this the right thing?

• Currently it takes 10s for 100 prompts while some sage files
  contain up to 1000 prompts. Profile Thebe and optimize it or use
  a separate thread to properly support large files.

• Expand the activate button with a menu or other widgets for user customization
  of the Jupyter server. This typically would let the user choose between:

  • tmpnb (will only be useful for Sage
    when tmpnb will include a Sage kernel; there are ongoing discussions for providing an alternative sage-enabled tmpnb)
  • a local Jupyter server,
  • whichever Jupyter server the browser is currently connected to?
  • a user-specified server.

• Check whether Jupyter could be configured to dynamically
  negotiate incoming connections that don't fall within the
  -NotebookApp.allow_origin pattern, by opening a user dialog such
  as "Page xxx requests starting a new kernel on this server; do
  you accept? yes/no/always for this site".

• Add support in Thebe for customizable (continuation) prompts, with striping and splitting as above, and automatic setting of the kernel. The customization option could look like:

  prompts = {
      "sage: ": {continuations=["....:", "...  "], kernel="sagemath"},
      ">>>> ": {continuation="...  ", kernel="python"}
  }
  

• Have Thebe provide good TAB-completion.

• Let the cursor switch to the next cell after evaluation (SHIFT+ENTER).

• Extract support for Thebe as a Sphinx extension with (see the related sage-cell extension):

  • Explicit markup to specify which code blocks are editable and with
    which kernel, or setup kernel auto-detection from the prompt.
  • The possibility for setting a default value for the above.

• Refactor the Sage Sphinx configuration to use the above.

• Configure sage-sample / sage-package to use it, to enable live documentation for Sage packages.

• Add support in Thebe for basic export to Jupyter notebooks. A
  quality loss (in particular in terms of the hierarchical
  structure) is acceptable.

• Allow to save the current state of the webpage as a .ipynb worksheet, to be able to continue working on it later, see [[/ticket/20690#comment:53]].

Tickets:

• Make Sage documentation functional #33309 Make Sage documentation functional (disables use of old thebe)
• Remove old thebe package #33529 Remove old thebe package
• Provide live documentation support in Jupyter using Thebe #24593 Replace Thebe by ThebeLab for live documentation support
• sagemath_doc_html: Use jupyter-sphinx for 3D graphics #33507 sagemath_doc_html: Use jupyter-sphinx for 3D graphics
• Rework on live sage documentation #33320 Rework on Thebe for live sage documentation

See also:

• Meta-ticket Meta-ticket: Sage docbuild #20080: Sage docbuild

Depends on #20690

CC: @vbraun @rbeezer @nthiery @hivert @slel @sagetrac-fcayre @jdemeyer @dimpase @kwankyu

Component: documentation

Keywords: thebe, thebelab

Issue created by migration from https://trac.sagemath.org/ticket/20893

[videlec]

Description changed:

--- 
+++ 
@@ -8,9 +8,7 @@
       contain up to 1000 prompts. Profile Thebe and optimize it or use
       a separate thread to properly support large files.
 
-- [ ] Expand the activate button with a menu or other widgets for user
-      customization of the Jupyter server. This typically would
-      let the user choose between:
+- [ ] Expand the activate button with a menu or other widgets for user customization of the Jupyter server. This typically would let the user choose between:
       - tmpnb (will only be useful for Sage when tmpnb will include a
         Sage kernel)
       - A local Jupyter server
@@ -23,9 +21,7 @@
       as "Page xxx requests starting a new kernel on this server; do
       you accept? yes/no/always for this site"
 
-- [ ] Add support in Thebe for customizable (continuation) prompts, with
-      striping and splitting as above, and automatic setting of the kernel.
-      The customization option could look like:
+- [ ] Add support in Thebe for customizable (continuation) prompts, with striping and splitting as above, and automatic setting of the kernel. The customization option could look like:
 
       ```
          prompts = {

[sagetrac-tmonteil]

Description changed:

--- 
+++ 
@@ -43,3 +43,6 @@
 - [ ] Add support in Thebe for basic export to Jupyter notebooks. A
       quality loss (in particular in terms of the hierarchical
       structure) is acceptable.
+
+- [ ] Have thebe.js provide good TAB completion.
+

[slel]

Description changed:

--- 
+++ 
@@ -2,47 +2,48 @@
 
 Some of the improvements mentioned in #20690 were:
 
-- [  ] Configure Sphinx to add a small header to our html page with Thebe configuration: use the Jupyter instance serving the page. We currently use `window.location.origin`; is this the right thing?
+- [  ] Configure Sphinx to add a small header to our html page with
+  Thebe configuration: use the Jupyter instance serving the page.
+  We currently use `window.location.origin`; is this the right thing?
 
 - [ ] Currently it takes 10s for 100 prompts while some sage files
-      contain up to 1000 prompts. Profile Thebe and optimize it or use
-      a separate thread to properly support large files.
+  contain up to 1000 prompts. Profile Thebe and optimize it or use
+  a separate thread to properly support large files.
 
-- [ ] Expand the activate button with a menu or other widgets for user customization of the Jupyter server. This typically would let the user choose between:
-      - tmpnb (will only be useful for Sage when tmpnb will include a
-        Sage kernel)
-      - A local Jupyter server
-      - Whichever Jupyter server the browser is currently connected to?
-      - a user specified server
+- [ ] Expand the activate button with a menu or other widgets for user customization
+  of the Jupyter server. This typically would let the user choose between:
+    - tmpnb (will only be useful for Sage when tmpnb will include a Sage kernel),
+    - a local Jupyter server,
+    - whichever Jupyter server the browser is currently connected to?
+    - a user-specified server.
 
 - [ ] Check whether Jupyter could be configured to dynamically
-      negotiate incoming connections that don't fall within the
-      `-NotebookApp.allow_origin pattern`, by opening a user dialog such
-      as "Page xxx requests starting a new kernel on this server; do
-      you accept? yes/no/always for this site"
+  negotiate incoming connections that don't fall within the
+  `-NotebookApp.allow_origin pattern`, by opening a user dialog such
+  as "Page xxx requests starting a new kernel on this server; do
+  you accept? yes/no/always for this site".
 
-- [ ] Add support in Thebe for customizable (continuation) prompts, with striping and splitting as above, and automatic setting of the kernel. The customization option could look like:
+- [ ] Add support in Thebe for customizable (continuation) prompts, with striping
+  and splitting as above, and automatic setting of the kernel. The customization
+  option could look like:
 
-      ```
-         prompts = {
-            "sage: ": {continuations=["....:", "...  "], kernel="sagemath"},
-            ">>>> ": {continuation="...  ", kernel="python"}
-         }
-      ```
+  ```
+  prompts = {
+      "sage: ": {continuations=["....:", "...  "], kernel="sagemath"},
+      ">>>> ": {continuation="...  ", kernel="python"}
+  }
+  ```
 
-- [ ] Contribute support for Thebe upstream in Sphinx, with:
+- [ ] Contribute support for Thebe upstream in [Sphinx](http://www.sphinx-doc.org/en/stable/), with:
+  - [ ] Explicit markup to specify which code blocks are editable and with
+    which kernel, or setup kernel auto-detection from the prompt.
+  - [ ] The possibility for setting a default value for the above.
 
-      - [ ] Explicit markup to specify which code blocks are editable
-            and with which kernel, or setup kernel auto-detection from
-            the prompt.
-
-      - [ ] The possibility for setting a default value for the above
-
-- [ ] Refactor the Sage sphinx configuration to use the above
+- [ ] Refactor the Sage sphinx configuration to use the above.
 
 - [ ] Add support in Thebe for basic export to Jupyter notebooks. A
-      quality loss (in particular in terms of the hierarchical
-      structure) is acceptable.
+  quality loss (in particular in terms of the hierarchical
+  structure) is acceptable.
 
-- [ ] Have thebe.js provide good TAB completion.
+- [ ] Have Thebe provide good TAB completion.
 

[slel]

comment:3

Fix formatting in ticket description.

Regarding "There are two ongoing projects within Jupyter for having this working",
would someone add links?

[slel]

Description changed:

--- 
+++ 
@@ -1,4 +1,7 @@
-Since #20690 we have access to documentation (with live execution) within the Jupyter notebook. It was decided to keep the code simple. There are two ongoing projects within Jupyter for having this working. It might be wiser to wait for them...
+Since #20690 we have access to documentation (with live execution)
+within the Jupyter notebook. It was decided to keep the code simple.
+There are two ongoing projects within Jupyter for having this working.
+It might be wiser to wait for them...
 
 Some of the improvements mentioned in #20690 were:
 
@@ -12,10 +15,11 @@
 
 - [ ] Expand the activate button with a menu or other widgets for user customization
   of the Jupyter server. This typically would let the user choose between:
-    - tmpnb (will only be useful for Sage when tmpnb will include a Sage kernel),
-    - a local Jupyter server,
-    - whichever Jupyter server the browser is currently connected to?
-    - a user-specified server.
+  - tmpnb (will only be useful for Sage
+    [when tmpnb will include a Sage kernel](https://github.com/jupyter/tmpnb/issues/203)),
+  - a local Jupyter server,
+  - whichever Jupyter server the browser is currently connected to?
+  - a user-specified server.
 
 - [ ] Check whether Jupyter could be configured to dynamically
   negotiate incoming connections that don't fall within the
@@ -39,11 +43,11 @@
     which kernel, or setup kernel auto-detection from the prompt.
   - [ ] The possibility for setting a default value for the above.
 
-- [ ] Refactor the Sage sphinx configuration to use the above.
+- [ ] Refactor the Sage Sphinx configuration to use the above.
 
 - [ ] Add support in Thebe for basic export to Jupyter notebooks. A
   quality loss (in particular in terms of the hierarchical
   structure) is acceptable.
 
-- [ ] Have Thebe provide good TAB completion.
+- [ ] Have Thebe provide good TAB-completion.
 

[slel]

comment:4

Editing ticket description to add link to
tmpnb issue 203: "Octave kernel and SageMath kernel".

[sagetrac-tmonteil]

Description changed:

--- 
+++ 
@@ -51,3 +51,5 @@
 
 - [ ] Have Thebe provide good TAB-completion.
 
+- [ ] Let the cursor switch to the next cell after evaluation (SHIFT+ENTER).
+

[sagetrac-tmonteil]

Description changed:

--- 
+++ 
@@ -53,3 +53,5 @@
 
 - [ ] Let the cursor switch to the next cell after evaluation (SHIFT+ENTER).
 
+- [ ] Allow to save the current state of the webpage as a `.ipynb` worksheet, to be able to continue working on it later, see [[/ticket/20690#comment:53]].
+

[nthiery]

Description changed:

--- 
+++ 
@@ -38,20 +38,22 @@
   }
   ```
 
-- [ ] Contribute support for Thebe upstream in [Sphinx](http://www.sphinx-doc.org/en/stable/), with:
+- [ ] Have Thebe provide good TAB-completion.
+
+- [ ] Let the cursor switch to the next cell after evaluation (SHIFT+ENTER).
+
+- [ ] Extractx support for Thebe as a [Sphinx](http://www.sphinx-doc.org/en/stable/)  extension with (see the related [sage-cell](https://github.com/sagemath/sphinx-sagecell-ext) extension):
   - [ ] Explicit markup to specify which code blocks are editable and with
     which kernel, or setup kernel auto-detection from the prompt.
   - [ ] The possibility for setting a default value for the above.
 
 - [ ] Refactor the Sage Sphinx configuration to use the above.
 
+- [ ] Configure [sage-sample](https://github.com/sagemath/sage_sample) / [sage-package](https://github.com/sagemath/sage-package) to use it.
+
 - [ ] Add support in Thebe for basic export to Jupyter notebooks. A
   quality loss (in particular in terms of the hierarchical
   structure) is acceptable.
 
-- [ ] Have Thebe provide good TAB-completion.
-
-- [ ] Let the cursor switch to the next cell after evaluation (SHIFT+ENTER).
-
 - [ ] Allow to save the current state of the webpage as a `.ipynb` worksheet, to be able to continue working on it later, see [[/ticket/20690#comment:53]].
 

[nthiery]

Description changed:

--- 
+++ 
@@ -16,7 +16,7 @@
 - [ ] Expand the activate button with a menu or other widgets for user customization
   of the Jupyter server. This typically would let the user choose between:
   - tmpnb (will only be useful for Sage
-    [when tmpnb will include a Sage kernel](https://github.com/jupyter/tmpnb/issues/203)),
+    [when tmpnb will include a Sage kernel](https://github.com/jupyter/tmpnb/issues/203); there are ongoing discussions for providing an alternative sage-enabled tmpnb)
   - a local Jupyter server,
   - whichever Jupyter server the browser is currently connected to?
   - a user-specified server.
@@ -27,9 +27,7 @@
   as "Page xxx requests starting a new kernel on this server; do
   you accept? yes/no/always for this site".
 
-- [ ] Add support in Thebe for customizable (continuation) prompts, with striping
-  and splitting as above, and automatic setting of the kernel. The customization
-  option could look like:
+- [ ] Add support in Thebe for customizable (continuation) prompts, with striping and splitting as above, and automatic setting of the kernel. The customization option could look like:
 
   ```
   prompts = {
@@ -49,7 +47,7 @@
 
 - [ ] Refactor the Sage Sphinx configuration to use the above.
 
-- [ ] Configure [sage-sample](https://github.com/sagemath/sage_sample) / [sage-package](https://github.com/sagemath/sage-package) to use it.
+- [ ] Configure [sage-sample](https://github.com/sagemath/sage_sample) / [sage-package](https://github.com/sagemath/sage-package) to use it, to enable live documentation for Sage packages.
 
 - [ ] Add support in Thebe for basic export to Jupyter notebooks. A
   quality loss (in particular in terms of the hierarchical