Demonstration of Pypi package OCaml for polyglot notebook

TL;DR: this Pypi ocaml package allows to embed a real and complete OCaml runtime (with full standard library) in an IPython console or Jupyter notebook running IPython kernel (Python 3). OCaml functions and tiny programs can then be called directly from the IPython/Jupyter console, and results can be printed, or affected to Python variables! It even supports Currying multi-argument functions!

See also: note, if you really want to use OCaml in a Jupyter notebook, the best solution is OCaml-jupyter kernel! (i opened an issue there to present this to the developer, just for his curiosity)

Introduction

See this issue I opened and assigned to myself.

If you run this:

In [12]:
!pip3 install ocaml

Then you can use basic OCaml expressions and standard library... from IPython or Jupyter notebook with IPython kernel, without having to install OCaml yourself on your laptop! It's pre-compiled. I don't know what version thought...

In [176]:
import ocaml
In [177]:
answer_to_life = %ocaml 40 + 2
In [178]:
print(answer_to_life)
print(type(answer_to_life))  # a real integer!
42
<class 'int'>

It would be a great project to show to students studying Python and C and OCaml : this tiny IPython magic makes the link between Python and OCaml (one way) using OCaml compiled as a dynamic extension to CPython 😍 !

Reference and missing documentation of ocaml Pypi package

This Jupyter notebook uses the ocaml v0.0.11 package from Pypi.

There is no documentation, I asked the author, who works at JaneStreet and he redirected me to this blog post by JaneStreet.

Apparently, being professional developers doesn't mean they restrain themselves from shipping unfinished and undocumented packages to Pypi. Okay... Why? It seems highly unprofessional! I understand being in a hurry, but then just don't publish to Pypi, and let user install it using pip but from a Git repository:

$ pip install git+https://github.com/<Username>/<Projects>

Their blog post states

Note that this package is not currently very well polished but it should give some ideas of what can be done through this Python-OCaml integration.

But come on, writing a short README or description in the setup.py is the least that they should do...

I don't do that! See my Pypi packages on https://pypi.org/user/Naereen/:

  • lempel-ziv-complexity (Last released on Dec 19, 2019): Fast implementation of the Lempel-Ziv complexity function.
  • SMPyBandits (Last released on Oct 25, 2019): SMPyBandits: Open-Source Python package for Single- and Multi-Players multi-armed Bandits algorithms.
  • parcoursup (Last released on Nov 14, 2018): ParcourSup.py : un clone en Python 3 de ParcoursSup, écrit à but didactique.
  • generatewordcloud (Last released on Oct 16, 2016): A simple Python (2 or 3) script to generate a PNG word-cloud image from a bunch of text files. Based on word_cloud.
  • ansicolortags (Last released on Jul 2, 2016): ansicolortags brings a simple and powerful way to use colours in a terminal application with Python 2 and 3.

Let's show the current version of the package, just for reference:

In [1]:
%load_ext watermark
In [6]:
%watermark -v -p ocaml
CPython 3.6.9
IPython 7.16.1

ocaml 0.0.11

A first example

As for other languages binding, we can have either a full cell, starting with %%ocaml :

In [23]:
%%ocaml
print_endline "Hello world from OCaml running in Jupyter (from IPython)!";;
Hello world from OCaml running in Jupyter (from IPython)!

Natively, IPython/Jupyter supports lots of "magic commands", and especially %%bash, %%perl, %%javascript and %%ruby interface to famous scripting languages, and a generic %%script one.

In [179]:
%%bash
echo "Hello world from Bash running in Jupyter (from IPython)"
Hello world from Bash running in Jupyter (from IPython)
In [180]:
%%script bash
echo "Hello world from Bash running in Jupyter (from IPython)"
Hello world from Bash running in Jupyter (from IPython)

Not that it has been possible, since a long time, to call an OCaml toplevel like this!

In [35]:
%%script ocaml
print_endline "Hello world from OCaml running in Jupyter (from IPython)!";;
        OCaml version 4.05.0

Findlib has been successfully loaded. Additional directives:
  #require "package";;      to load a package
  #list;;                   to list the available packages
  #camlp4o;;                to load camlp4 (standard syntax)
  #camlp4r;;                to load camlp4 (revised syntax)
  #predicates "p,q,...";;   to set these predicates
  Topfind.reset();;         to force that packages will be reloaded
  #thread;;                 to enable threads

# Hello world from OCaml running in Jupyter (from IPython)!
- : unit = ()
# 

But it does nothing else than opening a sub-process, running ocaml command, feeding it the content of the cell, and then exiting.

  • Cons: It's not so pretty, as it prints all this startup messages. I thought we could silent them, like python -q (quiet) option... but we can't!
  • Pros: it shows the value of each line, so it can be used to quickly check something, if needed...

More full %%ocaml cells

In [53]:
%%ocaml
let sum : (int list -> int) = List.fold_left (+) 0 in
let a_list (n:int) : int list = Array.to_list (Array.init n (fun i -> i*i+30)) in
for n = 300000 to 300010 do
    Format.printf "\nList of size %2.i had sum = %4.i.%!" n (sum (a_list n));
done;;
List of size 300000 had sum = 8999955009050000.
List of size 300001 had sum = 9000045009050030.
List of size 300002 had sum = 9000135009650061.
List of size 300003 had sum = 9000225010850095.
List of size 300004 had sum = 9000315012650134.
List of size 300005 had sum = 9000405015050180.
List of size 300006 had sum = 9000495018050235.
List of size 300007 had sum = 9000585021650301.
List of size 300008 had sum = 9000675025850380.
List of size 300009 had sum = 9000765030650474.
List of size 300010 had sum = 9000855036050585.

As I was saying, using %%script ocaml allows to quickly check things, like for instance the interface of a module!

In [38]:
%%script ocaml
#show Array;;
        OCaml version 4.05.0

Findlib has been successfully loaded. Additional directives:
  #require "package";;      to load a package
  #list;;                   to list the available packages
  #camlp4o;;                to load camlp4 (standard syntax)
  #camlp4r;;                to load camlp4 (revised syntax)
  #predicates "p,q,...";;   to set these predicates
  Topfind.reset();;         to force that packages will be reloaded
  #thread;;                 to enable threads

# module Array :
  sig
    external length : 'a array -> int = "%array_length"
    external get : 'a array -> int -> 'a = "%array_safe_get"
    external set : 'a array -> int -> 'a -> unit = "%array_safe_set"
    external make : int -> 'a -> 'a array = "caml_make_vect"
    external create : int -> 'a -> 'a array = "caml_make_vect"
    external create_float : int -> float array = "caml_make_float_vect"
    val make_float : int -> float array
    val init : int -> (int -> 'a) -> 'a array
    val make_matrix : int -> int -> 'a -> 'a array array
    val create_matrix : int -> int -> 'a -> 'a array array
    val append : 'a array -> 'a array -> 'a array
    val concat : 'a array list -> 'a array
    val sub : 'a array -> int -> int -> 'a array
    val copy : 'a array -> 'a array
    val fill : 'a array -> int -> int -> 'a -> unit
    val blit : 'a array -> int -> 'a array -> int -> int -> unit
    val to_list : 'a array -> 'a list
    val of_list : 'a list -> 'a array
    val iter : ('a -> unit) -> 'a array -> unit
    val iteri : (int -> 'a -> unit) -> 'a array -> unit
    val map : ('a -> 'b) -> 'a array -> 'b array
    val mapi : (int -> 'a -> 'b) -> 'a array -> 'b array
    val fold_left : ('a -> 'b -> 'a) -> 'a -> 'b array -> 'a
    val fold_right : ('b -> 'a -> 'a) -> 'b array -> 'a -> 'a
    val iter2 : ('a -> 'b -> unit) -> 'a array -> 'b array -> unit
    val map2 : ('a -> 'b -> 'c) -> 'a array -> 'b array -> 'c array
    val for_all : ('a -> bool) -> 'a array -> bool
    val exists : ('a -> bool) -> 'a array -> bool
    val mem : 'a -> 'a array -> bool
    val memq : 'a -> 'a array -> bool
    val sort : ('a -> 'a -> int) -> 'a array -> unit
    val stable_sort : ('a -> 'a -> int) -> 'a array -> unit
    val fast_sort : ('a -> 'a -> int) -> 'a array -> unit
    external unsafe_get : 'a array -> int -> 'a = "%array_unsafe_get"
    external unsafe_set : 'a array -> int -> 'a -> unit = "%array_unsafe_set"
  end
# 

Using OCaml returned values from Python

This package allows to use dynamically defined OCaml functions from Python, the same way it can be done for other languages lke Julia or R (see this blog post if you never saw these possibilities, or this one).

For instance:

In [192]:
b = %ocaml true
print(type(b), b)

s = %ocaml "OK ?"
print(type(s), s)

i = %ocaml 2021
print(type(i), i)

f = %ocaml 2.99792458
print(type(f), f)
<class 'bool'> True
<class 'str'> OK ?
<class 'int'> 2021
<class 'float'> 2.99792458

So booleans, strings, integers and floats get perfectly mapped from OCaml values to Python native values.

In [182]:
l = %ocaml [1, 3, 5]
print(type(l), l)

a = %ocaml [|2; 4; 6|]
print(type(a), a)

t = %ocaml (23, 02, 2021)
print(type(t), t)
<class 'list'> [(1, 3, 5)]
<class 'list'> [2, 4, 6]
<class 'tuple'> (23, 2, 2021)

So 'a list, 'a array and 'a * 'b * .. heterogeneous tuples get perfectly mapped from OCaml values to Python native values!

But it's not perfect, as for instance OCaml has a char type (similar to the one in C) but Python only has strings, so this'll fail:

ValueError: ocaml error (Failure"unknown type char")

In [183]:
c = %ocaml 'C'
print(type(c), c)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-183-19c1c29bb9b1> in <module>
----> 1 c = get_ipython().run_line_magic('ocaml', "'C'")
      2 print(type(c), c)

/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2324                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2325             with self.builtin_trap:
-> 2326                 result = fn(*args, **kwargs)
   2327             return result
   2328 

/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py in ocaml(line)
     13     def ocaml(line):
     14       with sys_pipes():
---> 15         return toploop.get(line)
     16 
     17     del ocaml

ValueError: ocaml error (Failure"unknown type char")

And for functions:

In [185]:
sum_ocaml_1 = %ocaml let sum : (int list -> int) = List.fold_left (+) 0 in sum
In [188]:
print(sum_ocaml_1, type(sum_ocaml_1))
<built-in method anonymous_closure of PyCapsule object at 0x7ff259159120> <class 'builtin_function_or_method'>
In [189]:
sum_ocaml_1 ([1, 2, 3, 4, 5])  # 15
Out[189]:
15

Or simply

In [62]:
sum_ocaml_2 = %ocaml List.fold_left (+) 0
In [63]:
sum_ocaml_2 ([1, 2, 3, 4, 5])  # 15
Out[63]:
15

What about user defined types?

In [190]:
%%ocaml
type state = TODO | DONE | Unknown of string;;

let print_state (s:state) =
    match s with
    | TODO -> Format.printf "TODO%!"
    | DONE -> Format.printf "DONE%!"
    | Unknown status -> Format.printf "%s%!" status
;;

print_state TODO;;
TODO

It fails:

SyntaxError: ocaml evaluation error on lines 1:11 to 1:15
Error: 
>  1: let out = (type TODO | DONE);;
Syntax error: operator expected.
In [191]:
t = %ocaml type TODO | DONE
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-191-92828381558a>", line 1, in <module>
    t = get_ipython().run_line_magic('ocaml', 'type TODO\xa0| DONE')

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2326, in run_line_magic
    result = fn(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py", line 15, in ocaml
    return toploop.get(line)

  File "<string>", line unknown
SyntaxError: ocaml evaluation error on lines 1:11 to 1:15
Error: 
>  1: let out = (type TODO | DONE);;
Syntax error: operator expected.

Indeed the %ocam magic only works for expression, with no ;;.

More datatypes conversions to Python?

We can still explore:

  • references, let x = ref 1
  • polymorphic functions like let smaller (x: 'a) (y: 'a) : true = x < y
  • 'a option type
  • functions with labels
  • records
  • polymorphic variants ?

Fore reference, see https://github.com/janestreet/ppx_python#conversions

References - fail

In [198]:
xref = %ocaml ref 1
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-198-8acf58a91aa8> in <module>
----> 1 xref = get_ipython().run_line_magic('ocaml', 'ref 1')

/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2324                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2325             with self.builtin_trap:
-> 2326                 result = fn(*args, **kwargs)
   2327             return result
   2328 

/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py in ocaml(line)
     13     def ocaml(line):
     14       with sys_pipes():
---> 15         return toploop.get(line)
     16 
     17     del ocaml

ValueError: ocaml error (Failure"unknown type ref")

Polymorphic function - works!

In [222]:
cons = %ocaml fun hd tl -> hd :: tl
print(cons, type(cons))
<built-in method anonymous_closure of PyCapsule object at 0x7ff259179f30> <class 'builtin_function_or_method'>
In [217]:
cons(10)([20, 30])
Out[217]:
[10, 20, 30]
In [221]:
cons(1.0)([2.0, 30])
Out[221]:
[1.0, 2.0, 30]

Woooo, somehow OCaml accepted a polymorphic list at some point?

In [208]:
head, tail = %ocaml List.hd, List.tl
In [211]:
a_list = [1, 2, 3]
a_list.append(a_list)
head(a_list), tail(a_list)
Out[211]:
(1, [2, 3, [1, 2, 3, [...]]])

Another example:

In [201]:
smaller = %ocaml fun (x: int) (y: int) -> x < y
print(smaller)
help(smaller)
<built-in method anonymous_closure of PyCapsule object at 0x7ff25cf80a20>
Help on built-in function anonymous_closure:

anonymous_closure(...) method of builtins.PyCapsule instance
    int -> int -> bool

In [202]:
smaller_poly = %ocaml fun (x: 'a) (y: 'a) -> x < y
print(smaller_poly)
help(smaller_poly)
<built-in method anonymous_closure of PyCapsule object at 0x7ff259179ea0>
Help on built-in function anonymous_closure:

anonymous_closure(...) method of builtins.PyCapsule instance
    pyobject -> pyobject -> bool

Option type

In [204]:
none = %ocaml None
print(none, type(none))
None <class 'NoneType'>
In [205]:
some_int = %ocaml Some 42
print(some_int, type(some_int))
42 <class 'int'>
In [206]:
# instinguishable from None, so that's weird!
some_None = %ocaml Some None
print(some_None, type(some_None))
None <class 'NoneType'>

Note that this limitation was explained:

Note that this makes the two OCaml values [Some None] and [None] indistinguishable on the Python side as both are represented using None.

Functions with labels - labels get erased

In [227]:
# val fold_left : f:('a -> 'b -> 'a) -> init:'a -> 'b list -> 'a
fold_left = %ocaml ListLabels.fold_left

print(fold_left, type(fold_left))
help(fold_left:)
<built-in method anonymous_closure of PyCapsule object at 0x7ff259179f00> <class 'builtin_function_or_method'>
Help on built-in function anonymous_closure:

anonymous_closure(...) method of builtins.PyCapsule instance
    pyobject -> pyobject -> pyobject -> pyobject -> (pyobject) list -> pyobject

In [230]:
fold_left(lambda x: lambda y: x + y)(0)([1, 2, 3, 4, 5])
Out[230]:
15

Record - fail

In [194]:
%%ocaml
type ratio = {num: int; denom: int};;
let add_ratio r1 r2 =
   {num = r1.num * r2.denom + r2.num * r1.denom;
    denom = r1.denom * r2.denom};;
add_ratio {num=1; denom=3} {num=2; denom=5};;
In [197]:
%ocaml {num=1; denom=3}
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-197-077c7ab12521> in <module>
----> 1 get_ipython().run_line_magic('ocaml', '{num=1; denom=3}')

/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2324                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2325             with self.builtin_trap:
-> 2326                 result = fn(*args, **kwargs)
   2327             return result
   2328 

/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py in ocaml(line)
     13     def ocaml(line):
     14       with sys_pipes():
---> 15         return toploop.get(line)
     16 
     17     del ocaml

ValueError: ocaml error (Failure"unknown type ratio")

Of course it fails!

But it could be translated to Python dictionaries.

Exceptions - fail

In [207]:
exc = %ocaml exception Empty_list
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-207-709df63f312a>", line 1, in <module>
    exc = get_ipython().run_line_magic('ocaml', 'exception Empty_list')

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2326, in run_line_magic
    result = fn(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py", line 15, in ocaml
    return toploop.get(line)

  File "<string>", line unknown
SyntaxError: ocaml evaluation error on lines 1:11 to 1:20
Error: 
>  1: let out = (exception Empty_list);;
Syntax error: operator expected.

Polymorphic variants - fail

See documentation

In [232]:
%%ocaml
Format.printf "%i%!" (let value `float = 0 in value `float);;
0
In [236]:
zero = %ocaml let value `float = 0 in value `float

print(zero, type(zero))
0 <class 'int'>
In [237]:
variant = %ocaml `float

print(variant, type(variant))
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-237-35ef86792a45>", line 1, in <module>
    variant = get_ipython().run_line_magic('ocaml', '`float')

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2326, in run_line_magic
    result = fn(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py", line 15, in ocaml
    return toploop.get(line)

  File "<string>", line unknown
SyntaxError: unsupported type Tvariant

Optional arguments - fail

It would be difficult to implement them correctly along side the (awesome) partial application closure feature...

In [240]:
%%ocaml
let bump ?(step = 1) x = x + step;;
Format.printf "\n%i%!" (bump 41);;
Format.printf "\n%i%!" (bump ~step:12 30);;
42
42
In [241]:
bump = %ocaml let bump ?(step = 1) x = x + step in bump
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-241-9255dc0fa695>", line 1, in <module>
    bump = get_ipython().run_line_magic('ocaml', 'let bump ?(step = 1) x = x + step in bump')

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2326, in run_line_magic
    result = fn(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py", line 15, in ocaml
    return toploop.get(line)

  File "<string>", line unknown
SyntaxError: optional arguments are not supported

Recursive list?

In [7]:
%%ocaml
let rec list1 = 0 :: list2
and list2 = 1 :: list1
in
Format.printf "%i -> %i -> %i -> %i ...%!" (List.hd list1) (List.hd list2) (List.hd (List.tl list2)) (List.hd (List.tl list1));;
0 -> 1 -> 0 -> 1 ...
In [6]:
# don't run
if False:
    list1, list2 = %ocaml let rec list1 = 0 :: list2 and list2 = 1 :: list1 in (list1, list2)

It fails, but takes 100% CPU and freezes. But in Python we can do it:

In [5]:
list1 = [0]
list2 = [1]
list1.append(list2)
list2.append(list1)
print(list1)
print(list2)
[0, [1, [...]]]
[1, [0, [...]]]

From standard library

What about Sets, mapped to set? What about HashTbl, mapped to dict? And Stack, Queue, etc?

TODO: left as an exercise for the reader.

Map - fail

In [9]:
%%ocaml

module IntPairs = struct
    type t = int * int
    let compare (x0,y0) (x1,y1) =
        match Stdlib.compare x0 x1 with
        0 -> Stdlib.compare y0 y1
        | c -> c
end

module PairsMap = Map.Make(IntPairs)

let m = PairsMap.(empty |> add (0,1) "hello" |> add (1,0) "world")

(* not an expression, not usable in %ocaml magic *)

Stack (or any module using a custom type) - fail

In [10]:
stack = %ocaml Stack.create()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-bbd04f52cfc6> in <module>
----> 1 stack = get_ipython().run_line_magic('ocaml', 'Stack.create()')

/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2324                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2325             with self.builtin_trap:
-> 2326                 result = fn(*args, **kwargs)
   2327             return result
   2328 

/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py in ocaml(line)
     13     def ocaml(line):
     14       with sys_pipes():
---> 15         return toploop.get(line)
     16 
     17     del ocaml

ValueError: ocaml error (Failure"unknown type t")

😮 Curryed functions!

Imagine you define this function in math: $p : (x,y,z) \mapsto x * y * z$, on $\mathbb{N}^3 \to \mathbb{N}$. Then the Curryed form states that it is equivalent to $p' : x \mapsto y \mapsto z \mapsto x * y * z$, informally defined on $\mathbb{N} \to \mathbb{N} \to \mathbb{N} \to \mathbb{N}$.

So for instance if $x=1$ and $y=2$, $p'(x)(y)$ is $z \mapsto 1 * 2 * z$, which is also $z \mapsto p(1, 2, z)$.

In Python function, that would be this function:

In [75]:
def product3values(x, y, z):
    return x * y * z

But you can't directly use it for partial application:

In [80]:
x = 1
y = 2
partial_product = product3values(x, y)
z = 10
print(f"With x = {x}, y = {y}, and {partial_product} applied to z = {z}, we got {partial_product(z)}")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-80-3ba473df199e> in <module>
      1 x = 1
      2 y = 2
----> 3 partial_product = product3values(x, y)
      4 z = 10
      5 print(f"With x = {x}, y = {y}, and {partial_product} applied to z = {z}, we got {partial_product(z)}")

TypeError: product3values() missing 1 required positional argument: 'z'

With the Python standard library, it's possible to use functools.partial to obtain partially evaluated functions, which can be viewed as a limited support of Curryed function.

In [82]:
import functools

partial_product = functools.partial(product3values, 1, 2)

z = 10
print(f"With x = {x}, y = {y}, and {partial_product} applied to z = {z}, we got {partial_product(z)}")
With x = 1, y = 2, and functools.partial(<function product3values at 0x7ff25cda0c80>, 1, 2) applied to z = 10, we got 20

But in OCaml, the conventions is to directly write functions in Curry form, rather than tuple form:

In [86]:
%%ocaml
(* this is advised *)
let product_curry (x:int) (y:int) (z:int) : int = x * y * z in
let x = 1 and y = 2 in
let partial_product = product_curry x y in
let z = 10 in
Format.printf "With x = %i, y = %i, and partial_product applied to z = %i, we got %i." x y z (partial_product z);;
With x = 1, y = 2, and partial_product applied to z = 10, we got 20.

Indeed, in most situations, the tuple form is just not "OCaml"esque, and tedious to use, and does not allow partial application!

In [89]:
%%ocaml
(* this is NOT advised *)
let product_curry (xyz : (int * int * int)) : int =
    let x, y, z = xyz in
    x * y * z
in
let x = 1 and y = 2 in
let partial_product = product_curry x y in
let z = 10 in
Format.printf "With x = %i, y = %i, and partial_product applied to z = %i, we got %i." x y z (partial_product z);;
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-89-a59ab8443eec>", line 1, in <module>
    get_ipython().run_cell_magic('ocaml', '', '(* this is NOT advised *)\nlet product_curry (xyz : (int * int * int)) : int =\n    let x, y, z = xyz in\n    x * y * z\nin\nlet x = 1 and y = 2 in\nlet partial_product = product_curry x y in\nlet z = 10 in\nFormat.printf "With x = %i, y = %i, and partial_product applied to z = %i, we got %i." x y z (partial_product z);;\n')

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2371, in run_cell_magic
    result = fn(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py", line 30, in ocaml
    return toploop.eval(cell)

  File "<string>", line unknown
SyntaxError: ocaml evaluation error on lines 7:165 to 7:178
Error: 
   2: let product_curry (xyz : (int * int * int)) : int =
   3:     let x, y, z = xyz in
   4:     x * y * z
   5: in
   6: let x = 1 and y = 2 in
>  7: let partial_product = product_curry x y in
   8: let z = 10 in
   9: Format.printf "With x = %i, y = %i, and partial_product applied to z = %i, we got %i." x y z (partial_product z);;
This function has type int * int * int -> int
It is applied to too many arguments; maybe you forgot a `;'.

Well that was some long explanation, but now comes the magic! If you use %ocaml to get in Python the values returned from OCaml, then functions are Curryed function!

In [90]:
product_curry = %ocaml let product_curry (x:int) (y:int) (z:int) : int = x * y * z in product_curry

The only information we have on this function is the OCaml signature, in its docstring:

In [92]:
help(product_curry)
Help on built-in function anonymous_closure:

anonymous_closure(...) method of builtins.PyCapsule instance
    int -> int -> int -> int

So we can't use it as a classical 3-arguments Python function:

In [93]:
product_curry(1, 2, 10)
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-93-30a4c1f01ec3>", line 1, in <module>
    product_curry(1, 2, 10)

  File "<string>", line unknown
SyntaxError: (Failure "expected int, got Tuple")

But we CAN use it as a Curryed function!

In [94]:
product_curry(1)(2)(10)
Out[94]:
20

Which is awesome because now we can do partial evaluation as in OCaml!

In [95]:
partial_product_1 = product_curry(1)
partial_product_1(2)(10)
Out[95]:
20
In [99]:
partial_product_2 = product_curry(1)(2)
partial_product_2_too = partial_product_1(2)
partial_product_2(10), partial_product_2_too(10)
Out[99]:
(20, 20)

What's very cool is that these functions docstrings keep showing the signature of the underlying OCaml function, even if they were obtained from pure Python cells!

In [101]:
help(partial_product_1)
help(partial_product_2)
help(partial_product_2_too)
Help on built-in function anonymous_closure:

anonymous_closure(...) method of builtins.PyCapsule instance
    int -> int -> int

Help on built-in function anonymous_closure:

anonymous_closure(...) method of builtins.PyCapsule instance
    int -> int

Help on built-in function anonymous_closure:

anonymous_closure(...) method of builtins.PyCapsule instance
    int -> int

That's it for this feature, it's cool and interesting.

Some questions

Can we share variables between two consecutive full cells?

In the %ocaml mode, nothing can be shared from the OCaml side, as it's just an expression, let's check:

In [65]:
%ocaml let x = 1 in x
Out[65]:
1
In [67]:
%ocaml x
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-67-269a4d148dbe>", line 1, in <module>
    get_ipython().run_line_magic('ocaml', 'x')

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2326, in run_line_magic
    result = fn(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py", line 15, in ocaml
    return toploop.get(line)

  File "<string>", line unknown
SyntaxError: ocaml evaluation error on lines 1:11 to 1:12
Error: 
>  1: let out = (x);;
Unbound value x

But what about full cell mode, %%ocaml?

In [70]:
%%ocaml
(* See https://en.wikipedia.org/wiki/42_(number) *)
let answer_to_life = 42 in
Format.printf "\n... « The answer to life, the universe, and everything is %i »%!" answer_to_life;;
... « The answer to life, the universe, and everything is 42 »
In [71]:
%%ocaml
Format.printf "\n... « The answer to life, the universe, and everything is %i »%!" answer_to_life;;
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-71-0d2d3c7f5c67>", line 1, in <module>
    get_ipython().run_cell_magic('ocaml', '', 'Format.printf "\\n... « The answer to life, the universe, and everything is %i »%!" answer_to_life;;\n')

  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2371, in run_cell_magic
    result = fn(*args, **kwargs)

  File "/usr/local/lib/python3.6/dist-packages/ocaml/__init__.py", line 30, in ocaml
    return toploop.eval(cell)

  File "<string>", line unknown
SyntaxError: ocaml evaluation error on lines 1:85 to 1:99
Error: 
>  1: Format.printf "\n... « The answer to life, the universe, and everything is %i »%!" answer_to_life;;
Unbound value answer_to_life

==> Answer: no, we cannot share any memory between two consecutive cells. Well, too bad, but it's not so important.

Is there any documentation of the magic or commands?

No.

In [74]:
?%ocaml
Docstring: <no docstring>
File:      /usr/local/lib/python3.6/dist-packages/ocaml/__init__.py

More remarks

Note that there blog post says that using the opttoploop could be used to compile the OCaml to a faster version (native code), but do not document about this.

Note that with the toploop module, the OCaml code is evaluated by compiling to bytecode which is not optimal, switching to the opttoploop module that generates native code should make it even faster.

In [164]:
"opttoploop" in dir(ocaml)
Out[164]:
False

Also note that the ocaml module is shipped with an example of a tiny module which was written in OCaml and compiled, being made available to Python directly:

In [167]:
# it doesn't have a docstring, don't try help(<...>) or <...>?
ocaml.ocaml.example_module.approx_pi
Out[167]:
<function PyCapsule.anonymous_closure>
In [168]:
ocaml.ocaml.example_module.approx_pi(1000000)
Out[168]:
3.1415916986605086

Final note: a tiny benchmark

Let's compare the speed of naive Python and naive OCaml sum of a list/array of floats, for various input size.

In [169]:
import ocaml
import numpy as np
In [170]:
python_sum = sum
ocaml_sum = %ocaml List.fold_left (+.) 0.
numpy_sum = np.sum
In [171]:
print(python_sum( [1.0, 2.0, 3.0, 4.0, 5.0] ))
print(ocaml_sum(  [1.0, 2.0, 3.0, 4.0, 5.0] ))
print(numpy_sum(  [1.0, 2.0, 3.0, 4.0, 5.0] ))
15.0
15.0
15.0

Now for a "large" array, let's use IPython %timeit magic for very quick benchmarking.

Science is about making hypotheses, designing experiments to check them, and conclude. My hypothesis here is that the OCaml version will be between 10 to 50 slower than the Python one (and Numpy version is 50-200 faster than Python).

In [120]:
sizes = [100, 1000, 10000, 100000, 1000000, 10000000]
print(f"Comparing time of python_sum and ocaml_sun :")
for size in sizes:
    print(f"\n- For size = {size}:")
    X = list(np.random.randn(size))
    
    print("\tFor python sum: ", end='')
    %timeit python_sum(X)
    
    assert np.isclose(python_sum(X), ocaml_sum(X))
    print("\tFor OCaml sum: ", end='')
    %timeit ocaml_sum(X)

    assert np.isclose(python_sum(X), numpy_sum(X))
    print("\tFor numpy.sum: ", end='')
    %timeit numpy_sum(X)
Comparing time of python_sum and ocaml_sun :

- For size = 10000000:
	For python sum: 677 ms ± 12.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
	For OCaml sum: 2.92 s ± 441 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
	For numpy.sum: 632 ms ± 4.41 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Well that's better than what I expected! It seems that the overhead is constant and not increasing when the size of the input is increasing!

It means that if the Python code runs in time $T_1(n)$ for inputs of size $n$, then then OCaml binding code runs in less than $T_2(n) \leq \alpha T_1(n) + \beta$, with two constants $\alpha, \beta$.

In [175]:
import matplotlib.pyplot as plt

µs = 1
ms = 1000*µs
s  = 1000*ms

X = sizes
# TODO: get this automatically?
    
Y_python = [ 7.27*µs, 72.1*µs, 786*µs, 7.55*ms, 68.2*ms, 677*ms ]
Y_ocaml  = [ 16*µs, 157*µs, 1.8*ms, 24.2*ms, 286*ms, 2.92*s ]
Y_numpy  = [ 12*µs, 67.7*µs, 615*µs, 6.25*ms, 62.6*ms, 632*ms ]

fig = plt.figure(figsize=(14, 10), dpi=300)

plt.loglog(X, Y_python, color="blue", marker="o", label="naive Python", lw=4, ms=15)
plt.loglog(X, Y_ocaml,  color="green", marker="d", label="using OCaml", lw=4, ms=15)
plt.loglog(X, Y_numpy,  color="orange", marker="s", label="using Numpy", lw=4, ms=15)

plt.ylabel("Time in micro-seconds")
plt.xlabel("Size of input list")
plt.legend()
plt.title("Tiny benchmark comparing OCaml binding to Python")
plt.show()

Just to check the experimental values of $\alpha$ and $\beta$ in my claim above, let's use numpy.polyfit function:

In [133]:
np.polyfit(Y_python, Y_ocaml, deg=1)
Out[133]:
array([    4.31769413, -3617.94315775])

So the OCaml bidding code runs about 4.5 times slower than the Python one!

It means that if you're doing a data analysis or some things in Python, and suddenly you think of an easy way to write an elegant and fast OCaml version, it's definitely viable to write it in OCaml, use ocaml_function = %ocaml ... and then use to solve your task! The overhead for using OCaml interpreted functions should not be too large.

Conclusion: demonstration of Pypi package OCaml for polyglot notebook

This (long) notebook dived in the details of this Pypi ocaml package allows to embed a real and complete OCaml runtime (with full standard library) in an IPython console or Jupyter notebook running IPython kernel (Python 3). OCaml functions and tiny programs can then be called directly from the IPython/Jupyter console, and results can be printed, or affected to Python variables! It even supports Currying multi-argument functions!

It's too bad that the package lack documentation, and is not open source, I would have liked to improve it a little bit!

See also: note, if you really want to use OCaml in a Jupyter notebook, the best solution is OCaml-jupyter kernel! (i opened an issue there to present this to the developer, just for his curiosity)

That's it for today! See other notebooks