Extending lolviz with a unified display function that chooses the best one based on the datatype

See this issue: « Could we add a "super" display function that chooses the best one based on the datatype? »

In [1]:
%load_ext watermark
%watermark -v -m -p lolviz
CPython 3.6.9
IPython 7.16.1

lolviz 1.4.4

compiler   : GCC 8.4.0
system     : Linux
release    : 5.4.0-65-generic
machine    : x86_64
processor  : x86_64
CPU cores  : 8
interpreter: 64bit
In [164]:
import lolviz

Adding this unified display function

In [316]:
modes = [ "str", "matrix", "call", "calls", "obj", "tree", "lol", "list", "tree" ]

def unified_lolviz(obj=None, mode:modes=None, **kwargs):
    """ Unified function to display `obj` with lolviz, in Jupyter notebook only."""
    try:
        if mode == "str" or isinstance(obj, str):
            return lolviz.strviz(obj, **kwargs)
        elif "<class 'numpy.ndarray'>" == str(type(obj)):
            # can't use isinstance(obj, np.ndarray) without import numpy!
            return lolviz.matrixviz(obj, **kwargs)
        elif mode == "matrix":
            # Experimental transparent use of np.array()
            try:
                from numpy import array as _
                return lolviz.matrixviz(_(obj), **kwargs)
                del _
            except:
                return lolviz.matrixviz(obj, **kwargs)
        elif mode == "call":
            from sys import _getframe
            return lolviz.callviz(frame=_getframe(1), **kwargs)
            del _getframe
        elif mode == "calls":
            return lolviz.callsviz( **kwargs)
        elif mode == "lol" or isinstance(obj, (tuple, list)) and obj and isinstance(obj[0], (tuple, list)):
            # obj is a list, is non empty, and obj[0] is a list!
            return lolviz.lolviz(obj, **kwargs)
        elif mode == "list" or isinstance(obj, (tuple, list)):
            return lolviz.listviz(obj, **kwargs)
        elif mode == "tree" or isinstance(obj, dict):
            return lolviz.treeviz(obj, **kwargs)  # default
        else:
            return lolviz.objviz(obj, **kwargs)  # default
    except TypeError:
        # unable to use lolviz, let's just return the object,
        # it will be nicely displayed by IPython
        return obj

viz = unified_lolviz

Testing naively

In [95]:
data = ['hi', 'mom', {3, 4}, {"parrt": "user"}]
viz(data)
Out[95]:
G node139754583851912 0 1 2 3 'hi' 'mom' {3, 4} {'parrt': 'user'}

Testing from within a Jupyter notebook

I test here all the features of lolviz :

List

In [96]:
squares = [ i**2 for i in range(10) ]
In [97]:
squares
Out[97]:
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
In [98]:
viz(squares)
Out[98]:
G node139754583807368 0 1 2 3 4 5 6 7 8 9 0 1 4 9 16 25 36 49 64 81

Under the hood of this unified function viz, it was calling lolvzi.listviz:

In [99]:
lolviz.listviz(squares)
Out[99]:
G node139754583807368 0 1 2 3 4 5 6 7 8 9 0 1 4 9 16 25 36 49 64 81

List of lists

In [100]:
n, m = 3, 4
example_matrix = [[0 if i != j else 1 for i in range(n)] for j in range(m)]
In [101]:
example_matrix
Out[101]:
[[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 0, 0]]
In [102]:
viz(example_matrix)
Out[102]:
G node139754583524616 0 1 2 3 node139754583553160 0 1 2 1 0 0 node139754583524616:0->node139754583553160:w node139754580861448 0 1 2 0 1 0 node139754583524616:1->node139754580861448:w node139754584005064 0 1 2 0 0 1 node139754583524616:2->node139754584005064:w node139754584017544 0 1 2 0 0 0 node139754583524616:3->node139754584017544:w

If we want, we can force the lolviz.matrixviz mode:

In [103]:
viz(example_matrix, mode='matrix')
Out[103]:
G node139754583226288 1 0 0 0 1 0 0 0 1 0 0 0

And this works without having to manually import numpy or convert the list-of-list to a numpy array!

In [104]:
np?
Object `np` not found.

List of lists of lists???

In [105]:
n, m, o = 2, 3, 4
example_3D_matrix = [[[
    1 if i < j < k else 0
    for i in range(n)]
    for j in range(m)]
    for k in range(o)]
In [106]:
example_3D_matrix
Out[106]:
[[[0, 0], [0, 0], [0, 0]],
 [[0, 0], [0, 0], [0, 0]],
 [[0, 0], [1, 0], [0, 0]],
 [[0, 0], [1, 0], [1, 1]]]
In [107]:
viz(example_3D_matrix)
Out[107]:
G node139754583973640 0 1 2 3 node139754584018632 0 1 2 [0, 0] [0, 0] [0, 0] node139754583973640:0->node139754584018632:w node139754583903240 0 1 2 [0, 0] [0, 0] [0, 0] node139754583973640:1->node139754583903240:w node139754583995272 0 1 2 [0, 0] [1, 0] [0, 0] node139754583973640:2->node139754583995272:w node139754583538440 0 1 2 [0, 0] [1, 0] [1, 1] node139754583973640:3->node139754583538440:w

It works, even if it is not as pretty.

Tree

Only for binary trees, apparently. Let's try with a dictionary that looks like a binary tree:

In [108]:
anakin = {
    "name": "Anakin Skywalker",
    "son": {
        "name": "Luke Skywalker",
    },
    "daughter": {
        "name": "Leia Skywalker",
    },
}
In [109]:
from pprint import pprint
pprint(anakin)
{'daughter': {'name': 'Leia Skywalker'},
 'name': 'Anakin Skywalker',
 'son': {'name': 'Luke Skywalker'}}

Then by remembering the correction function, we can do:

In [110]:
lolviz.treeviz(anakin)
Out[110]:
G node139754865714016 'name' 'Anakin Skywalker' 'son'     'daughter'     node139754864857448 'name' 'Luke Skywalker' node139754865714016:c->node139754864857448 node139754863124272 'name' 'Leia Skywalker' node139754865714016:c->node139754863124272

But it's simpler to use the automatic function!

In [111]:
viz(anakin)
Out[111]:
G node139754865714016 'name' 'Anakin Skywalker' 'son'     'daughter'     node139754864857448 'name' 'Luke Skywalker' node139754865714016:c->node139754864857448 node139754863124272 'name' 'Leia Skywalker' node139754865714016:c->node139754863124272

It works out of the box for dictionaries!

Let's check another example:

In [116]:
anakin_rec = {
    "name": "Anakin Skywalker",
}
In [117]:
luke_rec = {
    "name": "Luke Skywalker",
    "father": anakin_rec,
}
leia_rec = {
    "name": "Leia Skywalker",
    "father": anakin_rec,
},
In [118]:
anakin_rec.update({
    "son": luke_rec,
    "daughter": leia_rec,
})
In [121]:
from pprint import pprint
pprint(anakin_rec)
{'daughter': ({'father': <Recursion on dict with id=139754864347464>,
               'name': 'Leia Skywalker'},),
 'name': 'Anakin Skywalker',
 'son': {'father': <Recursion on dict with id=139754864347464>,
         'name': 'Luke Skywalker'}}

But it's simpler to use the automatic function!

In [124]:
viz(anakin_rec)
Out[124]:
G node139754864347464 'name' 'Anakin Skywalker' 'son'     'daughter'     node139754864052696 'name' 'Luke Skywalker' 'father'     node139754864347464:c->node139754864052696 node139754859924336 0   node139754864347464:c->node139754859924336 node139754864052696:c->node139754864347464 node139754862935784 'name' 'Leia Skywalker' 'father'     node139754859924336:c->node139754862935784 node139754862935784:c->node139754864347464

It works too for recursive dictionaries!

Let's check another example:

In [125]:
class Tree:
    def __init__(self, value, left=None, right=None):
        self.value = value
        self.left = left
        self.right = right
        
root = Tree('parrt',
            Tree('mary',
                 Tree('jim',
                      Tree('srinivasan'),
                      Tree('april'))),
            Tree('xue',None,Tree('mike')))
In [126]:
lolviz.treeviz(root)
Out[126]:
G node139754584139312 Tree value 'parrt' left right node139754584139200 Tree value 'mary' left right node139754584139312:c->node139754584139200 node139754584139256 Tree value 'xue' left right node139754584139312:c->node139754584139256 node139754584138864 Tree value 'jim' left right node139754584139200:c->node139754584138864 node139754584140040 Tree value 'srinivasan' left right node139754584138864:c->node139754584140040 node139754584139928 Tree value 'april' left right node139754584138864:c->node139754584139928 node139754584139088 Tree value 'mike' left right node139754584139256:c->node139754584139088
In [127]:
viz(root, mode="tree")
Out[127]:
G node139754584139312 Tree value 'parrt' left right node139754584139200 Tree value 'mary' left right node139754584139312:c->node139754584139200 node139754584139256 Tree value 'xue' left right node139754584139312:c->node139754584139256 node139754584138864 Tree value 'jim' left right node139754584139200:c->node139754584138864 node139754584140040 Tree value 'srinivasan' left right node139754584138864:c->node139754584140040 node139754584139928 Tree value 'april' left right node139754584138864:c->node139754584139928 node139754584139088 Tree value 'mike' left right node139754584139256:c->node139754584139088

Objects

In [128]:
viz(anakin)
Out[128]:
G node139754865714016 'name' 'Anakin Skywalker' 'son'     'daughter'     node139754864857448 'name' 'Luke Skywalker' node139754865714016:c->node139754864857448 node139754863124272 'name' 'Leia Skywalker' node139754865714016:c->node139754863124272
In [129]:
viz(anakin.values())
Out[129]:
G node139754578804232 0 1 2 node139754864857448 'name' 'Luke Skywalker' node139754578804232:1->node139754864857448:w node139754863124272 'name' 'Leia Skywalker' node139754578804232:2->node139754863124272:w
In [130]:
viz(anakin.items())
Out[130]:
G node139754578801480 0 1 2 node139754578215752 0 1 'name' 'Anakin Skywalker' node139754578801480:0->node139754578215752:w node139754579571144 0 1 'son'   node139754578801480:1->node139754579571144:w node139754578173512 0 1 'daughter'   node139754578801480:2->node139754578173512:w node139754864857448 'name' 'Luke Skywalker' node139754579571144:c->node139754864857448 node139754863124272 'name' 'Leia Skywalker' node139754578173512:c->node139754863124272

For complex numbers for instance?

In [131]:
z = 1+4j
In [132]:
print(z)
(1+4j)
In [133]:
viz(z)
Out[133]:
G node139754581383056 <complex:(1+4j)>

OK, this fails. We could improve viz for complex, but it's beyond the scope of this small experiment.

Calls

Let's explore a recursive function:

In [176]:
def factorial(n):
    if n < 0: return 0
    elif n == 0: return 1
    else: return n * factorial(n - 1)
In [177]:
for n in range(12):
    print(f"{n:>2}! = {factorial(n):>10}")
 0! =          1
 1! =          1
 2! =          2
 3! =          6
 4! =         24
 5! =        120
 6! =        720
 7! =       5040
 8! =      40320
 9! =     362880
10! =    3628800
11! =   39916800

And now with some visualization:

In [178]:
from IPython.display import display

This is what lolviz.callviz shows:

In [180]:
def factorial2(n):
    display(lolviz.callviz(varnames=["n"]))
    if n < 0: return 0
    elif n == 0: return 1
    else: return n * factorial2(n - 1)
    
n = 4
print(f"{n:>2}! = {factorial2(n):>10}")
G node139754583156104 factorial2 n 4
G node139755216231048 factorial2 n 3
G node50772520 factorial2 n 2
G node51975112 factorial2 n 1
G node52305768 factorial2 n 0
 4! =         24

We really see the "call stack" as the system keeps track of the nested calls. I like that! 👌

But again, I'm lazy, I want to just use the viz function:

In [181]:
def factorial3(n):
    display(viz(varnames=["n"], mode="call"))
    if n < 0: return 0
    elif n == 0: return 1
    else: return n * factorial3(n - 1)
In [183]:
n = 4
print(f"{n}! = {factorial3(n)}")
G node139754858422344 factorial3 n 4
G node52010408 factorial3 n 3
G node139755216156424 factorial3 n 2
G node51838072 factorial3 n 1
G node51345896 factorial3 n 0
4! = 24

Okay, by using the optional frame= argument of lolviz.callviz, and some sys._getframe black magic, we can do this!

See https://code.activestate.com/recipes/579105-how-a-python-function-can-find-the-name-of-its-cal/

Can we write a @show_recursion function decorator, to automatically do this visualization of recursive calls for us?

In [274]:
def show_recursion(varnames=None):
    def showing_recursion_with_varnames(function):
        nonlocal varnames
        # https://stackoverflow.com/questions/582056/getting-list-of-parameter-names-inside-python-function#4051447
        if varnames is None:
            varnames = function.__code__.co_varnames
        import lolviz
        from IPython.display import display
        from functools import wraps
        @wraps(function)
        def wrapped_function(*args, **kwargs):
            nonlocal display, wraps, varnames, lolviz
            ##print(f"args = {args}, of type = {type(args)}")
            ##print(f"kwargs = {kwargs}, of type = {type(kwargs)}")
            #print(f"varnames = {varnames}")
            display(lolviz.callsviz(varnames=varnames))
            return function(*args, **kwargs)
        return wrapped_function
    return showing_recursion_with_varnames
In [275]:
@show_recursion()
def factorial4(n, f0=0, f1=1):
    # this is automatically done: display(viz(varnames=["n"], mode="call"))
    if n < 0: return f0
    elif n == 0: return f1
    else: return n * factorial4(n - 1, f0=f0, f1=f1)
    
n = 4
print(f"{n}! = {factorial4(n)}")
G node52368264 globals n 4 node52300856 wrapped_function ...
G node52368264 globals n 4 node52300856 wrapped_function ... node52572888 factorial4 n 4 f0 0 f1 1 node52289640 wrapped_function ...
G node52368264 globals n 4 node52300856 wrapped_function ... node52572888 factorial4 n 4 f0 0 f1 1 node52289640 wrapped_function ... node52302344 factorial4 n 3 f0 0 f1 1 node51698888 wrapped_function ...
G node52368264 globals n 4 node52300856 wrapped_function ... node52572888 factorial4 n 4 f0 0 f1 1 node52289640 wrapped_function ... node52302344 factorial4 n 3 f0 0 f1 1 node51698888 wrapped_function ... node52518392 factorial4 n 2 f0 0 f1 1 node52508424 wrapped_function ...
G node52368264 globals n 4 node52300856 wrapped_function ... node52572888 factorial4 n 4 f0 0 f1 1 node52289640 wrapped_function ... node52302344 factorial4 n 3 f0 0 f1 1 node51698888 wrapped_function ... node52518392 factorial4 n 2 f0 0 f1 1 node52508424 wrapped_function ... node51700728 factorial4 n 1 f0 0 f1 1 node52233672 wrapped_function ...
4! = 24

Well it works, but it's not clean.

I would very much prefer something like rcviz, or a more powerful tool like pyan or pycallgraph.

From the standard library, I just discovered the trace module, but it seems to be oriented for the command line usage, not in a notebook.

Trying rcviz

In [278]:
!pip install git+https://github.com/DamnedFacts/rcviz
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/DamnedFacts/rcviz
  Cloning https://github.com/DamnedFacts/rcviz to /tmp/pip-req-build-5jsw_yji
Requirement already satisfied: graphviz in /usr/local/lib/python3.6/dist-packages (from rcviz==0.2) (0.16)
WARNING: You are using pip version 20.3.3; however, version 21.0.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
In [289]:
import rcviz
In [290]:
cg = rcviz.CallGraph()

@rcviz.viz(cg)
def factorial5(n):
    if n < 0: return 0
    elif n == 0: return 1
    else: return n * factorial5(n - 1)
    
n = 4
print(f"{n}! = {factorial5(n)}")

cg.render()
4! = 24
callviz: Rendering in inline in Jupyter Notebook
Out[290]:
%3 nodes=5 52454264 factorial5(4) ret: 24 51188824 factorial5(3) ret: 6 52454264->51188824 1 (⇑4) 52429000 factorial5(2) ret: 2 51188824->52429000 2 (⇑3) 52674728 factorial5(1) ret: 1 52429000->52674728 3 (⇑2) 52203256 factorial5(0) ret: 1 52674728->52203256 4 (⇑1)
In [291]:
cg = rcviz.CallGraph()

@rcviz.viz(cg)
def fibo(n):
    if n <= 0: return 0
    elif n == 1: return 1
    else: return fibo(n-1) + fibo(n-2)
    
n = 4
print(f"{n}! = {fibo(n)}")

cg.render()
4! = 3
callviz: Rendering in inline in Jupyter Notebook
Out[291]:
%3 nodes=9 52456296 fibo(4) ret: 3 50666760 fibo(3) ret: 2 52456296->50666760 1 (⇑5) 49738120 fibo(2) ret: 1 52456296->49738120 6 (⇑8) 51084392 fibo(2) ret: 1 50666760->51084392 2 (⇑3) 52434920 fibo(1) ret: 1 50666760->52434920 5 (⇑4) 50598744 fibo(1) ret: 1 51084392->50598744 3 (⇑1) 52216536 fibo(0) ret: 0 51084392->52216536 4 (⇑2) 51227256 fibo(1) ret: 1 49738120->51227256 7 (⇑6) 52430152 fibo(0) ret: 0 49738120->52430152 8 (⇑7)
In [292]:
from functools import lru_cache as memoize
cg = rcviz.CallGraph()

@rcviz.viz(cg)
@memoize(maxsize=None)
def fibo2(n):
    if n <= 0: return 0
    elif n == 1: return 1
    else: return fibo2(n-1) + fibo2(n-2)
    
n = 4
print(f"{n}! = {fibo2(n)}")

cg.render()
4! = 3
callviz: Rendering in inline in Jupyter Notebook
Out[292]:
%3 nodes=7 51835400 fibo2(4) ret: 3 52085992 fibo2(3) ret: 2 51835400->52085992 1 (⇑5) 52900488 fibo2(2) ret: 1 51835400->52900488 6 (⇑6) 52262952 fibo2(2) ret: 1 52085992->52262952 2 (⇑3) 53127960 fibo2(1) ret: 1 52085992->53127960 5 (⇑4) 52906936 fibo2(1) ret: 1 52262952->52906936 3 (⇑1) 53134024 fibo2(0) ret: 0 52262952->53134024 4 (⇑2)

More experiments on a "triple" recursive function

Just to complicate things more, let's write a "fibonnaci" like recursive function that calls three sub-terms:

In [338]:
cg = rcviz.CallGraph()

@rcviz.viz(cg)
def fibo_three(n):
    if   n <= 0: return 0
    elif n == 1: return 1
    elif n == 2: return 1
    else: return fibo_three(n-1) + fibo_three(n-2) + fibo_three(n-3)
In [339]:
n = 5
print(f"{n}! = {fibo_three(n)}")

cg.render()
5! = 7
callviz: Rendering in inline in Jupyter Notebook
Out[339]:
%3 nodes=13 54048328 fibo_three(5) ret: 7 54048888 fibo_three(4) ret: 4 54048328->54048888 1 (⇑7) 54052808 fibo_three(3) ret: 2 54048328->54052808 8 (⇑11) 54055048 fibo_three(2) ret: 1 54048328->54055048 12 (⇑12) 54049448 fibo_three(3) ret: 2 54048888->54049448 2 (⇑4) 54051688 fibo_three(2) ret: 1 54048888->54051688 6 (⇑5) 54052248 fibo_three(1) ret: 1 54048888->54052248 7 (⇑6) 54050008 fibo_three(2) ret: 1 54049448->54050008 3 (⇑1) 54050568 fibo_three(1) ret: 1 54049448->54050568 4 (⇑2) 54051128 fibo_three(0) ret: 0 54049448->54051128 5 (⇑3) 54053368 fibo_three(2) ret: 1 54052808->54053368 9 (⇑8) 54053928 fibo_three(1) ret: 1 54052808->54053928 10 (⇑9) 54054488 fibo_three(0) ret: 0 54052808->54054488 11 (⇑10)

Using this non-memoized function leads to already 13 nodes for just $n=5$!

In [340]:
cg = rcviz.CallGraph()

@rcviz.viz(cg)
@memoize(maxsize=None)
def fibo_three_memo(n):
    if   n <= 0: return 0
    elif n == 1: return 1
    elif n == 2: return 1
    else: return fibo_three_memo(n-1) + fibo_three_memo(n-2) + fibo_three_memo(n-3)
In [341]:
n = 5
print(f"{n}! = {fibo_three_memo(n)}")

cg.render()
5! = 7
callviz: Rendering in inline in Jupyter Notebook
Out[341]:
%3 nodes=10 54081128 fibo_three_memo(5) ret: 7 53618600 fibo_three_memo(4) ret: 4 54081128->53618600 1 (⇑7) 54046648 fibo_three_memo(3) ret: 2 54081128->54046648 8 (⇑8) 54045528 fibo_three_memo(2) ret: 1 54081128->54045528 9 (⇑9) 54044968 fibo_three_memo(3) ret: 2 53618600->54044968 2 (⇑4) 54080568 fibo_three_memo(2) ret: 1 53618600->54080568 6 (⇑5) 54056248 fibo_three_memo(1) ret: 1 53618600->54056248 7 (⇑6) 54046088 fibo_three_memo(2) ret: 1 54044968->54046088 3 (⇑1) 54047208 fibo_three_memo(1) ret: 1 54044968->54047208 4 (⇑2) 54047768 fibo_three_memo(0) ret: 0 54044968->54047768 5 (⇑3)

As before, we can see that the ternary tree is reduced: there is recursion only on the most lefty branch, thanks to memoization.

About rcviz

It's an awesome module! Thanks to @DamnedFacts/ for having his up-to-date fork.

String

In [294]:
import string
string.hexdigits
Out[294]:
'0123456789abcdefABCDEF'
In [295]:
viz(string.hexdigits)
Out[295]:
G node139755555686920 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 ' 0 1 2 3 4 5 6 7 8 9 a b c d e f A B C D E F '

Interactive test of all the modes

Using IPyWidgets:

In [317]:
from pprint import pprint
from ipywidgets import interact

Trying on all the objects used above:

In [321]:
objs = [
    string.ascii_lowercase,
    lolviz,
    data,
    squares,
    example_matrix,
    example_3D_matrix,
    anakin,
    anakin_rec,
    leia_rec,
    luke_rec,
    root,
    z,
]

And the different modes, except for "call" and "calls" which only work inside functions:

In [326]:
partial_modes = ['str', 'matrix', 'obj', 'tree', 'lol', 'list']
In [329]:
@interact(
    obj = objs,
    mode = partial_modes,
)
def showcase_unified_viz(obj, mode):
    print(f"\nObject of type = {type(obj)}, for mode = {mode}")
    pprint(obj)
    return viz(obj, mode=mode)
    return lolviz.strviz(obj)

Conclusion

Who it was interesting!

That's it. See this other example for more.