"Living in a noisy world...", using James Powell's (dutc) rwatch module

Goal : I want to write a context manager for Python 3.5+, so that inside the context manager, every number is seen noisy (with a white Gaussian noise, for instance).

It will be like being drunk, except that your it will be my Python interpretor and not me !

For instance, I will like to have this feature:

>>> x = 120193
>>> print(x)
120193
>>> np.random.seed(1234)
>>> with WhiteNoise():
>>>    print(x)
120193.47143516373249306

First, we will need numpy to have some random number generator, as numpy.random.normal to have some 1D Gaussian noise.

In [1]:
import numpy as np
In [2]:
np.random.seed(1234)
np.random.normal()
Out[2]:
0.47143516373249306

Then, the core part will be to install and import James Powell (dutc) rwatch module. If you don't have it installed :

  1. Be sure to have CPython 3.5. rwatch patches the CPython eval loop, so it's fixed to specific versions of Python & the author lazy about keeping it updated.
  2. Then pip install dutc-rwatch. It should work, but it fails for me.
  3. (alternative) You can just cd /tmp/ && git clone https://github.com/dutc/rwatch && cd rwatch/src/ && make and copy the rwatch.so dynamic library wherever you need...
In [5]:
%%bash
tmpdir=$(mktemp -d)
cd $tmpdir
git clone https://github.com/dutc/rwatch
cd rwatch/src/
make
ls -larth ./rwatch.so
file ./rwatch.so
# cp ./rwatch.so /where/ver/you/need/  # ~/publis/notebook/ for me
gcc `python3-config --cflags` `python3-config --includes` -fPIC -DPy_BUILD_CORE -c -o ceval.o ceval.c
gcc `python3-config --cflags` `python3-config --includes` -L`python3-config --prefix`/lib -Wl,--export-dynamic -fPIC -shared -o rwatch.so rwatch.c hook.c ceval.o -ldl `python3-config --libs`
-rwxrwxr-x 1 lilian lilian 453K mai   14 01:20 ./rwatch.so
./rwatch.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=f41ef744646e6247119bd702039725fc3a1c8e66, not stripped
Clonage dans 'rwatch'...
In file included from ceval.c:300:0:
ceval_gil.h:114:46: warning: initialization makes integer from pointer without a cast [-Wint-conversion]
 static _Py_atomic_address gil_last_holder = {NULL};
                                              ^~~~
ceval_gil.h:114:46: note: (near initialization for ‘gil_last_holder._value’)
ceval_gil.h: In function ‘create_gil’:
ceval_gil.h:145:5: warning: initialization makes integer from pointer without a cast [-Wint-conversion]
     _Py_atomic_store_relaxed(&gil_last_holder, NULL);
     ^~~~~~~~~~~~~~~~~~~~~~~~
ceval_gil.h: In function ‘drop_gil’:
ceval_gil.h:181:9: warning: initialization makes integer from pointer without a cast [-Wint-conversion]
         _Py_atomic_store_relaxed(&gil_last_holder, tstate);
         ^~~~~~~~~~~~~~~~~~~~~~~~
ceval_gil.h: In function ‘take_gil’:
ceval_gil.h:243:9: warning: initialization makes integer from pointer without a cast [-Wint-conversion]
         _Py_atomic_store_relaxed(&gil_last_holder, tstate);
         ^~~~~~~~~~~~~~~~~~~~~~~~

Anyhow, if rwatch is installed, we can import it, and it enables two new functions in the sys module:

In [6]:
import rwatch
from sys import setrwatch, getrwatch

setrwatch({})  # clean any previously installed rwatch
getrwatch()
Out[6]:
{}

Finally, we need the collections module and its defaultdict magical datastructure.

In [7]:
from collections import defaultdict

Defining a debugging context manager, just to try

This is the first example given in James presentation at PyCon Canada 2016.

We will first define and add a rwatch for just one object, let say a variable x, and then for any object using defaultdict. From there, writing a context manager that enables this feature only locally is easy.

Watching just one object

  1. Write the function, that needs two argument frame, obj, and should return obj,
  2. Install it...
  3. Check it!
In [15]:
def basic_view(frame, obj):
    print("Python saw the object {} from frame {}".format(obj, frame))
    return obj
In [16]:
x = "I am alive!"
In [17]:
setrwatch({
    id(x): basic_view
})
In [18]:
print(x)
Python saw the object I am alive! from frame <frame object at 0x7fe313a70b88>
Python saw the object I am alive! from frame <frame object at 0x7fe32c005258>
I am alive!

That's awesome, it works!

Can we delete the rwatch ?

Sure!

In [19]:
def delrwatch(idobj):
    getrwatch().pop(idobj, None)
In [20]:
print(x)
delrwatch(id(x))
print(x)  # no more rwatch on this!
print(x)  # no more rwatch on this!
Python saw the object I am alive! from frame <frame object at 0x7fe313a7b9a8>
Python saw the object I am alive! from frame <frame object at 0x7fe32c01a558>
I am alive!
Python saw the object I am alive! from frame <frame object at 0x7fe313a7b7c8>
I am alive!
I am alive!

We can also delete rwatches that are not defined, without a failure:

In [21]:
y = "I am Zorro !"
print(y)
delrwatch(y)  # No issue!
print(y)
I am Zorro !
I am Zorro !

More useful debuggin information

What is this frame thing? It is described in the documentation of the inspect module.

We can actually use it to display some useful information about the object and where was it called etc.

In [22]:
from inspect import getframeinfo

def debug_view(frame, obj):
    info = getframeinfo(frame)
    msg = '- Access to {!r} (@{}) at {}:{}:{}'
    print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
    return obj
In [23]:
setrwatch({})
setrwatch({
    id(x): debug_view
})
getrwatch()
Out[23]:
{140613264650544: <function __main__.debug_view>}
In [24]:
print(x)
- Access to 'I am alive!' (@0x7fe313b0b130) at <ipython-input-24-81745ac23551>:1:<module>
- Access to 'I am alive!' (@0x7fe313b0b130) at /usr/local/lib/python3.5/dist-packages/ipykernel/iostream.py:347:write
I am alive!

That can be quite useful!

Watching any object

We can actually pass a defaultdict to the setrwatch function, so that any object will have a rwatch!

Warning: obviously, this will crazily slowdown your interpreter!

So let be cautious, and only deal with strings here.

But I want to be safe, so it will only works if the frame indicate that the variable does not come from a file.

In [25]:
setrwatch({})
In [26]:
def debug_view_for_str(frame, obj):
    if isinstance(obj, str):
        info = getframeinfo(frame)
        if '<stdin>' in info.filename or '<ipython-' in info.filename:
            msg = '- Access to {!r} (@{}) at {}:{}:{}'
            print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
    return obj
In [27]:
setrwatch(defaultdict(lambda: debug_view_for_str))
In [28]:
print(x)
- Access to 'I am alive!' (@0x7fe313b0b130) at <ipython-input-28-81745ac23551>:1:<module>
I am alive!

Clearly, there is a lot of strings involved, mainly because this is a notebook and not the simple Python interpreter, so filtering on the info.filename as I did was smart.

In [29]:
setrwatch({})

But obviously, having this for all objects is incredibly verbose!

In [30]:
def debug_view_for_any_object(frame, obj):
    info = getframeinfo(frame)
    if '<stdin>' in info.filename or '<ipython-' in info.filename:
        msg = '- Access to {!r} (@{}) at {}:{}:{}'
        print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
    return obj

Let check that one, on a very simple example (which runs in less than 20 micro seconds):

In [52]:
print(x)
%time 123 + 134
I am alive!
CPU times: user 0 ns, sys: 0 ns, total: 0 ns
Wall time: 17.6 µs
Out[52]:
257
In [53]:
setrwatch({})
setrwatch(defaultdict(lambda: debug_view_for_any_object))
print(x)
%time 123 + 134
setrwatch({})
- Access to None (@0x55feae7373a0) at <ipython-input-53-734484c968c6>:2:<module>
- Access to None (@0x55feae7373a0) at <ipython-input-53-734484c968c6>:2:<module>
- Access to <built-in function print> (@0x7fe342f347e0) at <ipython-input-53-734484c968c6>:3:<module>
- Access to 'I am alive!' (@0x7fe313b0b130) at <ipython-input-53-734484c968c6>:3:<module>
I am alive!
- Access to None (@0x55feae7373a0) at <ipython-input-53-734484c968c6>:3:<module>
- Access to None (@0x55feae7373a0) at <ipython-input-53-734484c968c6>:3:<module>
- Access to <bound method InteractiveShell.get_ipython of <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7fe33b56d0b8>> (@0x7fe33b57d548) at <ipython-input-53-734484c968c6>:4:<module>
- Access to <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7fe33b56d0b8> (@0x7fe33b56d0b8) at <ipython-input-53-734484c968c6>:4:<module>
- Access to 'time 123 + 134' (@0x7fe312d049b0) at <ipython-input-53-734484c968c6>:4:<module>
CPU times: user 8 ms, sys: 0 ns, total: 8 ms
Wall time: 10.8 ms
- Access to 257 (@0x7fe312c98c70) at <ipython-input-53-734484c968c6>:4:<module>
Out[53]:
257
- Access to None (@0x55feae7373a0) at <ipython-input-53-734484c968c6>:4:<module>
- Access to <built-in function setrwatch> (@0x7fe328099678) at <ipython-input-53-734484c968c6>:5:<module>
- Access to {} (@0x7fe312d07908) at <ipython-input-53-734484c968c6>:5:<module>

It seems to work very well!

But it slows down everything, obviously the filtering takes time (for every object!) Computing 123 + 134 = 257 took about 10 miliseconds! That's just CRAZY!

A first context manager to have debugging for one object

It would be nice to be able to turn on and off this debugging tool whenever you want.

Well, it turns out that context managers are exactly meant for that! They are simple classes with just a __enter__() and __exit__() special methods.

First, let us write a context manager to debug ONE object.

In [54]:
class InspectThisObject(object):
    def __init__(self, obj):
        self.idobj = id(obj)
    
    def __enter__(self):
        getrwatch()[self.idobj] = debug_view

    def __exit__(self, exc_type, exc_val, exc_tb):
        delrwatch(self.idobj)

We can check it:

In [55]:
z = "I am Batman!"
print(z)

with InspectThisObject(z):
    print(z)

print(z)
I am Batman!
- Access to 'I am Batman!' (@0x7fe312d05af0) at <ipython-input-55-155013540368>:5:<module>
- Access to 'I am Batman!' (@0x7fe312d05af0) at /usr/local/lib/python3.5/dist-packages/ipykernel/iostream.py:347:write
I am Batman!
I am Batman!

The first debug information shows line 5, which is the line where print(z) is.

A second context manager to debug any object

Easy:

In [56]:
class InspectAllObjects(object):
    def __init__(self):
        pass

    def __enter__(self):
        setrwatch(defaultdict(lambda: debug_view_for_any_object))

    def __exit__(self, exc_type, exc_val, exc_tb):
        setrwatch({})

It will probably break everything in the notebook, but works in a basic Python interpreter.

In [57]:
with InspectAllObjects():
    print(0)
- Access to None (@0x55feae7373a0) at <ipython-input-56-7a0dc9fc43f8>:6:__enter__
- Access to None (@0x55feae7373a0) at <ipython-input-56-7a0dc9fc43f8>:6:__enter__
- Access to None (@0x55feae7373a0) at <ipython-input-57-3ce99ae3e8c0>:1:<module>
- Access to <built-in function print> (@0x7fe342f347e0) at <ipython-input-57-3ce99ae3e8c0>:2:<module>
- Access to 0 (@0x55feae761480) at <ipython-input-57-3ce99ae3e8c0>:2:<module>
0
- Access to None (@0x55feae7373a0) at <ipython-input-57-3ce99ae3e8c0>:2:<module>
- Access to None (@0x55feae7373a0) at <ipython-input-57-3ce99ae3e8c0>:2:<module>
- Access to <built-in function setrwatch> (@0x7fe328099678) at <ipython-input-56-7a0dc9fc43f8>:9:__exit__
- Access to {} (@0x7fe312b08208) at <ipython-input-56-7a0dc9fc43f8>:9:__exit__

The 5th debug information printed is Access to 0 (@0xXXX) at <ipython-input-41-XXX>:2:<module>, showing the access in line #2 of the constant 0.

In [58]:
with InspectAllObjects():
    print("Darth Vader -- No Luke, I am your Father!")
    print("Luke -- I have a father? Yay! Let's eat cookies together!")
- Access to None (@0x55feae7373a0) at <ipython-input-56-7a0dc9fc43f8>:6:__enter__
- Access to None (@0x55feae7373a0) at <ipython-input-56-7a0dc9fc43f8>:6:__enter__
- Access to None (@0x55feae7373a0) at <ipython-input-58-f38f4c4a3188>:1:<module>
- Access to <built-in function print> (@0x7fe342f347e0) at <ipython-input-58-f38f4c4a3188>:2:<module>
- Access to 'Darth Vader -- No Luke, I am your Father!' (@0x7fe312bedf90) at <ipython-input-58-f38f4c4a3188>:2:<module>
Darth Vader -- No Luke, I am your Father!
- Access to None (@0x55feae7373a0) at <ipython-input-58-f38f4c4a3188>:2:<module>
- Access to <built-in function print> (@0x7fe342f347e0) at <ipython-input-58-f38f4c4a3188>:3:<module>
- Access to "Luke -- I have a father? Yay! Let's eat cookies together!" (@0x7fe312b71f80) at <ipython-input-58-f38f4c4a3188>:3:<module>
Luke -- I have a father? Yay! Let's eat cookies together!
- Access to None (@0x55feae7373a0) at <ipython-input-58-f38f4c4a3188>:3:<module>
- Access to None (@0x55feae7373a0) at <ipython-input-58-f38f4c4a3188>:3:<module>
- Access to <built-in function setrwatch> (@0x7fe328099678) at <ipython-input-56-7a0dc9fc43f8>:9:__exit__
- Access to {} (@0x7fe312af3f08) at <ipython-input-56-7a0dc9fc43f8>:9:__exit__

We also see here the None and {} objects being given to the context manager (see the __enter__ method at first, and __exit__ at the end).


Defining a context manager to add white noise

Basically, we will do as above, but instead of debug information, a white noise sampled from a Normal distribution (i.e., $\sim \mathcal{N}(0, 1)$) will be added to any number.

Capturing any numerical value

To capture both integers and float numbers, the numbers.Number abstract class is useful.

In [59]:
from numbers import Number

Adding a white noise for numbers

This is very simple.

But I want to be safe, so it will only works if the frame indicate that the number does not come from a file, as previously.

In [60]:
def add_white_noise_to_numbers(frame, obj):
    if isinstance(obj, Number):
        info = getframeinfo(frame)
        if '<stdin>' in info.filename or '<ipython-' in info.filename:
            return obj + np.random.normal()
    return obj

Let us try it out!

In [63]:
np.random.seed(1234)
setrwatch({})
x = 1234
print(x)
getrwatch()[id(x)] = add_white_noise_to_numbers
print(x)  # huhoww, that's noisy!
print(10 * x + x + x**2)  # and noise propagate!
setrwatch({})
print(x)
print(10 * x + x + x**2)
1234
1234.4714351637324
1535547.9958216753
1234
1536330

It seems to work! Let's do it for any number then...

... Sadly, it's actually breaking the interpreter, which obviously has to have access to non-noisy constants and numbers to work !

We can lower the risk by only adding noise to complex numbers. I guess the interpreter doesn't need complex numbers, write?

In [64]:
def add_white_noise_to_complex(frame, obj):
    if isinstance(obj, complex):
        info = getframeinfo(frame)
        if '<stdin>' in info.filename or '<ipython-' in info.filename:
            return obj + np.random.normal() + np.random.normal() * 1j
    return obj
In [65]:
np.random.seed(1234)
setrwatch({})
y = 1234j
print(y)
setrwatch(defaultdict(lambda: add_white_noise_to_complex))
print(y)  # huhoww, that's noisy!
setrwatch({})
print(y)
1234j
(0.47143516373249306+1232.8090243052936j)
1234j

Awesome!

« Now, the real world is non noisy, but the complex one is! »

That's one sentence I thought I would never say!

WhiteNoiseComplex context manager

To stay cautious, I only add noise to complex numbers.

In [67]:
class WhiteNoiseComplex(object):
    def __init__(self):
        pass

    def __enter__(self):
        setrwatch(defaultdict(lambda: add_white_noise_to_complex))

    def __exit__(self, exc_type, exc_val, exc_tb):
        setrwatch({})

And it works as expected:

In [69]:
np.random.seed(120193)
print(120193, 120193j)
with WhiteNoiseComplex():
    print(120193, 120193j)  # Huhoo, noisy!
print(120193, 120193j)

print(0*1j)
with WhiteNoiseComplex():
    print(0*1j)  # Huhoo, noisy!
print(0*1j)
120193 120193j
120193 (-1.4079099824891002+120192.24863460941j)
120193 120193j
0j
(1.5232719431619033-1.3234485447145865j)
0j

Defining a generic noisy context manager

This will be a very simple change from the previous one, by letting the Noisy class accept any noisy function, which takes obj and return a noisy version of obj, only for complex-valued objects.

In [71]:
class Noisy(object):
    def __init__(self, noise):
        def add_white_noise_to_complex(frame, obj):
            if isinstance(obj, complex):
                info = getframeinfo(frame)
                if '<stdin>' in info.filename or '<ipython-' in info.filename:
                    return noise(obj)
            return obj

        self.rwatch = add_white_noise_to_complex

    def __enter__(self):
        setrwatch(defaultdict(lambda: self.rwatch))

    def __exit__(self, exc_type, exc_val, exc_tb):
        setrwatch({})
In [73]:
print(1j)
with Noisy(lambda obj: obj + np.random.normal()):
    print(1j)
print(1j)
1j
(0.5850862095293242+1j)
1j
In [74]:
print(1j)
with Noisy(lambda obj: obj * np.random.normal()):
    print(1j)
print(1j)
1j
(-0-0.582727025433589j)
1j
In [75]:
print(1j)
with Noisy(lambda obj: obj + np.random.normal(10, 0.1) + np.random.normal(10, 0.1) * 1j):
    print(1j)
print(1j)
1j
(10.101006520511199+11.184818402012283j)
1j

Conclusion

Clearly, that was a BAD idea, and not so useful.

But it was interesting!

I don't have any idea of a context where this could be useful, but still!