Working on this platform with Micro Python has been pretty interesting. It's a strange combination of low level driver access patterns, memory limitations and debugging with higher level python and interpreted code. Using the WebREPL for remote debugging and serial Python console afford pretty convenient and fast print and repl testing. Breakpoints would be nice and I think you can do that with ESP32.
Early on I started getting out-of-memory (OOM) errors when loading all the python drivers for the peripherals I was using. But I made an error in judgement, and rather than use the built-in tools to track down the exact memory usage of each module and memory used while the program was running. It would have saved me some time.
I decided that since each module was not needed simultaneously, rather than load all the modules into memory, I'd load them as needed, and then unload them. When you use a python "import" statement, the module gets loaded into memory. To unload it, you have to jump through some hooks. An additional complication is that any other modules that the module loads also probably need to be unloaded to return to your original memory usage. Here is an example:
#ModuleA
import ModuleB
import ModuleC
def func(param1, param2):
...
So, say we want to call ModuleA.func() and then unload it when done. We might do something like:
import sys
print(sys.modules.keys())
import ModuleA
ModuleA.func(1, 2)
print(sys.modules.keys())
We might see something in the console like:
>>> import sys
>>> print(sys.modules.keys())
dict_keys(['flashbdev', 'websocket_helper', 'webrepl', 'webrepl_cfg'])
>>> import ModuleA
>>> ModuleA.func(1,2)
>>> print(sys.modules.keys())
dict_keys(['flashbdev', 'websocket_helper', 'webrepl', 'webrepl_cfg',
'ModuleA', 'ModuleB', ModuleC'])
Manually keeping track of these modules would be pretty painful. I started writing code to do that for about 30 seconds before I realized what a mess it would be to read and maintain.
So I decided to write a small module you could point to modules to execute functions and then automatically unload any newly loaded modules. I'd take a "Before" module snapshot, utilize the __import__ keyword to programmatically load modules by name, get the function, call it, save the result, determine any newly loaded modules, and then unload them. The interface looked like:
#Ephemeral Run
def eph_run(module, function, params=[])
You'd call it like:
eph_run("ModuleA","func", [1,2])
It also did some garbage collection right before the loading to reduce the chance of OOM.
And it worked pretty well. That is, until I needed to call a module that needed to retain state! It turns out that for the flow meter, there was a global variable that needed to store the pulse counter which was updated every time the ISR was invoked. If I loaded and immediately unloaded the module, the counter would be reinitialized.
So now, my model needed an exceptional case, one out of six cases where I didn't want to to unload automatically. But it should still unload when requested. What a pain. The new function was:
def eph_run_ex(module, functions, params, persist)
(Is this a dead giveaway of an old Windows programmer?)
I had eph_run utilize it with persist=False and the main difference was that the _ex version would returns a list of module names that got loaded. The modules would persist until you called eph_free() with the list of modules.
Starting to get messy, but it worked. I measured before and after and there was some savings. I structured the modules with common getters, setters, and test methods. The code was looking a lot less like python now (if that's even possible given my coding style).
I was pretty happy with this, I wasn't running out of memory immediately, although after several test sessions, I would run out of memory. Was it fragmentation? And I was still occasionally seeing what looked like OOM issue. This is when I did what I should have done earlier. I measured the memory of each module and its runtime usage more closely. I also optimized some of them down pruning out methods I didn't use in my app. And while doing so, I found some bugs in the meter code. I've come to the conclusion that there really should be enough memory for me to load all the drivers. It's possible the issues in my ISR which got called hundreds of times a minute were having some unexpected side effects and allocations in the driver code.
I made a new branch from the diversion and applied some of the newfound improvements. And I'm ok with this. Not all code lives on. Developers experiment and learn and any particular approach or model may or may not work out. I feel like good development is a balancing act of engineering using the scientific method. You make a hypothesis, set up an experiment, run tests and evaluate results. My hypothesis was that I was running out of memory because my modules were too large for available memory. My experiment sought to address that. I tested and while max memory usage was reduced, issues persisted. Finally comparing new vs old, I saw some something suspicious which led me to a new hypothesis. And so the process continues.
If this were not just a personal project in my free time, I'd follow up with more investigation to confirm that the issue I found was indeed the problem. It could be that is was one of multiple. And to be honest, since I don't completely understand exactly what was happening, this may come back to bite me later. But I want to move on and the MCP23008 an driver that this is happening with is a stop-gap due to limited GPIOs. I'm already planning rev2, and will be using an ESP32 which has more GPIOs. So I can save cost on the part and not even need this driver.
The latest code seems pretty stable. Here is the output of a 5 minute watering run with measurements output every minute plus a summary:
WebREPL connected
>>> import test
>>> test.water(5)
Watering for 5 minutes
Water valve open
Soil moisture: [509, 514, 462, 546, 550]
Flow rate: 20.8
Soil moisture: [470, 492, 460, 533, 549]
Flow rate: 11.2
Soil moisture: [460, 457, 444, 460, 547]
Flow rate: 12.8
Soil moisture: [459, 458, 444, 461, 524]
Flow rate: 12.8
Soil moisture: [459, 457, 443, 461, 503]
Flow rate: 12.5333
Closing water valve
Initial moisture: [509, 513, 461, 545, 548]
Final moisture: [511, 461, 446, 462, 498]
Water used: 70.13L
And reports for brightness and temperature:
>>> test.get_brightness()
5515
>>> test.get_temperature()
71.204
Here's a picture of the system in a somewhat waterproof box along side a separate box to house the power adapters. I also lay the brightness and temperature sensors on top of the electrical box.
Comments
Post a Comment