python 2.7 - Why a "Lost Paradise" in code, that ran many months in py2.7, now np.ndarray() not found in namespaces, while [import numpy as np] was executed? -
was there namespace handling change between 2.7.9 , 2.7.11?
tl;dr: after providing 450+ answers on stack overflow. know stack overflow rules , strived narrow/isolate issue, below-mentioned reasons, practically impossible post fair mcve.
below posted in-vivo diagnostics has proved, there no namespace-issues np.ndarray calls until first line trying access / use numpy ( np ) symbol line 14644 (14630 in orignal numbering)
adsegfloat = np.ndarray( ( 80, 7 ), ... where thereto working numpy call raises exception:
unboundlocalerror: local variable 'np' referenced before assignment
intro:
purpose of quantitative modelling, rather extensive python module code being re-used in distributed processing framework operated technical analyses , predictions. due fact, quant-models ( being used predictions ) generated @ remarkable cost of small tens-of-cpu-core-hours, there deployed additional measures allowing stateful module reload( quantfx )-s, these protect elaborated instances of quant-models, allow module functionalities updated, tweaked , re-tested on-the-fly.
facts:
import quantfx ceased work , throws exceptions on numpy calls, not modified:
- module code used last 12 months, updating [quant-tools],[model] sections
- the code python 2.7 part of multi-party distributed processing, attempt mcve never reflect / replicate ecosystem behaviour
when problem started:
after loading recent anaconda ( miniconda 4.0.0 / 4.0.5 ) package manager vm02/wxp host, python got re-dressed 2.7.11 , module called quantfx.py, being until update run without problems in other anaconda 2.2.0 (32-bit) installations in python 2.7.9 stopped working strange looking exception
what demonstrates ill-functioning state-of-operations:
import quantfx goes dozen months, week week. call quantfx.aminiresponder() results in exception / yields traceback if numpy not import-ed @ ( ref. line 221 ) but assumed local variable.
... [aminiresponder]: messaging & signalling sockets rto:<infrastructure-setup> start: sun apr 17 20:19:33 2016 ... [aminiresponder]: final attempt de-block remote-processing failed @ axmitsocket.send() sig->: sun apr 17 20:19:33 2016 pub-lished <_|sig_exit|_> signal peers, sleep( 3 ) allow reception zmq: sun apr 17 20:19:36 2016 graceful exit done. ret(0) traceback (most recent call last): file "<stdin>", line 1, in <module> file "quantfx.py", line 14630, in aminiresponder adsegfloat= np.ndarray( ( 80, 7 ), unboundlocalerror: local variable 'np' referenced before assignment ( while manual call same numpy function on interpreter command line prompt,
respecting namespace numpy import-ed module ( line 221 below )
went well
|>>> quantfx.np.ndarray( ( 2, 3 ), dtype = quantfx.np.float32, order = 'f' ) array([[ 1.22325565e+10, -2.81789980e-05, 1.22325565e+10], [ 1.91215026e+00, -1.81719875e+00, 1.91215026e+00]], dtype=float32) +0:00:13.282000 22:20:54 | |>>> quantfx.np.__version__ '1.10.4' +0:00:24.438000 22:21:18 | ).
line 14630 context of traceback , details:
line 221 import numpy np + concept state-full reloads:
while quantfx module coding might attract objections regard pep* et al, syntax-constructors decided allow both context-full modus operandi via import/reload() , fast , safe context-less light-weight operations in minimalistic ram/cpu consuming remote terminals via select/copy/paste single component-code diagnostic or control-cli purposes ( aremotekeyboard() or aminiresponder() ) without need import whole quantfx module ( provisioning late-imports, including said numpy ( line 14426 ) such quantfx-context-less modus operandi ).
what sought:
any explanations changes between 2.7.9 , 2.7.11, related observed np.ndarray() collisions, introduced last friday.
any recommendations alternative module syntax-constructors' layout of high value if such proposal preserves both statefullness during reload( quantfx ) , permits both context-full , context-less code-base component usage.
update:
in contrast possible appearance, post preceded due review , debugging efforts adding in-vivo self-diagnostics allow problem isolation
""" debug: vm02, after about-a-year working here stable... throws exc. here: "unboundlocalerror: local variable 'np' referenced before assignment" ????? .exc on np.ndarray() >>> import quantfx quantfx_flag seen in dir() during <module> import: true section: ____identify____ [try].ok section: ____identify____ [fin] section: import [try].ok section: import [fin] section: fx-market contexts [try].ok section: fx-market contexts [fin] section: general tools [try].ok section: general tools [fin] section: data inputs [try].ok section: data inputs [fin] section: v41 process [try].ok section: v41 process [fin] section: v41 quant tools [try].ok section: v41 quant tools [fin] quantfx.py:10966: runtimewarning: invalid value encountered in divide st2 = np.where( st2 != 0, ( st[-llv.shape[0]:] - llv ) / st2, st2 ) # / st2 ( == hhv - llv ) # __main__:4162: runtimewarning: invalid value encountered in divide section: v41 model [try].ok section: v41 model [fin] section: v41 predictor [try].ok section: v41 predictor [fin] section: metatrader responder [try].ok section: metatrader responder [fin] section: ____template____ [try].ok section: ____template____ [fin] section: ____template____ [try].ok section: ____template____ [fin] section: ____template____ [try].ok section: ____template____ [fin] +0:00:01.219000 18:20:14 | |>>> quantfx.aminiresponder( atarget2bind2_url = "tcp://10.0.0.62:9999", aninstrumentdictofparams = quantfx.anfxctx[ quantfx.actxid ] ) [aminiresponder]: runs in it's own full quantfx context [aminiresponder]: imports done [aminiresponder]: variables done [aminiresponder]: messaging & signalling sockets rto:[azmqcontext] <class 'zmq.sugar.context.context'> setup 15.2.0 version [aminiresponder]: messaging & signalling sockets [axmitsocket]-pre [aminiresponder]: messaging & signalling sockets (axmitsocket)-to instantiated-via call azmqcontext.socket( zmq.pair ) [aminiresponder]: messaging & signalling sockets [axmitsocket]-post [aminiresponder]: messaging & signalling sockets [axmitsocket]-bind() [aminiresponder]: messaging & signalling sockets rto:[axmitsocket] [aminiresponder]: messaging & signalling sockets rto:[actrlsocket] [aminiresponder]: messaging & signalling sockets rto:[asigssocket] [aminiresponder]: messaging & signalling sockets rto:<infrastructure-setup> start: sun apr 17 20:19:33 2016 ... [aminiresponder]: final attempt de-block remote-processing failed @ axmitsocket.send() sig->: sun apr 17 20:19:33 2016 pub-lished <_|sig_exit|_> signal peers, sleep( 3 ) allow reception zmq: sun apr 17 20:19:36 2016 graceful exit done. ret(0) traceback (most recent call last): file "<stdin>", line 1, in <module> file "quantfx.py", line 14630, in aminiresponder adsegfloat= np.ndarray( ( 80, 7 ), unboundlocalerror: local variable 'np' referenced before assignment in-vivo numpy-nd.array-symbol-checker
with in-vivo checker idea proposal @viraptor
def debug_check_np_ndarray( acallersideinspectframeinfo ): # debug-check #ebug_check_np_ndarray( acallersideinspectframeinfo = getframeinfo( currentframe() ) ) #rom inspect import currentframe, getframeinfo # external responsibility import + setup acallersideinspectframeinfo .oncall #ef np_check(): # def: try: # try: np.* np.ndarray print 70*" ", "* np: np.ndarray call [ok], file: ", acallersideinspectframeinfo.filename, " line: ", acallersideinspectframeinfo.lineno except: # exc: found issue print 70*" ", "* np: np.ndarray call [**], file: ", acallersideinspectframeinfo.filename, " line: ", acallersideinspectframeinfo.lineno exc_type, exc_value, exc_traceback = sys.exc_info() # prepare traceback detail traceback.print_tb( exc_traceback, limit = 5, file = sys.stdout ) raise else: # else: ok, passed #rint("* np check ok") # optionally add current function name via traceback module return on launch provides detailed line-by-line confirmation there no np.ndarray symbol-masking until line 14644 crashes on adsegfloat = np.ndarray( ... ):
|>>> quantfx.aminiresponder() * np: np.ndarray call [ok], file: quantfx.py line: 14387 * np: np.ndarray call [ok], file: quantfx.py line: 14391 [aminiresponder]: runs in it's own full quantfx context * np: np.ndarray call [ok], file: quantfx.py line: 14395 * np: np.ndarray call [ok], file: quantfx.py line: 14404 * np: np.ndarray call [ok], file: quantfx.py line: 14410 * np: np.ndarray call [ok], file: quantfx.py line: 14412 * np: np.ndarray call [ok], file: quantfx.py line: 14419 * np: np.ndarray call [ok], file: quantfx.py line: 14423 [aminiresponder]: runs in full quantfx mode, few imports done [aminiresponder]: imports done * np: np.ndarray call [ok], file: quantfx.py line: 14441 * np: np.ndarray call [ok], file: quantfx.py line: 14443 [aminiresponder]: variables done * np: np.ndarray call [ok], file: quantfx.py line: 14454 * np: np.ndarray call [ok], file: quantfx.py line: 14527 [aminiresponder]: messaging & signalling sockets rto:[azmqcontext] <class 'zmq.sugar.context.context'> setup 15.2.0 version [aminiresponder]: messaging & signalling sockets rto:<infrastructure-setup> * np: np.ndarray call [ok], file: quantfx.py line: 14626 start: tue apr 19 16:02:12 2016 ... * np: np.ndarray call [ok], file: quantfx.py line: 14632 * np: np.ndarray call [ok], file: quantfx.py line: 14643 [aminiresponder].finally: start attempts de-block remote-processing ( after prior intentional sig_exit or unhandled exc ) [aminiresponder]: final attempt de-block remote-processing failed @ axmitsocket.send() [aminiresponder]: final attempt propagate signal peers failed @ asigssocket.send() exc. zmqerror(zmqerror('no such file or directory')) on asigssocket .setsockopt( zmq.linger, 0 ) / .close( asigssocket ) exc. zmqerror(zmqerror('no such file or directory')) on actrlsocket .setsockopt( zmq.linger, 0 ) / .close( actrlsocket ) exc. zmqerror(zmqerror('no such file or directory')) on axmitsocket .setsockopt( zmq.linger, 0 ) / .close( axmitsocket ) zmq: tue apr 19 16:02:15 2016 graceful exit done. ret(0) traceback (most recent call last): file "<stdin>", line 1, in <module> file "quantfx.py", line 14644, in aminiresponder adsegfloat = np.ndarray( ( 80, 7 ), # # dsegfloat[][] ### todo.ms: size-independent ( 80+ ) np.*() ::::::::::::::: unboundlocalerror: local variable 'np' referenced before assignment +0:00:10.734000 16:02:15 |
from code you're showing, it's unlikely/impossible python update changed in screenshoted code.
what change either environment, or code in other places supposed not set aminiresponder_in_full_quantfx_context. in case variable set, import numpy np commented , you'd exception you're showing here.
some debugging ideas: either run in debugger, or use print-debug , break/print in places think numpy should imported - you'll find you're not going branches.
another one: change 1 thing @ time. if understand question correctly, changed: python, package manager , deployment environment in 1 go , discovered stuff doesn't work anymore in way don't understand. go original environment , retest. upgrade python , nothing else. retest. upgrade package manager (but not packages!). retest. ... until find change breaks app.
another one: when fails , can't recover (for example failed import), don't continue. on second screenshot you've got lot of import statements in try block, ignore errors , continue execution. (l 328-332). if continue, explode later on. if rest of code looks - yes, you'll end silent failures , no idea come from.
another 1 check @hpaulj's suggestion. crude approach: create function like:
def np_check(): try: np.ndarray except: # found issue raise else: print("* np check ok") # optionally add current function name via traceback module sprinkle around code find out where's first place when np name "disappears". put first call right after import expect work, see @ least 1 "* np check ok" before error. need add more checks between place of last success , first failure, until isolate line causes issues.
Comments
Post a Comment