Changeset 7309


Ignore:
Timestamp:
Jul 12, 2009, 11:47:15 PM (15 years ago)
Author:
ole
Message:

Started migrating the caching module to Python 3.0 using the -3 flag.

Location:
anuga_core/source/anuga/caching
Files:
2 edited

Legend:

Unmodified
Added
Removed

  • anuga_core/source/anuga/caching/caching.py

    r7276 r7309  
    3030Public functions:
    3131
    32 cache(func,args) -- Cache values returned from func given args.
     32cache(my_F,args) -- Cache values returned from callable object my_F given args.
    3333cachestat() --      Reports statistics about cache hits and time saved.
    3434test() --       Conducts a basic test of the caching functionality.
    … …  
    5353import numpy as num
    5454
     55#from future
    5556
    5657cache_dir = '.python_cache'
    … …  
    112113  """
    113114
    114   if options.has_key(key):
     115  if key in options:
    115116    options[key] = value
    116117  else:
    … …  
    120121# Function cache - the main routine
    121122
    122 def cache(func,
     123def cache(my_F,
    123124          args=(),
    124125          kwargs={},
    … …  
    134135
    135136  USAGE:
    136     result = cache(func, args, kwargs, dependencies, cachedir, verbose,
     137    result = cache(my_F, args, kwargs, dependencies, cachedir, verbose,
    137138                   compression, evaluate, test, return_filename)
    138139
    139140  ARGUMENTS:
    140     func --            Function object (Required)
    141     args --            Arguments to func (Default: ())
    142     kwargs --          Keyword arguments to func (Default: {})   
    143     dependencies --    Filenames that func depends on (Default: None)
     141    my_F --            Callable object (Required)
     142    args --            Arguments to my_F (Default: ())
     143    kwargs --          Keyword arguments to my_F (Default: {})   
     144    dependencies --    Filenames that my_F depends on (Default: None)
    144145    cachedir --        Directory for cache files (Default: options['cachedir'])
    145146    verbose --         Flag verbose output to stdout
    146147                       (Default: options['verbose'])
    147148    compression --     Flag zlib compression (Default: options['compression'])
    148     evaluate --        Flag forced evaluation of func (Default: False)
     149    evaluate --        Flag forced evaluation of my_F (Default: False)
    149150    test --            Flag test for cached results (Default: False)
    150151    clear --           Flag delete cached results (Default: False)   
    … …  
    154155    A Python function call of the form
    155156
    156       result = func(arg1,...,argn)
     157      result = my_F(arg1,...,argn)
    157158
    158159    can be replaced by
    159160
    160161      from caching import cache
    161       result = cache(func,(arg1,...,argn))
     162      result = cache(my_F,(arg1,...,argn))
    162163
    163164  The latter form returns the same output as the former but reuses cached
    … …  
    165166  'result' and the arguments can be simple types, tuples, list, dictionaries or
    166167  objects, but not unhashable types such as functions or open file objects.
    167   The function 'func' may be a member function of an object or a module.
     168  The function 'my_F' may be a member function of an object or a module.
    168169
    169170  This type of caching is particularly useful for computationally intensive
    … …  
    177178
    178179  LIMITATIONS:
    179     1 Caching uses the apply function and will work with anything that can be
    180       pickled, so any limitation in apply or pickle extends to caching.
     180    1 Caching uses function(*args, **kwargs) to evaluate and will work
     181      with anything that can be pickled, so any limitation in function(,)
     182      or pickle extends to caching.
    181183    2 A function to be cached should not depend on global variables
    182184      as wrong results may occur if globals are changed after a result has
    … …  
    188190  Keyword args
    189191    Keyword arguments (kwargs) can be added as a dictionary of keyword: value
    190     pairs, following the syntax of the built-in function apply().
     192    pairs, following Python's 'extended call syntax'.
     193   
    191194    A Python function call of the form
    192195   
    193       result = func(arg1,...,argn, kwarg1=val1,...,kwargm=valm)   
     196      result = my_F(arg1,...,argn, kwarg1=val1,...,kwargm=valm)   
    194197
    195198    is then cached as follows
    196199
    197200      from caching import cache
    198       result = cache(func,(arg1,...,argn), {kwarg1:val1,...,kwargm:valm})
     201      result = cache(my_F,(arg1,...,argn), {kwarg1:val1,...,kwargm:valm})
    199202   
    200203    The default value of kwargs is {} 
    … …  
    202205  Explicit dependencies:
    203206    The call
    204       cache(func,(arg1,...,argn), dependencies = <list of filenames>)
     207      cache(my_F,(arg1,...,argn), dependencies = <list of filenames>)
    205208    Checks the size, creation time and modification time of each listed file.
    206209    If any file has changed the function is recomputed and the results stored
    … …  
    209212  Specify caching directory:
    210213    The call
    211       cache(func,(arg1,...,argn), cachedir = <cachedir>)
     214      cache(my_F,(arg1,...,argn), cachedir = <cachedir>)
    212215    designates <cachedir> where cached data are stored. Use ~ to indicate users
    213216    home directory - not $HOME. The default is ~/.python_cache on a UNIX
    … …  
    216219  Silent operation:
    217220    The call
    218       cache(func,(arg1,...,argn), verbose=False)
     221      cache(my_F,(arg1,...,argn), verbose=False)
    219222    suppresses messages to standard output.
    220223
    221224  Compression:
    222225    The call
    223       cache(func,(arg1,...,argn), compression=False)
     226      cache(my_F,(arg1,...,argn), compression=False)
    224227    disables compression. (Default: compression=True). If the requested compressed
    225228    or uncompressed file is not there, it'll try the other version.
    … …  
    227230  Forced evaluation:
    228231    The call
    229       cache(func,(arg1,...,argn), evaluate=True)
     232      cache(my_F,(arg1,...,argn), evaluate=True)
    230233    forces the function to evaluate even though cached data may exist.
    231234
    232235  Testing for presence of cached result:
    233236    The call
    234       cache(func,(arg1,...,argn), test=True)
     237      cache(my_F,(arg1,...,argn), test=True)
    235238    retrieves cached result if it exists, otherwise None. The function will not
    236239    be evaluated. If both evaluate and test are switched on, evaluate takes
    … …  
    242245  Obtain cache filenames:
    243246    The call   
    244       cache(func,(arg1,...,argn), return_filename=True)
     247      cache(my_F,(arg1,...,argn), return_filename=True)
    245248    returns the hashed base filename under which this function and its
    246249    arguments would be cached
    … …  
    248251  Clearing cached results:
    249252    The call
    250       cache(func,'clear')
    251     clears all cached data for 'func' and
     253      cache(my_F,'clear')
     254    clears all cached data for 'my_F' and
    252255      cache('clear')
    253256    clears all cached data.
    254257 
    255     NOTE: The string 'clear' can be passed an *argument* to func using
    256       cache(func,('clear',)) or cache(func,tuple(['clear'])).
     258    NOTE: The string 'clear' can be passed an *argument* to my_F using
     259      cache(my_F,('clear',)) or cache(my_F,tuple(['clear'])).
    257260
    258261    New form of clear:
    259       cache(func,(arg1,...,argn), clear=True)
    260     clears cached data for particular combination func and args
     262      cache(my_F,(arg1,...,argn), clear=True)
     263    clears cached data for particular combination my_F and args
    261264     
    262265  """
    … …  
    280283
    281284  # Handle the case cache('clear')
    282   if type(func) == types.StringType:
    283     if string.lower(func) == 'clear':
     285  if type(my_F) == types.StringType:
     286    if string.lower(my_F) == 'clear':
    284287      clear_cache(CD,verbose=verbose)
    285288      return
    286289
    287   # Handle the case cache(func, 'clear')
     290  # Handle the case cache(my_F, 'clear')
    288291  if type(args) == types.StringType:
    289292    if string.lower(args) == 'clear':
    290       clear_cache(CD,func,verbose=verbose)
     293      clear_cache(CD,my_F,verbose=verbose)
    291294      return
    292295
    … …  
    309312  deps = get_depstats(dependencies)
    310313
    311   # Extract function name from func object
    312   funcname = get_funcname(func)
     314  # Extract function name from my_F object
     315  funcname = get_funcname(my_F)
    313316
    314317  # Create cache filename
    … …  
    339342  # Check if previous computation has been cached
    340343  if evaluate is True:
    341     Retrieved = None  # Force evaluation of func regardless of caching status.
     344    Retrieved = None  # Force evaluation of my_F regardless of caching status.
    342345    reason = 5
    343346  else:
    344347    T, FN, Retrieved, reason, comptime, loadtime, compressed = \
    345         CacheLookup(CD, FN, func,
     348        CacheLookup(CD, FN, my_F,
    346349                    args, kwargs,
    347350                    deps,
    … …  
    368371      # Execute and time function with supplied arguments
    369372      t0 = time.time()
    370       T = apply(func,args,kwargs)
     373
     374      T = my_F(*args, **kwargs) # Built-in 'apply' deprecated in Py3K   
     375     
    371376      #comptime = round(time.time()-t0)
    372377      comptime = time.time()-t0
    … …  
    376381
    377382      # Save results and estimated loading time to cache
    378       loadtime = save_results_to_cache(T, CD, FN, func, deps, comptime, \
     383      loadtime = save_results_to_cache(T, CD, FN, my_F, deps, comptime, \
    379384                                       funcname, dependencies, compression)
    380385      if verbose is True:
    … …  
    785790# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    786791
    787 def CacheLookup(CD, FN, func, args, kwargs, deps, verbose, compression,
     792def CacheLookup(CD, FN, my_F, args, kwargs, deps, verbose, compression,
    788793                dependencies):
    789794  """Determine whether cached result exists and return info.
    … …  
    791796  USAGE:
    792797    (T, FN, Retrieved, reason, comptime, loadtime, compressed) = \ 
    793     CacheLookup(CD, FN, func, args, kwargs, deps, verbose, compression, \
     798    CacheLookup(CD, FN, my_F, args, kwargs, deps, verbose, compression, \
    794799                dependencies)
    795800
    … …  
    797802    CD --            Cache Directory
    798803    FN --            Suggested cache file name
    799     func --          Function object
     804    my_F --          Callable object
    800805    args --          Tuple of arguments
    801806    kwargs --        Dictionary of keyword arguments   
    … …  
    891896    return(None, FN, None, reason, None, None, None)
    892897
    893   # Get bytecode from func
     898  # Get bytecode from my_F
    894899  #
    895   bytecode = get_bytecode(func)
     900  bytecode = get_bytecode(my_F)
    896901
    897902  #print compare(argsref,args),   
    … …  
    926931    #
    927932    (T, FN, Retrieved, reason, comptime, loadtime, compressed) = \
    928         CacheLookup(CD, FN+'x', func, args, kwargs, deps,
     933        CacheLookup(CD, FN+'x', my_F, args, kwargs, deps,
    929934                    verbose, compression, dependencies)
    930935
    … …  
    948953# -----------------------------------------------------------------------------
    949954
    950 def clear_cache(CD,func=None, verbose=None):
    951   """Clear cache for func.
    952 
    953   USAGE:
    954      clear(CD, func, verbose)
     955def clear_cache(CD, my_F=None, verbose=None):
     956  """Clear cache for my_F.
     957
     958  USAGE:
     959     clear(CD, my_F, verbose)
    955960
    956961  ARGUMENTS:
    957962     CD --       Caching directory (required)
    958      func --     Function object (default: None)
     963     my_F --     Function object (default: None)
    959964     verbose --  Flag verbose output (default: None)
    960965
    961966  DESCRIPTION:
    962967
    963     If func == None, clear everything,
    964     otherwise clear only files pertaining to func.
     968    If my_F == None, clear everything,
     969    otherwise clear only files pertaining to my_F.
    965970  """
    966971
    … …  
    975980  # FIXME: Windows version needs to be tested
    976981
    977   if func:
    978     funcname = get_funcname(func)
     982  if my_F:
     983    funcname = get_funcname(my_F)
    979984    if verbose:
    980985      print 'MESSAGE (caching.py): Clearing', CD+funcname+'*'
    … …  
    10761081# -----------------------------------------------------------------------------
    10771082
    1078 def save_results_to_cache(T, CD, FN, func, deps, comptime, funcname,
     1083def save_results_to_cache(T, CD, FN, my_F, deps, comptime, funcname,
    10791084                          dependencies, compression):
    10801085  """Save computed results T and admin info to cache
    10811086
    10821087  USAGE:
    1083     save_results_to_cache(T, CD, FN, func, deps, comptime, funcname,
     1088    save_results_to_cache(T, CD, FN, my_F, deps, comptime, funcname,
    10841089                          dependencies, compression)
    10851090  """
    … …  
    11071112  savetime = time.time()-t0 
    11081113
    1109   bytecode = get_bytecode(func)  # Get bytecode from function object
     1114  bytecode = get_bytecode(my_F)  # Get bytecode from function object
    11101115  admtup = (deps, comptime, bytecode, funcname)  # Gather admin info
    11111116
    … …  
    13911396  elif type(T) == DictType:
    13921397      # Make dictionary ordering unique 
     1398     
     1399      # FIXME(Ole): Need new way of doing this in Python 3.0
    13931400      I = T.items()
    13941401      I.sort()   
    … …  
    14311438    iB = id(B)     
    14321439   
    1433     if ids.has_key((iA, iB)):
     1440    if (iA, iB) in ids:
    14341441        # A and B have been compared already
    14351442        #print 'Found', (iA, iB), A, B
    … …  
    15211528# -----------------------------------------------------------------------------
    15221529
    1523 def get_funcname(func):
     1530def get_funcname(my_F):
    15241531  """Retrieve name of function object func (depending on its type)
    15251532
    15261533  USAGE:
    1527     get_funcname(func)
     1534    get_funcname(my_F)
    15281535  """
    15291536
    15301537  import types, string
    15311538
    1532   if type(func) == types.FunctionType:
    1533     funcname = func.func_name
    1534   elif type(func) == types.BuiltinFunctionType:
    1535     funcname = func.__name__
     1539  if type(my_F) == types.FunctionType:
     1540    funcname = my_F.func_name
     1541  elif type(my_F) == types.BuiltinFunctionType:
     1542    funcname = my_F.__name__
    15361543  else:
    15371544    tab = string.maketrans("<>'","   ")
    1538     tmp = string.translate(`func`,tab)
     1545    tmp = string.translate(repr(my_F), tab)
    15391546    tmp = string.split(tmp)
    15401547    funcname = string.join(tmp)
    … …  
    15511558# -----------------------------------------------------------------------------
    15521559
    1553 def get_bytecode(func):
     1560def get_bytecode(my_F):
    15541561  """ Get bytecode from function object.
    15551562
    15561563  USAGE:
    1557     get_bytecode(func)
     1564    get_bytecode(my_F)
    15581565  """
    15591566
    15601567  import types
    15611568
    1562   if type(func) == types.FunctionType:
    1563     return get_func_code_details(func)
    1564   elif type(func) == types.MethodType:
    1565     return get_func_code_details(func.im_func)
    1566   elif type(func) == types.InstanceType:   
    1567     if callable(func):
     1569  if type(my_F) == types.FunctionType:
     1570    return get_func_code_details(my_F)
     1571  elif type(my_F) == types.MethodType:
     1572    return get_func_code_details(my_F.im_func)
     1573  elif type(my_F) == types.InstanceType:   
     1574    if hasattr(my_F, '__call__'):
    15681575      # Get bytecode from __call__ method
    1569       bytecode = get_func_code_details(func.__call__.im_func)
     1576      bytecode = get_func_code_details(my_F.__call__.im_func)
    15701577     
    15711578      # Add hash value of object to detect attribute changes
    1572       return bytecode + (myhash(func),)
     1579      return bytecode + (myhash(my_F),)
    15731580    else:
    1574       msg = 'Instance %s was passed into caching in the role of a function ' % str(func)
     1581      msg = 'Instance %s was passed into caching in the role of a function ' % str(my_F)
    15751582      msg = ' but it was not callable.'
    15761583      raise Exception, msg
    1577   elif type(func) in [types.BuiltinFunctionType, types.BuiltinMethodType]:     
     1584  elif type(my_F) in [types.BuiltinFunctionType, types.BuiltinMethodType]:     
    15781585    # Built-in functions are assumed not to change 
    15791586    return None, 0, 0, 0
    1580   elif type(func) == types.ClassType:
     1587  elif type(my_F) == types.ClassType:
    15811588      # Get bytecode from __init__ method
    1582       bytecode = get_func_code_details(func.__init__.im_func)   
     1589      bytecode = get_func_code_details(my_F.__init__.im_func)   
    15831590      return bytecode     
    15841591  else:
    1585     msg = 'Unknown function type: %s' % type(func)
     1592    msg = 'Unknown function type: %s' % type(my_F)
    15861593    raise Exception, msg
    15871594
    … …  
    15891596 
    15901597 
    1591 def get_func_code_details(func):
     1598def get_func_code_details(my_F):
    15921599  """Extract co_code, co_consts, co_argcount, func_defaults
    15931600  """
    15941601 
    1595   bytecode = func.func_code.co_code
    1596   consts = func.func_code.co_consts
    1597   argcount = func.func_code.co_argcount   
    1598   defaults = func.func_defaults       
     1602  bytecode = my_F.func_code.co_code
     1603  consts = my_F.func_code.co_consts
     1604  argcount = my_F.func_code.co_argcount   
     1605  defaults = my_F.func_defaults       
    15991606 
    16001607  return bytecode, consts, argcount, defaults 
    … …  
    19511958
    19521959          user     = record[1]
    1953           func     = record[2]
     1960          my_F     = record[2]
    19541961
    19551962          # Strip hash-stamp off
    19561963          #
    1957           i = find(func,'[')
    1958           func = func[:i]
     1964          i = find(my_F,'[')
     1965          my_F = my_F[:i]
    19591966
    19601967          size        = float(record[3])
    … …  
    19901997
    19911998            UpdateDict(UserDict,user,info)
    1992             UpdateDict(FuncDict,func,info)
     1999            UpdateDict(FuncDict,my_F,info)
    19932000          else:
    19942001            pass #Stats on recomputations and their reasons could go in here

  • anuga_core/source/anuga/caching/test_caching.py

    r7276 r7309  
    884884      print
    885885      print 'Initial_addr  ', initial_addr
    886       print 'Redefined addr', `Dummy_memorytest`
     886      print 'Redefined addr', repr(Dummy_memorytest)
    887887      msg = 'Redefined class ended up at same memory location as '
    888888      msg += 'original class making this test irrelevant. Try to run '
    889889      msg += 'it again and see if this error goes away.'
    890890      msg += 'If it persists contact Ole.Nielsen@ga.gov.au'
    891       assert initial_addr != `Dummy_memorytest`, msg   
     891      assert initial_addr != repr(Dummy_memorytest), msg   
    892892
    893893     
Note: See TracChangeset for help on using the changeset viewer.