Opened 17 years ago

Closed 17 years ago

#8 closed enhancement (fixed)

Least squares memory issue (A vs AtA)

Reported by: ole Owned by: duncan
Priority: high Milestone: least_squares module finished
Component: Efficiency and optimisation Version:
Severity: normal Keywords:
Cc: ole

Description

memory issue: !

If LS is used for fitting

compute self.Atz and self AtA directly and avoid self.A. need alpha

If LS is to be used for interpolation

compute self.A no alpha needed

Rethink design and API so that this is addressed without duplicating code.

Change History (8)

comment:1 Changed 17 years ago by ole

  • Priority changed from normal to high

comment:2 Changed 17 years ago by ole

  • Component changed from pyvolution to least_squares

comment:3 Changed 17 years ago by anonymous

done changes, unchecked in, to least squares. after 20 hrs on the test_data_management test it is at this stage; N:\inundation\pyvolution>python test_data_manager.py Reading from karratha_100m.sww Reading quantity depth Reducing quantity depth Creating grid Building mesh Building interpolation matrix Number of datapoints: 12061352 Number of basis functions: 65227 Doing 0 of 12061352 Traceback (most recent call last):

File "test_data_manager.py", line 2225, in ?

runner.run(suite)

File "C:\Python23\lib\unittest.py", line 658, in run

test(result)

File "C:\Python23\lib\unittest.py", line 389, in call

test(result)

File "C:\Python23\lib\unittest.py", line 229, in call

testMethod()

File "test_data_manager.py", line 2212, in xxxtestz_sww2ers_real

verbose = True)

File "N:\inundation\pyvolution\data_manager.py", line 2876, in sww2ers

verbose = verbose)

File "N:\inundation\pyvolution\least_squares.py", line 355, in init

precrop = precrop)

File "N:\inundation\pyvolution\least_squares.py", line 545, in build_interpola

tion_matrix_A

element_found, sigma0, sigma1, sigma2, k = \

File "N:\inundation\pyvolution\least_squares.py", line 614, in search_triangle

s_of_vertices

sigma1 = dot((x-xi2), n1)/dot((xi1-xi2), n1)

KeyboardInterrupt?

Note, it was not using all of the availabe memory. Seems like a slow algorithm issue

comment:4 Changed 17 years ago by anonymous

ran same test as 29 sept. Same results, longer time (160 hrs). meemory usage 399,708 K, cpu at 100%. What was to screen when I stopped it: I:\inundation\pyvolution>test_data_manager.py Reading from karratha_100m.sww Reading quantity depth Reducing quantity depth Creating grid Building mesh Building interpolation matrix Number of datapoints: 12061352 Number of basis functions: 65227 Doing 0 of 12061352

Therefore not a memory problem, but a slow algorithm problem.

comment:5 Changed 17 years ago by duncan

  • Severity changed from normal to enhancement

comment:6 Changed 17 years ago by ole

  • Component changed from Testing and validation to Efficiency and optimisation

comment:7 Changed 17 years ago by duncan

  • Status changed from new to assigned

comment:8 Changed 17 years ago by duncan

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.