Opened 17 years ago

Closed 17 years ago

#120 closed defect (fixed)

Make "set_quantity" read in blocks of file into memory and not the entire file.

Reported by: nick Owned by: duncan
Priority: normal Milestone:
Component: Efficiency and optimisation Version: 1.0
Severity: normal Keywords:

Description (last modified by nick)

Currently domain.set_quantity is very slow... especial as we are getting bigger and more detail elevation files.

When a large file (greater 200 Meg) is loaded with set_quantity the CPU with start using swap very quickly and the CPU drops to less that 10% :(

And loading a 700M file is taking ages.... 10hours?

especially bad as everytime the mesh changes this is recalculated

Also can this style of loading a file be applied to the file_boundary

Change History (9)

comment:1 Changed 17 years ago by nick

  • Owner changed from ole to duncan

comment:2 Changed 17 years ago by ole

This is a good idea. Here are some thoughts.

I would suggest equipping Geospatial data with the capability of returning one block at a time. To create from file

G = Geospatial_data(filename)

This will open filename and read in the first block. In case all data fits in a block, close the file again and all will be as usual.

G.get_data() will then return the block read in and if needed read in the next.

This can then be used in conjunction with the blocking already implemented in the least_squares code.

Cheers Ole

comment:3 Changed 17 years ago by nick

  • Priority changed from high to highest

An update on loading the 700Meg file.... it is still running after 4 days! need a solution ASAP

comment:4 Changed 17 years ago by nick

  • Description modified (diff)

comment:5 Changed 17 years ago by nick

More info...

I have cut down the 700meg file to about 125megs and it now loads in only 30min. This provides a idea of the size a file can be, to loaded quickly.

comment:6 Changed 17 years ago by nick

more info: a 165Meg .pts file loaded in under 30 mins

interesting hmmm...

ps. the 700meg .xya is still running... maybe i should kill it

comment:7 Changed 17 years ago by nick

There are 20,850,493 points in the 700Meg file

and there are 3,773,002 points in the 125 Meg file which loaded without thrashing

comment:8 Changed 17 years ago by duncan

  • Priority changed from highest to normal
  • Status changed from new to assigned

I've got it blocking for comma seperated value files. It doesn't block on .xya or .pts files, yet.

comment:9 Changed 17 years ago by duncan <duncan.gray@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

It blocks on .pts files now. It will not be made to block on xya files. They are being phased out

Note: See TracTickets for help on using tickets.