Make "set_quantity" read in blocks of file into memory and not the entire file.

[nick]

Currently domain.set_quantity is very slow... especial as we are getting bigger and more detail elevation files.

When a large file (greater 200 Meg) is loaded with set_quantity the CPU with start using swap very quickly and the CPU drops to less that 10% :(

And loading a 700M file is taking ages.... 10hours?

especially bad as everytime the mesh changes this is recalculated

Also can this style of loading a file be applied to the file_boundary

[ole]

This is a good idea. Here are some thoughts.

I would suggest equipping Geospatial data with the capability of returning one block at a time. To create from file

G = Geospatial_data(filename)

This will open filename and read in the first block. In case all data fits in a block, close the file again and all will be as usual.

G.get_data() will then return the block read in and if needed read in the next.

This can then be used in conjunction with the blocking already implemented in the least_squares code.

Cheers Ole

[nick]

An update on loading the 700Meg file.... it is still running after 4 days! need a solution ASAP

[nick]

More info...

I have cut down the 700meg file to about 125megs and it now loads in only 30min. This provides a idea of the size a file can be, to loaded quickly.

[nick]

more info: a 165Meg .pts file loaded in under 30 mins

interesting hmmm...

ps. the 700meg .xya is still running... maybe i should kill it

[nick]

There are 20,850,493 points in the 700Meg file

and there are 3,773,002 points in the 125 Meg file which loaded without thrashing

[duncan]

I've got it blocking for comma seperated value files. It doesn't block on .xya or .pts files, yet.

[duncan <duncan.gray@…>]

It blocks on .pts files now. It will not be made to block on xya files. They are being phased out