Separating data into pages

From SLIS Second Life Wiki

[edit] Proposal for separating data into pages

Edmund Edgar, 2007-09-03

Sometimes data we want to fetch from Moodle is too long to be handled by a single script's available memory.

To get around this problem, we will split responses to requests from LSL scripts into pages.

By default, a page will be 2048 bytes long. However, if multi-byte characters span the 2048-byte boundary, the page size returned will be reduced to the nearest whole character.

If an optional argument called sloodlepagesize is passed by the client as a GET or POST argument, a page of the specified size (in bytes) will be used instead.

If an optional argument called sloodlepageresponseid is passed by the client as a GET or POST argument, data from that page response will be used. If data from that page response is not available, an error will be returned. [TODO: Specify this and other errors and put them in the error codes list]

If an optional argument called sloodlepagestart is passed by the client as a GET or POST argument, the response starting at that byte number will be returned.

When a request is made whereby the response has to be split into pages, the server will return the following information in its header: - The ID of the response. - The number of the first byte returned. - The number of bytes being returned. - The number of bytes remaining. NB. This differs from my original proposal of using numbered pages. Using numbers of bytes works better because a) it allows scripts to vary the page size depending on available memory. b) it allows pages to be slightly smaller to avoid breaking multi-byte characters.

The server will save the entire reponse in a dedicated database table called prefix_sloodle_page_responses, with a unique ID for each response. This means that even if the content of the data returned has changed between page requests, the server will continue to provide data that was current at the time of the original request.

TODO: Figure out if there's some point where we can delete this data.

TODO: Figure out what to do on the client side - split this into a different script or what...