mò 7N5?c@sëdZdkZdkZdkZdkZdkZdkZdkZdkZdk Z dk Z dk Z dk l Z e iZdkTdZeZdZeeZdZdZdZdZd Zd Zd Zd Zd Zd Zdeifd„ƒYZdefd„ƒYZ d„Z!dfd„ƒYZ"dfd„ƒYZ#de#e"fd„ƒYZ$ee$ƒdfd„ƒYZ%de%e"fd„ƒYZ&ee&ƒde"fd„ƒYZ'ee'ƒe(djo%e$d d!d ƒZ)d d"d#d$d%d&ge*dƒZ+e,e)i-e+ƒZ.e,e)i/e.ƒZ0e+e0jo d'GHd(GHd)Ge+GHd*Ge0GHnd+GHe)i1ƒ\Z2Z3Z4d,e5e2ƒe5e3ƒe5e4ƒfGHd-Ge)i6i7Gd.Ge)i6i8GHd/Ge)i9i7Gd.Ge)i9i8GHd0GHd d"d#d$d%d&gd1d2gdZ+e,d3„e+e.ƒZ.e,e)i/e.ƒZ0e+e0jod4GHd5Ge+GHd6Ge0GHne)i1ƒ\Z2Z3Z4d,e5e2ƒe5e3ƒe5e4ƒfGHd-Ge)i6i7Gd.Ge)i6i8GHd/Ge)i9i7Gd.Ge)i9i8GHd7GHe.d8„Z:e)i;e:ƒe,e)i/e.ƒZ0e+e0jod4GHd5Ge+GHd6Ge0GHne)i1ƒ\Z2Z3Z4d,e5e2ƒe5e3ƒe5e4ƒfGHd-Ge)i6i7Gd.Ge)i6i8GHd/Ge)i9i7Gd.Ge)i9i8GHHe'd9d!d ƒZ<hZ=xCe*d:ƒD]5Z>d;e?e@e>ƒƒe>eAe>ƒfe=eBe>ƒGHe<i1ƒ\Z2Z3Z4d,e5e2ƒe5e3ƒe5e4ƒfGHd-Ge<i6i7Gd.Ge<i6i8GHd/Ge<i9i7Gd.Ge<i9i8GHH[)[<ndS(?s´ BeeStorage - Flatfile data storage facility. Definitions: block: minimal amount of storage allocated in the file record: Header + content + padding Copyright (c) 1998-2000, Marc-Andre Lemburg; mailto:mal@lemburg.com Copyright (c) 2000-2001, eGenix.com Software GmbH; mailto:info@egenix.com See the documentation for further information on copyrights, or contact the author. All Rights Reserved. N(sTools(t*i iièsÛsøsûs*cold***s*hot****iitErrorcBstZdZRS(s2 Baseclass for Errors related to this module. (t__name__t __module__t__doc__(((t@/data/zmath/lib/python2.4/site-packages/mx/BeeBase/BeeStorage.pyR;s tRunRecoveryErrorcBstZdZRS(s^ Error raised in case the storage file is damaged and recovery could be possible. (RRR(((RRAs cCsdS(sü This callback is used by the .collect() method to inform an object using the storage facility of a change in record layout. raw_data contains the raw data contents of the record. Use storage.decode() to decode it. N((t old_positiont new_positiontraw_data((Rtdummy_callbackJst BeeStoragecBstZdZdZeZeZdZeZe Z dZ dZ eZ dZeZeZdde ddd„Zeeeeed„Zed„Zed„Zed„Zd„Zeid ƒZd „Z e!i"e#d „Z$d „Z%d „Z&ee'd„Z(d„Z)d„Z*d„Z+e,i-e.e/e0e1fd„Z2e1e,i3e4i5e/ed„Z6de1e.e,i3e4i5e/ed„Z7e1d„Z8e0ed„Z9e9Z:e9Z;ed„Z<e<Z=e<Z>d„Z?e@iAd„ZBeBZCeDeEiFd„ZGd„ZHedd„ZIedd„ZJRS(s¾ Baseclass for flatfile data storage facilities. This class implements a simple database which can store and restore objects. The database is designed to be easily reconstructable in case something goes terribly wrong. File layout: · [Fileheader] (length FILEHEADERSIZE) · [Marker block] (length BLOCKSIZE) · STARTOFDATA: ...[datablock] (length mulitple of BLOCKSIZE)... Datablocks layout: · [ID] (1 byte) · [length of whole block] (4 bytes, little endian) · [CODE] (1 bytes) · [raw data] XXX Todo: * Implement write cache. * Add more recovery tools s1.2icCs||_totitƒn|tjo t}n|tdjo|tdt}n||_|ojt i |ƒ|_ } |o?y| i ƒWqÕt ij o| iƒ| i ƒqÕXqÙ| i ƒn|o d}nd}||_ y=t||ƒ|_}|iddƒ|iƒ|_}WnÍtj oÁ} |o t| ‚ntottd|ƒnt|dƒ|_}|i|ƒ|itƒ|iƒ}|tdjo)|idt|tƒ|iƒ}n||_d|_n Xd|_|oKd|_t i t!ƒ|_"} t i t!t#ƒ|_$} | | g|_%n|tdjo| ot&d |‚n|it'ƒ|i(t)t*ƒƒ} | t*jo4|o | t,j o| ot&d t-| ƒ‚nt|_.|i/|ƒd S( s» Create an instance using filename as data file. If lock is true, a filelock will be enabled until the file is closed. cache controls whether to enable a cache or not (should only be used where OS level caches are not available). min_recordsize can be set to have all records be padded to at least this size (might reduce overhead due to reallocation of records that have become too small for updates). readonly can be set to true to have existing files opened in read-only mode. Writes will then cause an IOError, opening a non-existing file in read-only mode will too. Opening the storage file in recover mode will disable some of the checks normally done. This flag should only be used if a previous normal opening failed with a hint to run recovery. iitrbsr+bisCreating a new storage file %ssw+bt s0storage file is damaged; run recovery ! (EOF=%i)s3storage file is damaged; run recovery ! (marker=%s)N(0treadonlytselft_debugtlogtcallt SYSTEM_DEBUGtmin_recordsizet MINRECORDSIZEt BLOCKSIZEtlocktFileLocktfilenametfilelocktrecoverRt remove_locktmodetopentfiletseekttelltEOFtIOErrortwhyt SYSTEM_INFOtwrite_fileheadertmarktCOLDtwritetis_newtcachetcachingtCachet MAXCACHESIZEt header_cachet CACHELIMITt record_cachetcachesRtFILEHEADERSIZEtreadtlent COLD_MARKERtmarkert HOT_MARKERtreprtstatetcheck_fileheader(RRRR+RRRR"RR7R/R1R$RR((Rt__init__ƒsn                * cCs¯|i|jodSn|io td‚nto+ttd|i||jdpdƒn|i i |ƒ||jo|i i |ƒn|i i |ƒ||_dS(s[ Change the state of the storage file. The state indicates whether the file has changed in a way that needs proper shutdown (HOT). An unchanged or stable file should be marked COLD. This is an internal interface, use .start/end_transaction() for externally setting the state. Nsstorage is read-onlysMarking the file "%s": %stHOTR((RR:RRRRRRR=RR R3R)R8R6(RR:R=R8R6R3((RR'ôs  " cCs|i|ddƒdS(sC Run recovery. callback is a call back function that will be called for every valid record and has the same signature as the one used for .collect(). To open a storage file in recovery mode, pass the keyword 'recover=1' to the constructor. RiN(Rtcollecttcallback(RR?((RRs cCs|i|ƒdS(s¥ Start a sequence of storage manipulation commands. Note that every write/free command automatically starts a transaction sequence. N(RR'R=(RR=((Rtstart_transaction scCs|i|ƒdS(s: End a sequence of storage manipulation commands. N(RR'R((RR(((Rtend_transaction+scCsd|ii|itf}|dtt|ƒdd}ttdjo"|dttdtt}n|i dƒ|i |ƒdS(sª Write a new header to the open file. Changes the file's position: moves the file's position to the start of the data area. s%s version %s blocksize %i R is itN( Rt __class__RtversionRt fileheaderR3R5RR R)(RRRE((RR&3s " s'(\w+) version ([\w.]+) blocksize (\d+) cCs|idƒ|itƒ}t|ƒtjotd|‚n|ii |ƒ}|djotd|‚n|i ƒ\}}}||iijotd||iif‚n||ijotd||if‚nti|ƒtjotd|tf‚ndS(s… Checks the file header and verifies that all parameters are the same. Changes the file's position. isheader is damaged: "%s"swrong header format: "%s"s%wrong storage class: %s (expected %s)swrong version: %s (expected %s)s$blocksize mismatch: %s (expected %i)N(RR R4R3RER5RRt header_checktmatchtmtNonetgroupstnameRDt blocksizeRCRtstringtatoiR(RRRKRERLRHRD((RR;Os  cCs td‚dS(sE Encode an object giving a string. Since the records are usually larger than the data size, it is important to store the string length or at least mark the end of data in some way. This method must be overloaded in order to implement an encoding schemes. s .encode() needs to be overriddenN(R(Rtobjectt StringTypettype((Rtencodehs cCs td‚dS(sD Decode a string giving an object. The data string may well be larger than the string returned by the .encode method. This method will have to determine the true length on its own. This method must be overloaded in order to implement an encoding scheme. s .decode() needs to be overriddenN(R(Rtdata((Rtdecodews cCs%|iot|idfƒndS(sQ Clears the caches used (flushing any data not yet written). tclearN(RR,t method_mapplyR2(R((Rt clear_cache„s cCs|io!|iƒ||idfƒn|io6|i o!|i|jo|i |ƒn|`n|i o|i i ƒ|` ndS(s6 Flush buffers and close the associated file. RUN( RR,tflushRVR2RRR:R(R'Rtunlock(RR(RV((Rtcloses      cCs7tottd|ƒn|io|iƒndS(Nsdel %s(RRRRRRZ(R((Rt__del__Ÿs cCsdS(s Flush all buffers. N((R((RRX¦scCs d|ii|it|ƒfS(Ns<%s instance for "%s" at 0x%x>(RRCRRtid(R((Rt__repr__¬sc CsP|ioB|ii|d ƒ}|d j o|ii|dƒ|SqLn||djp||i jot d|‚n|i}|i|ƒ|i dƒ}|pt d|‚n|d|jp|d|jot d|t|ƒf‚n|d|dd!ƒd}||d|df}|io|ii||ƒn|S( sÊ Read the header located at position and return a tuple (record size, statebyte, data area size). statebyte is one of the state constants. record size is the total number of bytes reserved for the record, data area size the number of bytes in its data area. self.file is positioned to point to the records data area. May raise errors if the position is invalid. iisinvalid position: %isposition %i is beyond EOFisinvalid header at %i: %sss    cCsp|i|jo|i|ƒn|i}|i|dƒ|i|ƒ|i ot |i d|fƒndS(s° Deletes an already written record by marking it OLD. The next garbage collection will make the change permanent and free the occupied space. itdeleteN( RR:R=R'RR R_R)tOLDR,RVR2(RR_RwR=R((RRrPs    cCs`|i|ƒ}|djo |i}n|i||ƒ}|io|i i ||ƒn|S(s‚ Write the encoded object to the file and return the file address where the data was written. If position is given or None, the object is assumed to be replacing an old data record. The implementation tries to use the old record for writing the new data. In case it doesn't fit the old record is marked OLD and another record is used. The return value will be different from the passed position in the latter case. Note: Records that are marked OLD will be collected by the next run of the garbage collection. N( RRRtobjRSR_RIR"RsR,R1Rf(RRxR_RS((RR)gs   cCs|i||ƒdS(N(RR)RxR_(RR_Rx((Rt __setitem__†scCsŸ|io7|ii||ƒ}||j o|i|ƒSqAn|i}|i |ƒ\}}}|i |ƒ}|io|ii||ƒn|i|ƒS(s= Load an object from the file at the given position. N(RR,R1R^R_t NotCachedRSRTRRgReRkRpR4Rf(RR_RzRkRRpRSRe((RR4Šs    cCsT|i}|tjo t}n||jo |}n|} g} g}g}|i }|i } | i }|i } | tdjo| tdt} nxª| |joœy|| ƒ\}}} Wn0tj o$| t} | | tfƒqnX|tjo|| |fƒn"|tjo| | |fƒn| |} qW| ||fS(sà Scans the data file for valid, old and invalid records and returns a list of positions to these records. start and end can be given to narrow down the search space. iiN(RR"tstartt STARTOFDATAtstopR_tvalidtinvalidtoldRgtappendt old_appendt valid_appendtinvalid_appendRReRkRpRRqRw(RR{R}R€R"RgRƒRkRR~R_RpR‚R„Re((Rt find_records¡s<             c CsÒt}|i}d}d}d}|i}x˜||joŠy||ƒ\} }}Wn*t j o|t }|t }q-nX|t jo|| }n|tjo|| }n|| }q-W|||fS(sÁ Scans the data file for valid, old and invalid records and returns a tuple valid, old, invalid indicating the number of bytes for each class of records/blocks. iN(R|R_RR"R~RR€RgReRkRpRRRqRw( RR€R"RgRkRR~R_RpRe((Rt statisticsËs*       cCsô|i}|o^|iddƒ|iƒ}|tdjo-t|t} |id| ƒ|| }qwn |i}|i }t }} |i } | o|iƒd|_ n|itjo|itƒnx5||jo'y||ƒ\} } }Wn;tj o/|t}|pttd|ƒqÕqÕnX| tjo¤|| joN|i|ƒ|i| ƒ}|i| ƒ|i|ƒ||| |dƒn<|o4|i|ƒ|i| ƒ}||| |dƒn| | } n| tjon|| }qÕWy|i| ƒWnPtj oD|i| ƒx2| |jo|idtƒ| t} q<WnX| }|tdjoN|o:|i|ƒt|t} |id| ƒ|| }qÓtd‚n||_| o d|_ ndS( s Collect garbage that accumulated since the last .collect() run. Garbage is collected by moving all VALID records to the beginning of the file and then truncating it to the new (reduced) size. Collecting will be done without using the cache. It also starts a new transaction (if not already marked HOT). For every move operation the callback is called with arguments (old_position,new_position,raw_data). raw_data is the raw data stored in the record that is being moved; use .decode[_key] to decode it. If recover is true, the callback is called for all valid records, not just the ones that are actually being moved. iiRBs+Skipping unallocated/misaligned block at %iisEOF malaligned after collect()iN(RRRR R!R"RtpadsizeR)RgR|tsourcetdestR,RWR:R=R'ReRkRpRRtSYSTEM_WARNINGRqR4trecordR?RwttruncatetAttributeError(RR?RR"RRgRˆRpR‹R‰ReRkR‡R,((RR>çs|                      i cCsœ|p|id}nt|dƒ}|itjo|itƒn|i}|i dƒx-|i |ƒ}|pPn|i |ƒqaW|iƒdS(sM Issues a backup request using archiveext as filename extension. The archive file is a simple copy of the current storage file. If no name is given self.filename + '.backup' is used. buffersize gives the size of the buffer used for copying the file. s.backuptwbiiN(tarchiveRRRtarchfileR:R=R'RR R4t buffersizetbufferR)RZ(RRR‘RR’R((RtbackupSs   (KRRRRDRIRRR"RRRR,RR:R*R/R1R<R=R8R6R3R'R RR@R(RAR&tretcompileRFR;ttypesRPRQRRRTRWRVRZR[RXR]tstructRdRRbRwRqRgRjRMRiRlRsRuRrRvt __delitem__R)RtaddRyR-RzR4t __getitem__R|tsystmaxintR…R†R>R“(((RR UsZ q        ,$H   * lt PickleMixincBs,tZdZeid„Zeid„ZRS(s4 Pickle encoding. Uses binary pickles. cCs ||dƒS(s Encode an object giving a string. This method can be overloaded in order to implement other encoding schemes. iN(tdumpsRO(RRORž((RRRzscCs ||ƒS(s Decode a string giving an object. This method can be overloaded in order to implement other encoding schemes. N(tloadsRO(RRORŸ((RRT…s(RRRtcPickleRžRRRŸRT(((RRts  tBeePickleStoragecBstZdZRS(s< Flatfile data storage facility for pickleable objects. (RRR(((RR¡s t MarshalMixincBs,tZdZeid„Zeid„ZRS(s Marshal encoding. cCs ||ƒS(s Encode an object giving a string. This method can be overloaded in order to implement other encoding schemes. N(RžRO(RRORž((RRRžscCs ||ƒS(s Decode a string giving an object. This method can be overloaded in order to implement other encoding schemes. N(RŸRO(RRORŸ((RRT©s(RRRtmarshalRžRRRŸRT(((RR¢™s  tBeeMarshalStoragecBstZdZRS(s> Flatfile data storage facility for marshallable objects. (RRR(((RR¤´s tBeeKeyValueStoragecBs‰tZdZeZd„Zeeid„ZeZ eZ ei d„Z ei eid„Zei eieid„ZeZd„ZRS(s‘ Flatfile storage for key,value pairs. keys and values must be pickleable object. The main difference between this class and the base class is that keys are readable separately from the values, e.g. values can be multi-MB big and are only read if this is really requested. NOTE: The .en/decode methods are NOT used. Uses binary pickles. cOsUtti|f||ƒ|io-titt ƒ|_ }|i i |ƒndS(N( tapplyR R<RtargstkwsR,R-R.R0t key_cacheR2R(RR§R¨R©((RR<Îs cCs€||dƒ||dƒ}|djo |i}n|i||ƒ}|i o*|i i ||ƒ|i i ||ƒn|S(s Write key and value to position. Returns the position under which the record was stored. If position is None, the implementation chooses a new one. iN( RžtkeytvalueRSR_RIRR"RsR,R1RfR©(RRªR«R_RžRS((RR)Õs    cCs ||ƒS(s9 Decode the key part of a raw data record field. N(RŸR (RR RŸ((Rt decode_keyñscCsy|io.|ii||ƒ}||j o|Sq8n|i|ƒ||i ƒ}|io|ii ||ƒn|S(sY Load the key part of an object from the file at the given position. N( RR,R©R^R_RzRªRgtloadRRf(RR_R­RzRª((Rtread_keyùs     c Cs |ioX|ii||ƒ}||j o2||ƒ}||ƒ}||ƒ} || fSqbn|i |ƒ\} }} |io]|i| ƒ}||ƒ}||ƒ}||ƒ} |ii||ƒ|ii||ƒn"|i}||ƒ}||ƒ} || fS(sl Load an object from the file at the given position and return it as tuple (key,value). N(RR,R1R^R_RzR‹tStringIORR­RªRSRgReRkRpR4RfR©( RR_R­R¯RzRªRkR‹RRpRSRe((RR4s(            cCs|i|ƒdS(s[ Load the value part of an object from the file at the given position. iN(RR4R_(RR_((Rt read_value2s(RRRRIR©R<R RžR)RR™RŸR¬R­R-RzR®t cStringIOR¯R4RšR°(((RR¥½s  "t__main__stest-BeePickleStorage.datR+tblablasHi theref2.3433999999999999ix0.07.0sBeePickleStorage: sResults differ:sorig:srest:sBeePickleStorage works.s% %i valid records, %i old, %i invalids r cache hits:s misses:s h cache hits:s rewrite...txidcCsti||ƒS(N(tfR)R«toldpos(R«R¶((RtQss Results differ:s orig:s rest:s collect...cCs|i|ƒ}|||tgtdtiR9tchrtocttstrtitemstvRtaddr(1R±R6RR¢R-R¾R RqR|R€R(R¥RRÍRRwR—RRRR”R~R R.R¡R R3RMRÌR›R=RRbR–RR8RÆRÅRµRÇR¸RHRmR?R0R»R¤R½R£((Rt?sÀQ    ÿÿÿ"  |  #   $(   $     $ 3 $