====== In Memory Hashtables in OpenInsight (Performance) ====== ====== ====== ==== Created at 25 APR 2013 01:10PM ==== ====== In Memory Hashtables in OpenInsight ====== ====== Overview ====== Storing frequently accessed values in memory rather than on disk is a technique for improving your applications performance.  One useful tool is an in-memory hashtable, also known as key/value storage. The hashtable lets you write a value to a key and read it later.  OpenInsight offers several tools to help you hold values in memory.  Each tool has strengths and weaknesses.   |Program |Description |Notes | |RTP65 |In memory hashtable, C dll |Fast.  No more than 100 tables.  Not designed for large number of Items.  No keys method. | |RTI_HASHTABLE |A wrapper around RTP65 |Same as RTP65, easier interface to program, adds keys  method | |Revelation.RTIDotNetHashtable|A COM interface to the .Net Hashtable |Unlimited number of tables\\ \\ Can store objects as well as strings.  Used by BFS connectors | |RTI_HASHTABLE2 |A Basic+ wrapper around the .Net hashtable |Same as Revelation.RTIDotNetHashtable, with easier interface | |Scripting.Dictionary |A Microsoft hashtable |Ole object you can use for key value storage. Unlimited number of tables.  Not designed for large number of keys.| |RTI_HASHTABLE3 |A Basic+ wrapper around the Scripting Dictionary |  | |Variable named commons |You can use variable named commons to implement a key/value sort of storage|Can store dimensioned arrays, multiple variables. Used by OpenInsight to implement window properties |   See RTP65 RTI_HASHTABLE Revelation.RTIDotNetHashtable Scripting.Dictionary Variable named commons ====== RTP65Subroutine ====== |Description |The RTP65 subroutine allows you to read and write key/value pairs to an in memory table.results. | |Syntax |RTP65( Method, Handle, Key, Record, Reserved, Status )\\ \\   | |Methods\\ \\   |0 - Create Table\\ \\ 1 - Open Table\\ \\ 2 - Close Table\\ \\ 3 - Clear Table\\ \\ 4 - Read Record\\ \\ 5 - Write Record\\ \\ 6 - Delete Record\\ \\  \\ \\ You will want to make equates for these methods. For example\\ \\  \\ \\ EQU CREATE_CACHE$  TO 0\\ \\ EQU OPEN_CACHE$    TO 1\\ \\ EQU CLOSE_CACHE$   TO 2\\ \\ EQU CLEAR_CACHE$   TO 3\\ \\ EQU READ_CACHE$    TO 4\\ \\ EQU WRITE_CACHE$   TO 5\\ \\ EQU DELETE_CACHE$  TO 6\\ \\ EQU MAX_CACHE_CNT$ to 100\\ \\   | |Method = Create table |rtp65 (CREATE_CACHE$, Handle,  tableName, '', '', Status)\\ \\  \\ \\ |Parameter|  |Description |\\ |Method |In |0 |\\ |Handle |  |Null |\\ |Key |In |Name of the table to create |\\ |Record |  |Null |\\ |Reserved |  |Null |\\ |Status |Out|0 - Success\\ \\ 1 - Table already exists\\ \\ 5 - Max number of tables exceeded|\\ \\  \\ \\   | |Method = Open Table |rtp65 (OPEN_CACHE$, Handle,  tableName, '', '', Status)\\ \\  \\ \\ |Parameter|  |Description |\\ |Method |In |1 |\\ |Handle |Out|Handle for the table |\\ |Key |In |Name of the table to open |\\ |Record |  |Null |\\ |Reserved |  |Null |\\ |Status |Out|0 - Success\\ \\ 2 - Table does not exist|\\ \\  \\ \\   | |Method = Clear table |rtp65 (CLEAR_CACHE$, Handle,  '', '', '', '')\\ \\  \\ \\ |Parameter|  |Description |\\ |Method |In|3 |\\ |Handle |In|Handle from previously opened table|\\ |Key |  |Null |\\ |Record |  |Null |\\ |Reserved |  |Null |\\ |Status |  |Null |\\ \\  \\ \\ Clear erases all keys and values from the table. The table remains open. | |Method = Read Record |rtp65 (READ_CACHE$, Handle,  Key, Record, '', Status)\\ \\  \\ \\ |Parameter|  |Description |\\ |Method |In |4 |\\ |Handle |In |Handle from previously opened table |\\ |Key |In |Key of the record |\\ |Record |Out|Value of the record |\\ |Reserved |  |Null |\\ |Status |Out|0 - Success\\ \\ 3 - Invalid handle\\ \\ 4 - Table not open or Record not found|\\ \\  \\ \\   | |Method = Write Record |rtp65 (WRITE_CACHE$, Handle,  Key, Record, '', Status)\\ \\  \\ \\ |Parameter|  |Description |\\ |Method |In |5 |\\ |Handle |In |Handle from previously opened table |\\ |Key |In |Key of the record |\\ |Record |In |Value of the record |\\ |Reserved |  |Null |\\ |Status |Out|0 - Success\\ \\ 3 - Invalid handle\\ \\ 4 - Table not open\\ \\ 6 - Memory Error|\\ \\  \\ \\  | |Method = Delete Record|rtp65 (DELETE_CACHE$, '',  tableName, '', '', Status)\\ \\  \\ \\ |Parameter|  |Description |\\ |Method |In |6 |\\ |Handle |In |Handle from previously opened table |\\ |Key |In |Key of the record |\\ |Record |  |Null |\\ |Reserved |  |Null |\\ |Status |Out|0 - Success\\ \\ 3 - Invalid handle\\ \\ 4 - Table not open or Record not found|\\ \\  \\ \\   |   |Remarks|RTP65 is simple and fast.  Before OI 9 is is limited to a total of 10 tables. As of 9.1 the limit is raised to 100 tabled. Openinsight uses a few tables itself.  RTP65 will slow down when there are a large number of keys.| |  |  |   |Examples |  | |  |  | |Example 1 |Subroutine RTP65_EXAMPLE(table)\\ \\ /*\\ \\ %%**%%  Demonstrate reading and writing cached recrods\\ \\ */\\ \\  \\ \\ Declare Subroutine RTP65\\ \\ $insert Logical\\ \\  \\ \\ EQU CREATE_CACHE$  TO 0\\ \\ EQU OPEN_CACHE$    TO 1\\ \\ EQU CLOSE_CACHE$   TO 2\\ \\ EQU CLEAR_CACHE$   TO 3\\ \\ EQU READ_CACHE$    TO 4\\ \\ EQU WRITE_CACHE$   TO 5\\ \\ EQU DELETE_CACHE$  TO 6\\ \\ EQU MAX_CACHE_CNT$ to 100\\ \\  \\ \\ If assigned(table) Else table = ''\\ \\ If table = '' then\\ \\     table = 'SYSPROCS'\\ \\ End\\ \\                \\ \\ Open table To f_table Else\\ \\                 isOk = false$\\ \\                 Return ''\\ \\ end\\ \\  \\ \\ * Create Cache\\ \\ CacheName = "MYCACHE"\\ \\ cache_status = 0\\ \\ hCache = ''\\ \\  \\ \\ rtp65 (CREATE_CACHE$, '',  CacheName, '', '', cache_status)\\ \\ isOk = ( cache_status eq 0 )\\ \\ If isOK then\\ \\                 rtp65 (OPEN_CACHE$, hCache, CacheName, '', '', cache_status)\\ \\                 isOk = ( cache_status eq 0 )\\ \\ End\\ \\  \\ \\ * put all the records in a cache\\ \\ Call Rlist('SELECT ' : table, 5, '', '', '')\\ \\ done = false$\\ \\ ids = ''\\ \\ Loop\\ \\                 Readnext id Else done = true$\\ \\ Until done Or Not(isOk)\\ \\                 Read record From f_table, id Then\\ \\                                 ids := id:@fm\\ \\                                 rtp65(WRITE_CACHE$, hCache, id, record, '', cache_status)\\ \\                                 isOk = ( cache_status eq 0 )\\ \\                 end\\ \\ repeat\\ \\  \\ \\ * pull them all back out\\ \\ col = ''\\ \\ Loop\\ \\                 Remove id From ids At col Setting mark\\ \\ While id ne "" And isOK\\ \\                 record = ''\\ \\                 rtp65(READ_CACHE$, hCache, id, record, '', cache_status)           \\ \\                 isOk = ( cache_status eq 0 )\\ \\ repeat\\ \\  \\ \\ rtp65 (CLEAR_CACHE$, hCache, '', '', '', cache_status)\\ \\  \\ \\ Return | |Example 2 - CACHE_MFS|Subroutine CACHE_MFS(CODE, BFS, HANDLE, NAME, FMC, RECORD, STATUS)\\ \\  \\ \\ /* Name : CACHE_MFS\\ \\ * Description:\\ \\ * Cache records in lists\\ \\  \\ \\ * Open -- put the cache number in the handle\\ \\ * Read/Reado -- add the record to the cache\\ \\ * write/delete/clear -- Update remote, local copy\\ \\ * omnievent, FMC = 3 - clear cache\\ \\ *\\ \\ *\\ \\ * Side Effects:\\ \\ * Cache does not expire, so once read, will not refresh from disk\\ \\ * Notes :\\ \\ can call RTP57(OMNI.SCRIPT, BFS, HANDLE, NAME, CLEAR_CACHE$, RECORD, STATUS) clear the cache manually\\ \\ Will not cache records with key[1,1] eq ‘%’\\ \\  \\ \\ */\\ \\  \\ \\ $insert Logical\\ \\ $insert FSErrors_100\\ \\ $Insert FILE.SYSTEM.EQUATES\\ \\  \\ \\ *$insert Cache_mfs_common\\ \\ EQU CREATE_CACHE$  TO 0\\ \\ EQU OPEN_CACHE$    TO 1\\ \\ EQU CLOSE_CACHE$   TO 2\\ \\ EQU CLEAR_CACHE$   TO 3\\ \\ EQU READ_CACHE$    TO 4\\ \\ EQU WRITE_CACHE$   TO 5\\ \\ EQU DELETE_CACHE$  TO 6\\ \\ EQU MAX_CACHE_CNT$ to 100\\ \\  \\ \\ EQU CACHE_HANDLE_DELIM$ to \F7\\\ \\ Equ CACHE_HANDLE_POS$ To 6\\ \\  \\ \\ declare subroutine rtp65\\ \\  \\ \\ common /cache_mfs_Common/init@,handle_cnt@,names@,file_handles@(max_cache_cnt$),cache_handles@,full_cache_flags@\\ \\ cache_nr = Field(handle, cache_handle_delim$, cache_handle_pos$,1)\\ \\ if cache_nr then\\ \\                 transfer handle to hold\\ \\                 handle = file_handles@(cache_nr)\\ \\                 hCache = cache_handles@\\ \\                \\ \\                 $Insert File.System.OnGoSub\\ \\                \\ \\                 transfer hold to handle\\ \\ end else\\ \\                 $Insert File.System.OnGoSub\\ \\ end\\ \\  \\ \\ Return\\ \\ * -------------- Main Subs -----------------\\ \\ READ.RECORD:\\ \\ READO.RECORD:\\ \\ RECORD = ''\\ \\ cache_status = ''\\ \\ if name[1,1] = '%' then\\ \\                 GOSUB NEXT.MFS\\ \\ end else\\ \\                 rtp65(READ_CACHE$, hCache, NAME, RECORD, '', cache_status)\\ \\                 If cache_status = 0 then\\ \\                                 status = true$\\ \\                 End else\\ \\                                 * read from file\\ \\                                 GOSUB NEXT.MFS                          \\ \\                                 if status then\\ \\                                                 * cache for next read\\ \\                                                 rtp65(WRITE_CACHE$, hCache, NAME, RECORD, '', cache_status)\\ \\                                 end\\ \\                 End\\ \\ end\\ \\ Return\\ \\  \\ \\ WRITE.RECORD:\\ \\ GOSUB NEXT.MFS\\ \\ if status then\\ \\                 rtp65(WRITE_CACHE$, hCache, NAME, RECORD, '', cache_status)\\ \\ End\\ \\ Return\\ \\  \\ \\ DELETE.RECORD:\\ \\ GOSUB NEXT.MFS\\ \\ rtp65(DELETE_CACHE$, hCache, NAME, '', '', cache_status)\\ \\ Return\\ \\  \\ \\ CLEARFILE:\\ \\ GOSUB NEXT.MFS\\ \\ rtp65(CLEAR_CACHE$, hCache, NAME, '', '', cache_status)\\ \\ Return\\ \\  \\ \\ DELETE.FILE:\\ \\ GOSUB NEXT.MFS\\ \\ rtp65(CLEAR_CACHE$, hCache, NAME, '', '', cache_status)\\ \\ Return\\ \\ * ---------------------------------------\\ \\ * non-chained filing system calls\\ \\ * ---------------------------------------\\ \\ Flush:\\ \\ Unlock.All:\\ \\ Record = ""\\ \\ Status = TRUE$\\ \\ Return\\ \\  \\ \\ Install:\\ \\ if Assigned(init@) Else init@ = false$\\ \\ If init@ else\\ \\                 init@ = true$\\ \\                 handle_cnt@ = 0\\ \\                 mat file_handles@ = ''\\ \\                 cache_handles@ = ''\\ \\                 names@ = ''\\ \\                 full_cache_flags@ = ''\\ \\ end\\ \\ Status = TRUE$\\ \\  \\ \\ Return\\ \\ * ---------------------------------------\\ \\ * ---------------------------------------\\ \\ * Chained Filing System Calls\\ \\ * ---------------------------------------\\ \\ LOCK.RECORD:\\ \\ UNLOCK.RECORD:\\ \\ GOSUB NEXT.MFS\\ \\ RETURN\\ \\  \\ \\ *--------------------------------------------------\\ \\ SELECT:\\ \\ READNEXT:\\ \\ CLEARSELECT:\\ \\ RECORD.COUNT:\\ \\ * ---------------------------------------\\ \\ CREATE.INDEX:\\ \\ DELETE.INDEX:\\ \\ UPDATE.INDEX:\\ \\ SELECT.INDEX:\\ \\ READNEXT.INDEX:\\ \\ * ---------------------------------------\\ \\ Omni.Script:\\ \\ omniCall = If Assigned(FMC) then FMC<1> Else ''\\ \\ If omniCall eq CLEAR_CACHE$ Then\\ \\    If hCache then\\ \\       rtp65 (CLEAR_CACHE$, hCache, '', '', '', cache_status)\\ \\    end\\ \\ End else\\ \\    gosub Next.MFS\\ \\ end\\ \\ Return\\ \\ Reserved:\\ \\ * there is a critical error if this line is reached\\ \\ Status = FALSE$\\ \\ Return\\ \\ * ---------------------------------------\\ \\ OPEN.FILE:\\ \\ GOSUB NEXT.MFS\\ \\ if status then\\ \\                 cache_status = ''\\ \\                 locate name in names@ using @fm setting pos then\\ \\                                 * File has already been opened once\\ \\                                 * Is it the same file or a new one with same name.  The handles should tell.\\ \\                                 * If the new handle (RECORD) is different, then the cached records are invalid\\ \\                                 hCache = cache_handles@\\ \\                                 orig_handle  = file_handles@(pos)\\ \\                                 if record # orig_handle then\\ \\                                                 rtp65 (CLEAR_CACHE$, hCache, '', '', '', cache_status)\\ \\                                 end\\ \\                 end else\\ \\                                 hcache = ''\\ \\                                 handle_cnt@+=1\\ \\                                 if handle_cnt@ < = max_cache_cnt$ then\\ \\                                                 * Create Cache\\ \\                                                 rtp65 (CREATE_CACHE$, '',  NAME, '', '', cache_status)\\ \\                                                 rtp65 (OPEN_CACHE$, hCache, NAME, '', '', cache_status)\\ \\                                                 rtp65 (CLEAR_CACHE$, hCache, '', '', '', cache_status)\\ \\                                                \\ \\                                                 * add it to the list\\ \\                                                 handle_cnt@+=1\\ \\                                                 names@ = name\\ \\                                                 cache_handles@=hCache\\ \\                                                 file_handles@(handle_cnt@)= record\\ \\                                                 full_cache_flags@= false$\\ \\                                 end\\ \\                 end\\ \\                 * embed the cache in RECORD, which is the handle of the file being opened\\ \\                 if hCache then\\ \\                                 record = FieldStore(record, CACHE_HANDLE_DELIM$, CACHE_HANDLE_POS$,1, handle_cnt@ )\\ \\                 end\\ \\ End\\ \\ retval = status\\ \\ Return\\ \\  \\ \\ CREATE.FILE:\\ \\ RENAME.FILE:\\ \\ MOVE.FILE:\\ \\ REMAKE.FILE:\\ \\ * ----------------------------------------\\ \\ Open.Media:\\ \\ CREATE.MEDIA:\\ \\ READ.MEDIA:\\ \\ WRITE.MEDIA:\\ \\ Close.Media:\\ \\ gosub Next.MFS\\ \\ Return\\ \\ * ---------- End of Subroutine\\ \\ * ==================================\\ \\ * execute filing system chain\\ \\ * ==================================\\ \\ Next.MFS:\\ \\ * Strips this MFS leaving the next fs as first element in array\\ \\ FSList = delete(BFS, 1, 1, 1)\\ \\ NextFS = FSList<1,1,1>\\ \\ if len(NextFS) then\\ \\                 call @NextFS(Code, FSList, Handle, Name, Fmc, Record, Status)\\ \\ End\\ \\ Return|