Making Progress With Selects

One of our current mega-projects is converting a very large AREV system over to OpenInsight and of course like most large systems it relies heavily on batch processes that perform selects and refine these selects before presenting the results. AREV had the useful status line update facility to let you know how processing was going but regretfully OpenInsight omitted this - presumably because lacking a dedicated status line there was nowhere to put it. With hindsight a SEND_INFO every 1% would at least have allowed the developer to intercept and update the information from a select.

We tried the usual approach of looking the other way but when some of the queries were taking upwards of five minutes to return results to the user we decided that enough was enough.

The problem as we saw it was split into two areas - selects against entire non-indexed files and selects against indexes. This article deals with the former. The next article in this series will deal with the latter.

To start let's be frank. This blog isn't about giving away the family silver. It's about telling you what is possible using Rev products and giving you enough information to do it yourself if you want to. To this end we won't be publishing reams of code - we'll be publishing snippets and explanations of how to achieve the end result!

The steps involved At its simplest the easiest way of providing the user with a progress indicator when reading through a file or a resolved active select list is as follows :- include within the select statement an additional column which makes a callback to a user defined routine to provide feedback to the user about where we are in the process. Like most consultancy houses, Sprezz has a generic progress window that we use to keep the user informed of progress through a process. This is called in one of four ways :-

• To start the progress window
• To update the progress window
• To update the log portion of the progress window
• To end the progress window

Thinking about this, this means that for our select statement to report back to us, we need to do the following :-

• Initialise our counter variables and any labelled common
• Start our progress window
• Call our rlist select command
• From within the rlist select command call back to a routine to update the progress window
• On termination close down the progress window

Now that's all very well, but it's a faff to have to go in and change all of our existing calls to RLIST to do this set of actions, so it makes sense to create our own select program that encapsulates all of the above steps.

The Result Having done this we can then just issue a call to our replacement routine passing it the select statement required, and Bob as they say, is the brother of your mother. So in our simple example we issue the command :-

Call gen_Select_With_Progress( "SELECT CLAIMS" )

and the result we see is :-

(Please excuse the background colour of the MDI Frame. Like most development shops we use three systems, development, testing and live and we colour the background vividly to remind the person working on the system just where they currently are!)

The Breakdown So looking at the important parts, let's see how they work - firstly let's look at a simplified version of our gen_select_with_progress. Our full version has more flexibility built in but we've stripped this down to the bare bones to make it easier to understand.

The shell program

Function PList( object, method, param1, param2, param3, param4) /*    Author      AMcA

   Purpose     To do a select with a progress bar

*/

   equ version$   To "1.0.0"    $uses CMG*APPROWGEN_CONFIGURATION:GEN_RESOURCES    Declare Function get.reccount, gen_progress, retStack, zzx_res2Str    $insert gen_dict_callback_equates    $insert gen_module_equates    retVal = TRUE$    atSelf = retStack()<1>    if method = "" Or method = "SELECT" then       call set_status( 0 )        /*          assume that object contains the select statement To execute       */       object = trim( object )       table = field( object, " ", 2)       open table To vTable then          if @rec.count then             rowCount = @rec.count          end Else             rowCount = get.recCount( vTable, "", "")          end          loopCounter@ = 0 ; * loop counter          * start the progress window          retVal = gen_progress( "STARTWIN", rowCount, caption)          * log what is being done          retVal = gen_Progress("LOG", "", object)          * add our callback into the select - it will now be called once           * per row evaluation          addOn = " WITH ZZX_CALLBACK = 1"          if index( object , " WITH ", 1) then             addOn = " AND " : addOn          end          call rlist( object : addOn, 5, "", "", "" ) ; * do the actual select           call gen_progress( "END" )       end else          call fsMsg()       end    end else       locate method In "UPDATE" using "," setting pos then          on pos goSub update       end    end return retVal That copes with list items 1, 2, 3 and 5 so how does the CALLBACK dictionary item work? CALLBACK Dictionary Item If you place a compiled dictionary item into the SYSDICT table it can be used in all of your SELECT statements. All that this has to do is to call your update program. Again a simplified version of our SYSDICT item…    declare function gen_select_with_progress, isEventContext    $insert gen_dict_callback_equates    @ANS = ''    if isEventContext() then       dict   = @DICT       id     = @ID       mv     = @MV       record = @RECORD       rnc    = @RN.COUNTER         @ans = gen_select_with_progress( "", "UPDATE")         @DICT   = dict       @ID     = id       @MV     = mv       @RECORD = record       @RN.COUNTER = rnc    end return @ans As an aside, for efficiency we'd be better off using the transfer statement to move @RECORD and @DICT in the above code as all strings over 20 characters in length are held on the heap rather than in a descriptor, so a copy statement wastes resource. The Update Routine** Finally the part of gen_select_with_progress that updates the progress meter.

update:

   loopCounter@ += 1    if mod(20, loopCounter@) = 0 then       call gen_select_with_progress("UPDATE",loopCounter@, |                                     "{COUNT} of {MAX} - estimated time remaining|                                     {ETR}")    end

return 

The only thing to mention is that loopCounter@ is a labelled common variable. Because the update program and the calling dictionary item are on the program stack there is minimal overhead to this approach. It also allows for the user to press a cancel button and exit a protracted select. In our testing we've seen this add anywhere between 5 and 20% overhead. This might seem like a lot but when a user is staring at a blank screen waiting for a result time drags a lot more slowly. In addition the estimated end time allows the user to plan their time more effectively :).

In the next article we'll look at indexed selects.