I've posted previously on this issue, but was unable to get an answer to resolve the problem.

It's not so much that the entire index search is slow but a particular part has slowed down. I'll elaborate.

It brings up the shared index relatively quickly and the index values you want to select can be browsed and entered with good speed. Once the information is saved it will popup XXXX rows selected in a couple of seconds. The problem is that it will take from 20-30 seconds to display the resulting popup.

Any help on this would be appreciated, it's becoming a frustrating problem.

Sounds as though you are getting several hundred hits for the popup? If so, then reading several hundred records over a slow network will take a few seconds.

What do you mean "the information is saved"? Are you doing a SAVELIST? If you are, then naturally it will take some time to convert the keylist from a cursor to the savelist file. However your LISTS file maybe in serious need of cleanup. Unless your application is poorly written and uses the LISTS file for application settings and tables you can probably do a clearfile on it. You may want to do a dump on it and check the sizelock setting too.

The search is not your problem. You wrote that it is the popup that is taking too long to come up.

Is any of the stuff you're using in-house? Like the pop-up? I believe the native ARev stuff passes the keys selected into the popup instantly. If the popup is home-made, maybe there is some slow stuff in it. Also, if the popup has a lot of work to do displaying fields, I've seen that slow things down too. I.e., if the field in question is a complex symbolic, you 're sunk. The strategy used to provide the information to the popup is probably at the root of your problem.

Thanks for the input, but it doesn't seem how any of those are to blame.

I'll explain how the index is created.

The fields to be included in the index search are dependent upon flags set by the client i.e. the number of index fields and their type are variable. The basic procedure for generating the index search is as follows:

1) Determine fields to include in search

2) Construct variable containing field information

3) Pass variable to CATALYST with 'B' option.

There is no SAVE-LIST performed. (When I said saved, I meant when you hit F9 to "save/enter" the index values that you want to search on).

The example that I outlined in my original post was an index search on 3 fields. All of them were data fields, no symbolics. The number of values returned was 4. I don't see why it should take so long to display these values.

Hopefully, this additional information will be helpful.

It sounds to me like it may be bogged down in updating the saved queries. To eliminate this as a facter, set number of saved queries in your environment to 0. What will happen if this is not 0 is that the system will be busy "forgetting" the oldest query (press CTRL-F10 to get a listing of saved queries). If the bottom query is a very large list, it will take a while to "forget".

You also might want to check the sizelock on SYSTEMP and LISTS.

Indexing is one of my weaker areas, but I believe you will have to live with it since you are selecting on multiple indexed fields.

   As you are aware, indexing is quicker because the system maintains lists of record keys associated with a specific value for the indexed field.  Instead of opening each record in the file and verifying whether it contains the indexed value, the system opens the list and retrieve the record keys.  This works well when selecting on one indexed field.

   If one tries to search on several indexed fields, the best that can be done, as far as I know, is that the system retrieves the list for one of the indexed values.  Once it has those record keys, it cannot do a second indexed search; there is no index of the indexed list.  The system has to sequentially read the record for each key in the list to determine whether it satisfies the criteria of the other filtered elements.

   I think the ARev manual lists in which situations it will use the indexes to resolve a filter based upon multiple fields.

I'm still unclear about several things:

How long is it before the message pops up with the number of records returned? That is when the search is occurring. I got the impression from your original post on this string that this was not where the hangup occurred. Rather, the hangup occurred after the values (you mentioned there were 4 records returned) were extracted, when the popup was being built. The only thing that can slow down a popup from being built and executed is if the popup has to read the records of the keys passed in to get display values for the columns you have in the popup. Since there are only four keys returned, that will be instantaneous.

Other responses are talking about some of the other events that occure during index searches.

Are all of the fields in the query indexed? I may have missed this. If two parts of the query narrow the list minimally, say from 50,000 records to 49,000, and the last one narrows it down to 4, but is not indexed, that will slow the search down. But I think I understand that the hangup occurs after the search.

If there is some kind of hangup in building the popup after the four records are extracted, and sometimes things happen that we didn't anticipate but that really hose you, you can do a loop through the four records and build and execute the popup yourself. That might be a good exercise anyway, and might give you a clue as to where the hangup is.

When you execute the CATALYST function, it returns the four values and ARev places them in the "active list", whatever/where ever that is. The next operation that occurs will try to use that active list, if it has a hook that checks for active lists, and it will grab the list and do something with it. Popups will operate on active lists, and that sounds like what you are doing.

Do you find any delays like this in other areas ofyour application? Maybe there is a rogue MFS out there.

Okay, I looked at your old message and it said the slow down started when you modified the dictionaries.

How are the systems configured:

- versions of ARev,workstation OS, Network client, network, network protocol etc?

- Arev configured how? Indexing station, case insensitive queries, update indexes on queries?

What did you modify in the dictionary?

Were these previously indexed fields?

Did the modifications require you to convert data?

What kind of data are indexed?

Did you rebuild the indexes? If so, how did you rebuild them?

I guess I have a few question to answer -

Firstly, I checked the sizelocks on SYSTEMP and LISTS on my system and they are fine (I did need to remake the LISTS table however). Reduce the number of saved queries. This seem to work help significantly. The problem is that this slow down is exhibited not only on my system by on multiple client systems. I can't see how they can all suffer a slow down if this was the problem. Unless I'm always executing a very large query somewhere else.

-

ARev version 3.12 on my and client systems.

OS - some DOS/Win 3.11, others Win95. I'm on NT.

Network - some using Windows, some Novell, don't know the specifics

No devoted indexing machine. Case insensitive queries. No update on queries.

Symbolics have been added to the dictionary - none of these are indexed nor are they accessed by the popup.

No previously indexed fields have been removed or added.

No data has been converted.

Data are VARCHAR types

I have rebuilt indexes on my system. I removed all the index and rebuilt them from scratch. This did not help.

-

The message appears with number of records returned in a few seconds. The problem is after this where the popup is being displayed.

All the fields are indexed.

I will try and build the popup manually.

I have about 3 other search which are performed in a similar manner, but none of these exhibit the slowdown that I have with this one.

I hope I've covered everything

Is there a difference in case sensitivity between your index and your environment?

All indexes are set as case-insensitive, as is the environment.

I think that I and others who have responded are a bit confused about what you are actually doing. Could you describe more precisely what it is that is occurring?

]It brings up the shared index relatively quickly

What do you mean by "shared index"?

]and the index values you want to select can be browsed and entered with good speed.

Again, what do you mean here? This seems to contradict what you say subsequently.

]Once the information is saved

What process is being used to save the information? What information?

]it will popup XXXX rows selected in a couple of seconds. The problem is that it will take from 20-30 seconds to display the resulting popup.

Is this a standard popup you are referring to, like from a B catalyst call perhaps? Did you write your own popup logic? ….

Perhaps if you could post some code or pseudocode….

OK to paraphrase, you Ctrl-F10, choose Indexed Selections, enter your index values, it goes off and gets al the matchs then says "found x" all in perfectly acceptable times.

Then it displays a popup showing you your hits and this popup displays very slowly.

So - the problem is not in your index, it is either in your popup definition or you construction of data row.

Do any other popups display slowly or just this one?

If you redefine the popup to include just one field does it stil display slowly?

If not which field introduces the slowness?

If after displaying them all, you select them all for your browse list then save your browse list as a Save list, you can try doing a getlist at TCL and just listing the records to see if this is also slow. If so the problem is NOTHING to do with the popup.

A few avenues to explore

Giles:

How many records are in the datafiles and how many hits are being returned?

Is there a difference when selecting on only one indexed field vs several fields?

Do typical searches involve entering several search values for for one index field?

There were/are conditions where queries will do a full resolve against the file (bypass the btree indexes). I don't recall what they are nor can I find my notes and as I've noted before my copies of the Tech Bulletins are no longer complete.

In the popup, how many display fields are there? Are any of these (and the index fields) MV'd? Are any symbolics?

Select POPUPs can be slow if there is a large number of records being returned with display fields as the process has to either fully resolve (read all the records from the datafile to get the display data) or build it dynmically on the fly (read only the records that are being displayed). Display fields that are symbolics can slow the process even further (particularly XLATE's).

Symbolics can be opitmized by replacing any braces '{}' references with direct references (@record vs {CUST_NAME}) and making sure symbolics don't call another symbolic (either clone the symbolic code of one into another or call a subroutine).

None of the indexed fields or display fields are symbolics. None of them are MV'd either. In my table there are about 100 records and I might return 1 to 10 on some tests. In client tables there might be a few hundred to a few thousand. It doesn't appear to make a difference how many index fields are used. The slowness doesn't appear in the search by the final display.

Hi Giles:

Sorry if I might seem repetitive in my questions or seem like I'm not really understanding the problem. Part of the difficulty is the intervals between messages an also because of the structure of this discussion area that I can't easily walk through the threads as I could have on the CompuServe forum with an offline reader like TapCIS.

Anyway, where exactly is the slowness in the process?

1) When the shared index collector window first opens from your catalyst call?

2) If and when a user presses F2 in any of the search fields?

3) After pressing F9 and before the 'x number of record(s) selected' message?

4) After the 'x number of record(s) selected' message and before the selection popup appears?

5) During the initial display of the selection popup?

6) When paging through the popup?

7) after tagging items in the popup and press F9?

Also what sort of keys are on the indexed files? Are they complex (e.g. abc*efg*etc) or very large keys (gt 8 characters)?

The slowness occurs at the stages you've noted as 4 & 5. It produces a message giving the number of rows quite rapidly, however, it will take some time to start displaying the popup (10+ seconds). Once it starts to display the popup, it continues to take some time until the full popup is displayed. For instance, It may sit for about 5 to 10 seconds with the header of the popup on the screen before the rest appears… and this is for maybe 4 rows returned.

Most keys would be less than 8 characters, but some will go up to 15.

The problem would seem to be an I/O bottleneck of some sort as the popup will read as many records as it can, parse out the display data, and buffer it as an I/O string so that paging through the popup will be rapid.

The I/O bottleneck could be any number of things: slow drives/network, files with a great deal of overflow frames (sizelocked files, large records, records with a key structure that hash poorly.

You may want to try running VERIFYLH from the Utilities - Developer - Advanced menu with the D and F options.

View this thread on the forum...