Trying to build (thru Rebuild or Update) a BTree index for a (real) column in a table with some 330,000 rows, the process "hungs" while in the "sorting" pass, apparently when it's just about to finish that pass: it says "329800/329826 records…", or 99% done. It worked fine, however, before adding the last 18,000 rows to the table.

The column to index is multivalued, so the total number of index values to process should be around 1,500,000.

My PC is a Pentium III, 800MHz with 128 MB RAM, and I'm using Windows 98 SE.

There is no error message, and the only way for me to get out is Ctrl-Alt-Del, which then reports that the "Building …" window does not respond, and I have to force its termination. After that, OpenEngine remains and I have to force it termination too.

Is this something known to anyone?

Thank you in advance.

Possibly… possibly…

I believe that ths "sort" process "gas gague" is actually a gague for the "build list" part of the "sort" process. After the build, there is the actual sort, merge, and extract processes. Maybe those other three processes are actually working and taking a long time. Especially since it is a multivalued field being sorted.

I have noticed a delay when I build indexes with large quantities of items. I have built an index with 500,000 items. Although it was not multivalued.

So, is the HDD light on or flashing during this "hang"? And how long is the actual "hang" before you do the reset? And how long does the process take before the "hang"?

The HDD light is steadily on (not blinking at all).

I've waited for about two hours and a half.

The process before the "hang" takes about twenty minutes.

Before the last increase in 18,000 rows, the same index took less than an hour to complete. In fact, to complete two similar indexes on the same file.

Let me add that I have also tried the Update_Index command from the System Editor, with the same result (except for the process window).

Eduardo,

Your message caught my eyes as we are hoping to convert an AREV app to OI-32 soon. We're expecting to rebuild indexes in our conversion and are hoping for no significant problems. Some of the files may have 500,000 to over a million items similar to your enviornment. I know that AREV can handle the bebuilding of indexes in this environment as we had to do so recently . Therefore, I'm hoping that what you are experiencing is not a reflection of any weakness (or bug) in OI-32 when handling indexes. Maybe RTI will comment on this.

Meanwhile, let me ask this questions, it may not help, but can you try rebuilding the indexes using a Windows 2000 workstation instead? I see that you're currently using Win98.

Ray Chan

Ray,

provided no alternative, I would try using Windows 2000 Workstation or Windows NT Workstation.

In fact, I once experienced a problem due to the use of Windows 98 SE, precisely when coding the import program now in use to generate the offending table: I found a "buffer limit exceeded" or something like it, when the program was about 45K, and in this same forum I was told that the problem would probably disappear if I turned to some NT-like version of Windows. But I don't know if my index problem is due to some comparable reason.

Eduardo,

I doubt that your index problems will be solved by a different OS. Likely there is a corruption in the index table or a problem with the data itself. Questions:

1. Have you been able to update/rebuild this index in other versions of OI?

2. How about AREV?

3. Have you tried to manually remove the index and build it from scratch?

4. Have you examined your datafile for GFE's, delimiters in keys, or other anomalies?

dbakke@srpcs.com

SRP Computer Solutions, Inc.

Don,

1. No. I have upgraded to the 32-bit version. I'm afraid I'd have to re-install the original OI and its upgrades until the last 16-bit one. But the tables have been already updated under the 32-bit version. Nevertheless, to prevent conflicts between versions, it seems I'd have to install the complete system on another PC, isn't it?

By the way, under this same 32-bit OI, I indexed the same table, though without the last 18,000 rows added.

2. The tables having been enlarged by the 32-bit OI, I'd use AREV to access them just for reading - which, in fact, I plan to do. But I'd be afraid of writing anything via AREV or 16-bit OI. Am I wrong?

3. Yes, I tried, via the Remove option in Database, Utilities, Indexes. As it failed, I removed all the table indexes again the same way and then deleted the !tablename table from the System Editor (RUN DELETE_TABLE …'. But then the table became unavailable and I restored it from an internal backup. Now I wonder that I'd better have removed the !tablename thru the Database, File, Add menu option,

but this would not delete the !tablename table.

4. I have examined all the datafiles for GFE's (I do it after each important processing, and before backing up previous to any index rebuild). No GFE's, even in the bang file.

As for delimiters in keys, etc, I will examine the file (actually, according to the way the importing program works, the keys should be just sequential numbers, but …)

Eduardo,

1. No. I have upgraded to the 32-bit version. I'm afraid I'd have to re-install the original OI and its upgrades until the last 16-bit one. But the tables have been already updated under the 32-bit version. Nevertheless, to prevent conflicts between versions, it seems I'd have to install the complete system on another PC, isn't it? By the way, under this same 32-bit OI, I indexed the same table, though without the last 18,000 rows added.

Assuming this was an upgraded app from OI16 I thought you might have a backup hanging around somewhere. You wouldn't have to install this on a different PC, just make sure you copy the tables.

2. The tables having been enlarged by the 32-bit OI, I'd use AREV to access them just for reading - which, in fact, I plan to do. But I'd be afraid of writing anything via AREV or 16-bit OI. Am I wrong?

Enlarged in terms of number of rows in the table or are you creating rows that are ]64K in size? If the former is true then there is nothing to worry about. The linear hash structure is the same for the data, dictionary, and index tables.

3. Yes, I tried, via the Remove option in Database, Utilities, Indexes. As it failed, I removed all the table indexes again the same way and then deleted the !tablename table from the System Editor (RUN DELETE_TABLE …'. But then the table became unavailable and I restored it from an internal backup. Now I wonder that I'd better have removed the !tablename thru the Database, File, Add menu option, but this would not delete the !tablename table.

Did you also clear the %FIELDS% and %PROTECT.SPEC% from the dictionary. Did you remove the flag from each dictionary record that had an index assigned to it. Finally, did you remove the SI.MFS entry from the REVMEDIA table entry for this table? This article gives you step-by-step details on how to do a true manual removal and build of indexes.

4. I have examined all the datafiles for GFE's (I do it after each important processing, and before backing up previous to any index rebuild). No GFE's, even in the bang file. As for delimiters in keys, etc, I will examine the file (actually, according to the way the importing program works, the keys should be just sequential numbers, but …)

I don't know if this would affect the ability for OI to rebuild indexes, but it is always a good idea to make sure your dictionary lengths for keys and indexed fields are sufficiently long enough.

dbakke@srpcs.com

SRP Computer Solutions, Inc.

Don,

I'll try first to correctly remove indexes manually following the article you so gently point to. I will do this tonight, and then let you know the results.

Thank you.

Eduardo

Don,

I have read thru the file making sure that all keys are pure non-zero integers. I deleted the bang file carefully following the Knowledge Base article you suggested. After that, just in case, I examined the file again to ensure the absence of any GFE's. Dictionary lengths are more than long enough.

Then I added the index and ran the Update. Unfortunately, the result has been the same (I waited almost an hour after all disk activity ended though the HDD was on and no blinking).

But this time I managed to get a message, presumably from Windows, indicating "Insufficient System Resources".

Regrettably, this seem to accuse my Windows 98 SE.

On the other side, I'd prefer not to have to rely again on OI16 or AREV.

What can I do now?

Eduardo

PS: Provided OI32 works OK under, say, Windows 2000, does the upgrade to that Windows version force to clear the disk and then re-install everything from scratch?

Eduardo,

If you have "Insufficient System Resources" you might try to

- Boot the computer in Safe Mode

- Using the Windows Task Manager (press Ctl-Alt-Del), examine the applications and processes that are running and choose 'End Task' to end any unnecessary running items

- Check the amount of available hard disk space to see if there is sufficient space for the swap file

- Remove any applications from the Startup folder.

(I scavenged these instructions from some web page about problems with "Insufficient System Resources" in some other program, but they should be equally appliccable to OI apps.)

- Oystein -

Oysten,

Thank you, I'll try that. But I'm afraid that at best it would postpone the problem, because the file will continue to increase (when the import finishes, it will have some 440,000 records and we expect that once it is functioning normally, 3,000 records per month will be added).

I could also increase the PC RAM to 256MB, but I'm afraid that the physical increment doesn't become an available resource for OI thru Windows, if the problem lies in a Windows limitation.

By the way, disk space doesn't seem a concern in my case: I have more than 3.5 GB free in the same drive.

Contrary to my previous intention, in case the try you've suggested fails, I'm also considering going back to OI16 and try with it. I wonder what would happen if I renamed the C:\revsoft directory and installed the original OI (16 bits) in that same drive, having previously backed up my application via the AppBackup executable form.

I'm reasonably certain that all records in my files are less than 64K in length, and it will continue to be true in the future.

Eduardo

Eduardo,

(This is a single user system on a local computer, right? No network stuff?)

…postpone the problem…

Yes and no. My suggestion was to free some resources (like running in Safe Mode, removing programs at startup, etc) before running the indexing again. If the indexing now runs successfully you might try to increase the load again (add startup programs, etc) until it fails. Perhaps then you can pinpoint the cause of the problem.

But of course such experiments can be very time-consuming, with the large amounts of data you have got.

- Oystein -

Yes, this is a single user system on a local computer. No network stuff.

I will try to choose carefully my lines of experimentation because, as you have pointed out, they are very time-consuming. Probably they will take several days.

Wishing to let you (who have helped me about this topic) know the results at which I may eventually arrive, should I add to this chain or initiate a new topic? Provided, of course, you are interested in them, and taking into account that several days may then have elapsed.

Eduardo

Eduardo,

I read everything in the OpenInsight categories so you can report however you like.

- Oystein -

Eduardo,

I like threads to stay together. I use the Flat by Date view so new posts to old threads don't get skipped.

dbakke@srpcs.com

SRP Computer Solutions, Inc.

Don,

Talking about keeping threads together - have you noticed how some postings don't show in their thread? It used to happen when people went back and edited their postings. That somehow caused later replies not to show in their thread. I thought Revelation (Jennifer) fixed it a couple of years ago, but it still happens.

- Oystein -

Oystein,

Yes, I have noticed that. It doesn't happen very often but occasionally I've seen orphaned posts. Weird.

dbakke@srpcs.com

SRP Computer Solutions, Inc.

The problem was finally solved when I created a new folder and copied (via copy_table) the files of the database to it. Prior to that, I'd got involved in an almost chaotic situation for not having reviewed the article about the database tables (.DBT). My fault.

Eduardo Murphy

PS: In fact, the problem was solved about two weeks ago. Please excuse me for posting this so late. And thank you all.

View this thread on the Works forum...