{{tag>category:"OpenInsight 32-bit" author:"Matthew Jones" author:"bob carten" author:"matthew jones" author:"Aaron Kaplan"}} [[https://www.revelation.com/the-works|Join The Works program to have access to the most current content, and to be able to ask questions and get answers from Revelation staff and the Revelation community]] ==== Continuing DBT Corruption Issues (OpenInsight 32-bit) ==== === At 24 MAY 2012 12:30:26AM Matthew Jones wrote: === OI9.3.1 Our customers experience infrequent corruption of their app.DBT files. When it happens, those already in OI are ok, but no-one else can log in because of the corrupt DBT. The fix is to restore a backup copy of the DBT afterwhich the system is ok again. At the recent Sydney roadshow it was suggested that we make the DBT read-only at the OS level. This turns out to be problematic for us. Our reporting platform creates temporary OI workfiles, publishes the report data, then deletes the temporary OI workfile. We have found that the OI DELETE_TABLE command causes the DBT file to be "touched" (the date modified is changed, but the DBT file content is unchanged). So, in making the DBT file read-only, we encounter FS103 errors every time we run a report. This is not acceptable. We also have a server app that failed to start on a client site this morning due to this infrequent DBT corruption issue. The server app would also be stopped dead by FS103 errors whenever it created a report if we made the DBT read-only - no good either. Our data tables are not saved in the DBT and we cannot save them as part of the DBT as our app uses mulitple identical data volumes. This has been a problem for some time now. Does anyone know of a solution or at least a good workaround. TAI. Matt Jones. ---- === At 24 MAY 2012 12:05PM bob carten wrote: === Hi Matthew, If you leave the dbt read only and call Set_Status(0) after the DELETE_TABLE does it prevent the FS103 error from stopping your reports? - Bob ---- === At 25 MAY 2012 12:05AM Matthew Jones wrote: === Hi Bob, Yes, the FS103 message does not get called if I put Set_Status(0) directly after the DELETE_TABLE statement. So this means it's not the DELETE_TABLE that's directly touching the DBT, but rather some other process after the DELETE_TABLE in response to the DELETE_TABLE's setting of the status!! So setting the status would certainly help by preventing the FS103 from stopping reports. However, this was not the original problem. The original problem is that the DBT is being intermittently corrupted by unknown cause. If we make the DBT read-only and implement the Set_Status as described above, that will help our lan users. For them, if they encounter a DBT corruption, or something else touching the DBT (hopefully not very often), then they will be able to just click through the message. I think we (they) can put up with that if it's infrequent enough. However, we are still faced with a larger problem for our server based product. This has multiple OI's running, spawning multiple OI's to perform tasks on a scheduled basis. If the DBT becomes corrupt at any time (or if any process throws a message), these server based routines fail big time. Any suggestions for that side of things on the server? Thanks and cheers, Matt Jones. ---- === At 25 MAY 2012 08:22AM bob carten wrote: === Delete_table calls define_Database, which is probably not thread safe. We'll look into that for the next release, but in the meantime, here are some options: - Instead of calling delete_table directly, write the name of the table to another table, have another centralized process which periodically selects from that file to perform the actual deletes - Create the temporary table in a one-off folder in the user's %temp% location, when done, detach the table and delete the whole folder. That way you never use delete table - don't delete the table, just clear it, re-use it. - If you like living dangerously, you could try making the table using the RTI_MEMBFS, an experiment I added to 9.31. It is perfect for making temporary reporting tables. [url=https://docs.google.com/document/d/1nH6dYhDVzMxkBnPIOQFW35z98srbPqY35tfWqRJaxXY/edit]test_rti_membfs[/url] shows how to use it. ---- === At 25 MAY 2012 10:31AM matthew jones wrote: === Thanks Bob. Please let me stress that this Delete_Table issue is only a bi-product of trying to overcome a separate problem - that of intermittent DBT file corruption from unknown causes. I will have a closer look at your suggestions on Monday, but some immediate thoughts: . Why should Delete_table call Define_Database at all; Create_Table obviously doesn't call Define_Database, otherwise the error message would've come at that time, so why should Delete_Table call it? . The automatic update of the DBT assumes the application wants all tables attached at all times. This is certainly not the case. We have multiple business data volumes, so these are not and cannot be included in the DBT . Our report work tables are kept in a separate volume, some only persist for the generation of the single report, some longer, and we certainly don't want those included in the DBT either . Wouldn't it be preferable to call Define_Database completely under programmatic or manual control, and have no OI process call it automatically at all (apart from the menu option to save the database definition in the Database Manager)? Unfortunately, as I said, none of this solves the original problem - intermittent corrupt DBT that stops anyone being able to log into OI, and stops our server processes working. I still have no idea what causes this corruption or quite how to stop it. We had another occurence of it the other night on our largest client site - our server product that runs 20 or so recurring reports each night (and sends the output to mailboxes and report library files) couldn't start after backup processes due to a corrupt DBT, so the reports didn't run. This is a real limiting issue for us in terms of our product's credibility as a serious contender against other products with background processing, multi-threading etc (which we simulate with our multi-engine task processor, except when it fails like this). I very much appreciate your suggestions and the exchange of ideas (and will consider further Monday). Hoping I can resolve the corruption issue as well. Any further thoughts you might have are most welcome. :) Cheers Bob, Matt Jones. ---- === At 25 MAY 2012 10:38AM bob carten wrote: === Hi Matt - I agree. I have given you workarounds. The real solution is to prevent the corruption from occurring. DELETE_TABLE should look in the dbt, only modify it if the table is there. Define database should lock a placeholder before trying to change the dbt. What I [i]really[/i] want to do is put the dbt records in sysenv or some other non detachable table, set OI to use an LH read there before it drops to OSREAD from the OI folder. I'll see if I can make you something to test. - Bob ---- === At 25 MAY 2012 10:52AM matthew jones wrote: === Hmmm, it sounds like you're confident that the corruption is occuring as a result of the Define_Database, triggered from the Delete_Table. If you think that's the case, I feel a bit more comfortable, because I wasn't specifically linking the 2 "issues" together (didn't know if there was definitely cause and effect, or just coincidence). Ah, and as we can have a number of reports being generated at the same time on a server with our task processor in particular, now I think I get your comment about it not being thread-safe. Hmmm, perhaps the light is coming on for me. Unfortunate really, as it's time I turned the light off and gave up for the night. :) Thanks again Bob. I look forward to trying something. Cheers, Matt Jones. ---- === At 25 MAY 2012 12:30PM Aaron Kaplan wrote: === Hi Matt - I agree. I have given you workarounds. The real solution is to prevent the corruption from occurring. DELETE_TABLE should look in the dbt, only modify it if the table is there. Define database should lock a placeholder before trying to change the dbt. What I [i]really[/i] want to do is put the dbt records in sysenv or some other non detachable table, set OI to use an LH read there before it drops to OSREAD from the OI folder. I'll see if I can make you something to test. - Bob An image of the system tables stored in SYSENV seems like a nice idea. You can store something we can call an attach image, that you can set though a SET_ATTACH_IMAGE function, and then use it through maybe a USE_ATTACH_IMAGE function. :biggrin: Sorry, that wasn't really nice, was it? Aaron Kaplan, the Patrick Henry of the DOS world. [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] [url=http://www.sprezzatura.com/blog]The Sprezzatura Blog[/url] [i]World leaders in all things RevSoft[/i] [img]http://www.sprezzatura.com/zz.gif[/img] ---- === At 25 MAY 2012 12:34PM Aaron Kaplan wrote: === Thanks Bob. Please let me stress that this Delete_Table issue is only a bi-product of trying to overcome a separate problem - that of intermittent DBT file corruption from unknown causes. I/quote] The real question is, how is it corrupt? The DBT is just an @RM delimited dump of @SYSTABLES and @SYSVOLUMES and isn't really that complicated to manipulate. My guess here is that is there is some corruption, then something is happening to this string as it's written out to disk, probably something dropping into the middle of the string, dropping the delimiters out. Have you compared the bad with the good? [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] [url=http://www.sprezzatura.com/blog]The Sprezzatura Blog[/url] [i]World leaders in all things RevSoft[/i] [img]http://www.sprezzatura.com/zz.gif[/img] [[https://www.revelation.com/revweb/oecgi4p.php/O4W_HANDOFF?DESTN=O4W_RUN_FORM&INQID=WORKS_READ&SUMMARY=1&KEY=BC36BD2E00041F956B25E7300|View this thread on the Works forum...]]