Hello,

Im importing large CSV files into AREV 3.12 using OSBREAD and it works for all but the biggest (3.2gig) where the OSBREAD loop stops before its hits EOF.

DOS prompt reports the file size as 3282283854 AREV DIR() returns 1012683442 and OSBREAD only reads upto 2147483648 then gives up.

Ive heard about an external routine called IMPORT.READ will this help me or have I hit upon a limit of AREV/OSBREAD?

Thanks in advance.

You've hit on a 2 Gig limit. The maximum value for a signed 32 bit intiger is 2147483647. To understand the relationship between the two numbers (the 3Gig and 1Gig numbers), open up your calculator and put it in scientific mode. Enter the 2 gig number and convert it to binary. Now swap the 1s with 0s and vice versa (because it looks like a negative number). Add 1 and you have the binary version of that DIR() reports.

If you search the forum, there are some posts about the 2 gig limits and more about 4 gig limits.

DOS can only enable a file pointer to proceed 2^31 bytes into a file if you are shifting the pointer relative to an offset. That's around the 2.1GB mark.

Good explanation guys, thanks, guess I should now break the big files up into segments of less than 2 gig in size.

Where are you putting the data? Depending on what driver you're using, you may be transferring the problem from your OSBREAD statement to your WRITE statement. Maybe someone else can share what the size limits (for the LK and OV files) are for the various drivers. You can get around the file size limitation by using and MFS to divide the data among various physical files. Arev, above the MFS level, sees them as just a single file.

Funny enough that was going to be my next question ie:

Is there also a 2.1 gig limit on LK/OV files?

We are writing the data from this CSV file into one AREV table.

We use OI as well but this could not read the 3.2 gig CSV file properly either, should it be able to?

The LK/OV limit is 4GB. Theoretically 32 Bit OI (4.1 and above) should be able to OSBread anything but this would have to be confirmed by Revelation.

The Sprezzatura Group

World leaders in all things RevSoft

We also use OI 7.2.1 in which the DIR() function reports the size of this 3.2 gig file to be -1,012,683,442 (yes negative)and its OSBREAD reads upto 10,000,007,168 then ends. ie seems to read past the EOF.

Does that 4 gig limit apply to the size of the OV and LK files separately or the combined size?

Whoops - looks like that needs revisiting then.

4GB per LK and OV.

The Sprezzatura Group

World leaders in all things RevSoft

Well I split the 3.2 gig CSV file into quarters and read the data into a new AREV table and got a LK size of 160 meg and a OV size of 2.8 gig! I assume the size of an OV (or LK) file can go up to 4 gig but as it is this file is very overflown. Guess I should have used custom CREATETABLE attributes but we dont usually and Ive never really understood what these attributes should be anyway.

Could anyone tell me how to calculate the best LH attributes?

BTW is it expected that both DUMP and LHVERIFY both report a corrupt header on this large REV table? Can still use the table.

The default frame size is 1024 bytes. This includes overhead used by the filing system. It is possible to size a file with larger frames, but years ago, Sprez. (I think) revealed that doing so sets up a situation where a buffer is initialized to accomodate the largest frame size. Other files with smaller frames will sort of double-clutch, reading ahead more than they have to.

I don't know if this is relevant for modern drivers, though. And anyway, your reason to change frame size is something other than performance.

Judging by your LK/OV size ratio, I would guess that your record sizes are rather large. If they were somewhat smaller than 1K, then I'd expect a more even allocation between the LK and OV files.

The parameters you mention don't really amount to much. What's really important is the frame size. The parameters are number of records, average size, dictionary records, frame size, and Resize threshold. The first three fields will be used to determine how big to initially build the file. The size of the pieces it uses to build the file is determined by frame size. For an evenly distributed (between LK and OV) file, the frame size should be larger than the average record size. If you change frame size, you probably want to make sure that you use multiples of 1024 bytes so frame boundaries line up with cluster boundaries.

Resize threshold specifies how much of the alpha must fill up before the file is resized larger. The alpha is essentially the available space in the LK file, which is determined only by file modulo and frame size.

If I were in your position, I'd be looking into ways to split the file into smaller files, perhaps using an MFS, or by other means.

Victor

Guess we will truck on… Will be a bit of a concern if the size limit on LK and OV files is 2.1 gig as it seems to be given the fact that DUMPLH nor VERIFYLH works on LH files over this size.

Turns out we were getting this Novell 4.2 gig limit because we used the native filing system rather than NSS (what ever that means) but now we can create OV/LK files much larger than 4.2 gig - given enough time! Took a few weeks to find this out.

View this thread on the forum...