"suggested size with the LH/NLM and LH/NT Service is 32768; suggested size with the LH/NPP on a LAN is in the range of 1024-4096"

This is from the code of REMAKE_TABLE as posted by Cameron. Well now, this is news to me - I mean the size with the NLM/NT service. Can anyone at RevSoft comment on the performance implications of this statement?

Hi Nick,

None of this is gospel yet … just some information from several different customers that our back-end products are measurably faster as the frame size increases. The concept is that NT and Novell work with chunks of disk space typically 32k (32768 bytes) at a time so it is most efficient to work with the file in the same size blocks (

so one low-level read in the NLM is one lower-level read in the NOS).

The reason you would not necessarily want bigger frames with the NPP "local mode" (no back end service) is that all operations are on frames so an entire frame has to be transported to do any LH operation, even if only a much smaller amount of data is required. (With the local mode, the average record size seems to be more important when determining optimal frame size, but that is mostly theoretical.)

Until recently, frame sizes did not matter that much. (Note: Some developers would strongly disagree with this statement.) After some serious testing, we found that the default (1k) was the quickest. That was without NLM or NT Service though.

Cameron Purdy

Revelation Software

Hi Cameron

Thanks for the response.

I'll do some testing on this and post the results. We NEVER install on networks without the NLM/NT Service.

Regards,

Nick

This is the story as I understand it. The following was taking from a TB/KB article I started writing before I left Revelation:

The Effects of Frame Size on Memory Allocation and File Access Speed

A brief description of Linear Hash Files

Advanced Revelation's primary filing system stored data in a hashed structure. These files are called Linear Hash files. Hashed filing systems operate by dividing the file into groups. Each group in the filing system is further broken down into frames. There are two types of frames, primary and overflow. The first frame in the group is the primary frame. Taken together, all of the primary frames are called the primary space. All other frames in the group are overflow frames. Each group in the filing system is defined to exactly the same size. This size can vary from 256 bytes to 10,000 bytes.

Each record is then identified through the use of a primary key. Through a mathematical algorithm known as hashing, the filing system determines which group this record belongs in. Whenever the record is read or written to disk, the filing system only needs to access this one group. The system first tries to allocate the record into the primary frame. If the primary frame is full, the system places the record in the first available overflow frame. More than one record can be in a single frame. A record can also span more than one frame.

Each Linear Hash file has a threshold point. The threshold point is a percentage of the total amount of primary space available to the file. When the amount of primary space exceeds the threshold, the system allocates more groups to the file. When the amount of primary space is 10% below the threshold amount, the system decreases the number of groups available.

Advanced Revelation allows you to modify the threshold value and the frame size. For the purposes of this article, we will only be discussing the effects of frame size. For more information on the effects of changing the threshold setting, please refer to Technical Bulletin # 29 (The Linear Hash Filing System).

The default frame size for Linear Hash files is 1024 bytes. The only time you can modify a file's frame size to at creation time. (The REMAKEFILE command creates a new file and copies each record into the new file. Direct manipulation would be difficult due to hashing considerations.) Optimally, you would allocate frame size so that only one record hashes to a single group and the record completely resides in the primary frame. (This is called a perfectly hashed file.). It might also be desired to match frame size to network packet size to improve transfers over the wire. It would not be wise to go out and remake all of you files just yet. Frame size has an interesting effect on memory and file access.

Frame Buffers and Cache

Advanced Revelation maintains six internal frame buffers. Each buffer is allocated to the largest frame size for the current session. By using the default 1K frame size, the system allocates 6K of memory to the frames buffer. By increasing a single file's frame size to 4K, the total size of the frame buffer increases to 24K (six 4K buffers).

During any read request, the system fills up the frame buffers. If reading from primary frames, the buffers contain only primary frames. If reading from overflow, the buffers only contain overflow frames. When all the frame sizes in a system are identical, each frame buffer contains exactly one frame. When you have one frame that is larger than the rest, the system begins to run into what I call frame buffer overflow problems.

LH will attempt to fill up the buffers, especially on selects and readnexts. Generally, this does not matter much when dealing with primary frames, but makes a big difference with overflow frames. LH will completely fill up the buffers needed. If the buffer is 4K, the system will load up 4K into the buffer. If you have a 1K frame, this means that 4 frames will be in each buffer.

When the system checks for records, it only looks at the frame size, it does not take the buffer size into account. When it cannot find the record in the first buffer, it jumps to the next buffer. The next buffer does not contain the next frame however. It contains the 5th frame. LH's error checking comes in, finds what could be a GFE, so it re-reads in the frames to validate the error. This time, it doesn't find one, so processing continues. This error/re-read loop continues until the record is found.

end original article!

This frame size buffer thing is essentially a bug and has been in just about all versions of LH. This is why support and others always maintained keeping your frame buffers at 1K and modifying threshold values to help balance out the file. If you need to modify frame sizes, you should do it on all files.

With the new network drivers, this has changed. The buffer bug was fixed and the reads take into account the changing sizes, so it's acceptable to change framesizes on any file to suit the application's need.

It's important to note that with the NT Service and NLM, you won't see much of a difference in many actions. A good reason for modifying frame size was to eliminate large numbers of frames being sent across the network, and to try and match the frame size to the packet size to eliminate extra network requests. With the NT Service and NLM, you're only sending the entire record back. The Services still have to read in the entire frame to process the record.

Your big advantage will be in individual record reads. By modifying frame size you can come closer to setting up fewer records in each group. This makes each record read faster, since there are less records to check once it determines the correct group.

As a side note, portains of the network cache buffer are related to the frame size issue. By linking this in with the NT Service and the NLM, I have it on good authority that you can turn this back on, and replace the 020 back in REVBOOT.

akaplan@sprezzatura.com

Sprezzatura, Inc.

View this thread on the forum...