The Saga of ShellExecute

One of the most popular "raw" Windows API functions that OpenInsight developers have used over the years is the ShellExecute function, which allows you to launch an application via its filename association, e.g. you can launch Word by using a document file name, or Excel using a spreadsheet filename and so on.

However, because it was never really made an "official" part of the product (it was normally passed on in forums), developers were left to create their own DLL Prototype definitions in order to use it - this gave rise to many variations over the years, many of which were not compatible with others. For example, some use LPCHAR as an argument type, some use LPSTR or LPASTR, whilst others use LPVOID with GetPointer(); some definitions use the "Wide" version of the function, some the "Ansi" version, and there are many different aliases, with or without the "A/W" suffix too. The list goes on.

For OpenInsight 10 we decided that we couldn't move forward with this as we would run the risk of conflicting with established applications, so we moved all of the DLL Prototypes we used into a new namespace called "MSWIN_" and claimed it as our own. This left developers to bring forward their own DLL prototypes into version 10 as and when needed, and therefore we didn't supply a "ShellExecute" function as such, though we did supply "MsWin_ShellExecute" instead (see below).

Another decision we took was to try and move away from the need for developers to use raw Windows API function calls as much as possible, as some of them can be complex and require knowledge of C/C++ programming, which is not necessarily a skill set that everyone has the time or desire to learn. Ergo, we moved a lot of functionality into the Presentation Server (PS) and created some Basic+ wrapper functions around others to shield developers from the sometimes gory internals.

(We also chose to use the "W" versions of functions rather than the "A" versions where possible, because these would translate better when in UTF8 mode and remove the need for an extra "A"→"W" conversion in Windows itself.)

So, coming back to ShellExecute, and in light of the above, we have three "official" and supported ways of calling it in OpenInsight 10 as detailed below:

• The SYSTEM object SHELLEXEC method
• The RTI_ShellExecuteEx stored procedure
• The MSWin_ShellExecute DLL Prototype stored procedure

If your program is running in "Event Context", (i.e. it is executing in response to an event originating from the PS) then you may use the SYSTEM SHELLEXEC method which invokes ShellExecuteW internally.

RetVal = Exec_Method( "SYSTEM", "SHELLEXEC", OwnerForm, Operation, File, |
                      Parameters, WorkingDir, ShowCmd )

The return value is the value returned by ShellExecuteW.

This stored procedure is a wrapper around the Windows API ShellExecuteExW function (which is used internally by ShellExecuteW itself), and may be used outside of event context - it can also return the handle to any new process it starts as a result of executing the document. As you can see it's quite similar to the SHELLEXEC method:

RetVal = RTI_ShellExecuteEx( Hwnd, Verb, File, Parameters, |
                             Directory, nShow, hProcess )

The return value is the value returned by ShellExecuteExW.

This is the "raw" DLL function that is included with OI10, and the definition can be found in the MSWIN_SHELL32 DLLPROTOTYPE entity:

Because we're using LPWSTR data types there is no need to null-terminate any of your variables so using it is quite simple:

RetVal = MsWin_ShellExecute( 0, "open", "stuff.docx", "", "c:\docs", 1 )

Whilst you are free to use one of the methods outlined above, this may not be optimal if you are still sharing code between your existing version 9 application and your new version 10 one. In this case there are a couple of options you could use:

• Define your preferred DLL prototype in v10.
• Use a wrapper procedure and conditional compilation.

This is probably the easiest option - you simply use the same prototype in v10 that you did in version 9, with the same alias (if any), and this way the code that uses it doesn't need to be changed. The only downside to this if you've used any 32-bit specific data types instead of 32/64-bit safe types like HANDLE (this could happen if you have a really old prototype) - you must ensure that you use types that are 64-bit compliant.

This is a technique we used when writing the initial parts of v10 in a v9 system so our stored procedures would run the correct code depending on the platform they were executing on (it was actually first used to share code between ARev and OI many years ago!).

The v10 Basic+ compiler defines a token called "REVENG64" which is not present in the v9 compiler - this means that you can check for this in your source code with "#ifdef/#ifndef" directives and write code for the different compiler versions.

For example, you could write your own wrapper procedure for ShellExecute that looks something like this:

Compile Function My_ShellExecute( hwnd, verb, file, params, dir, nShow )
 
#ifdef REVENG64
   // V10 Compiler - use RTI function
   Declare Function RTI_ShellExecuteEx
   RetVal = RTI_ShellExecuteEx( hwnd, verb, file, params, dir, nShow, "" )
#endif
 
#ifndef REVENG64
   // V9 Compiler - use existing "raw" prototype
   Declare Function ShellExecute
   RetVal = ShellExecute( hwnd, verb, file, params, dir, nShow )
#endif
 
Return RetVal

And then call My_ShellExecute from your own code.

So, there ends the Saga of ShellExecute … at least for now.