Chapter 4 - Resource Managers

Resource Managers

Introduction

In order to give QNX/Neutrino a great degree of flexibility, to minimize the runtime memory requirements of the final system, and to cope with the wide variety of devices that may be found in a custom embedded system, QNX allows user-written processes to act as resource managers that can be started and stopped dynamically.

Neutrino resource managers are responsible for presenting an interface to various types of devices. This may involve managing actual hardware devices (like serial ports, parallel ports, network cards, and disk drives) or virtual devices (like /dev/null, a network filesystem, and pseudo-ttys).

In other operating systems, this functionality is traditionally associated with device drivers. But unlike device drivers, Neutrino's resource managers don't require any special arrangements with the kernel. In fact, a resource manager looks just like any other user-level program.

What is a resource manager?

Because QNX/Neutrino is a distributed, microkernel OS with virtually all non-kernel functionality provided by user-installable programs, a clean and well-defined interface is required between client programs and resource managers. All resource manager functions are documented; there's no magic or private interface between the kernel and a resource manager.

In fact, a resource manager is basically a user-level server program that accepts messages from other programs and, optionally, communicates with hardware. Again, the power and flexibility of the native QNX/Neutrino IPC services allow the resource manager to be decoupled from the OS.

The binding between the resource manager and the client programs that use the associated resource is done through a flexible mechanism called pathname space mapping.

In pathname space mapping, an association is made between a pathname and a resource manager. The resource manager sets up this pathname space mapping by informing the QNX/Neutrino Process Manager that it is the one responsible for handling requests at (or below, in the case of filesystems), a certain mountpoint. This allows the Process Manager to associate services (i.e. functions provided by resource managers) with pathnames.

For example, a serial port may be managed by a resource manager called Devc.ser, but the actual resource may be called /dev/ser1 in the pathname space. Therefore, when a program requests serial port services, it typically does so by opening a serial port - in this case /dev/ser1.

Client side

Once the resource manager has established its pathname prefix, it will receive messages whenever any client program tries to do an open(), read(), write(), etc. on that pathname. For example, after Devc.ser has taken over the pathname /dev/ser1, and a client program executes:

fd = open ("dev/ser1", O_RDONLY);

the client's C library will construct an io_open message, which it then sends to the Devc.ser resource manager via QNX/Neutrino IPC.

Some time later, when the client program executes:

read (fd, buf, BUFSIZ);

the client's C library constructs an io_read message, which is then sent to the resource manager.

A key point is that all communications between the client program and the resource manager are done through native QNX/Neutrino IPC messaging. This allows for a number of unique features:

  • A well-defined interface to application programs. In a development environment, this allows a very clean division of labor for the implementation of the client side and the resource manager side.
  • A simple interface to the resource manager. Because all interactions with the resource manager go through QNX/Neutrino IPC, and there are no special "back door" hooks or arrangements with the OS, the writer of a resource manager can focus on the task at hand, rather than worry about all the special considerations needed in other operating systems.
  • Free network transparency. Because the underlying QNX/Neutrino IPC messaging mechanism is inherently network-distributed without any additional effort required by the client or server (resource manager), programs can seamlessly access resources on other nodes in the network without even being aware that they're going over a network.

All QNX/Neutrino device drivers and filesystems are implemented as resource managers. This means that everything that a "native" Neutrino device driver or filesystem can do, a user-written resource manager can do as well.

Consider FTP filesystems, for instance. Here a resource manager would take over a portion of the pathname space (e.g. /ftp) and allow users to cd into ftp sites to get files. For example, cd /ftp/rtfm.mit.edu/pub, would connect to the ftp site rtfm.mit.edu and change directory to /pub. After that point, the user could open, edit, or copy files.

Application-specific filesystems would be another example of a user-written resource manager. Given an application that makes extensive use of disk-based files, a custom tailored filesystem can be written that works with that application and delivers superior performance.

For example, for USENET news, instead of storing hundreds of thousands of small files under /usr/spool/news, a custom resource manager can be written that maps those files into expiry-ordered heap files, allowing expire operations to occur in seconds, not hours, and saving disk space. (For details, see the article "Improving USENET News Performance," in Doctor Dobb's Journal, May 1996.)

The possibilities for custom resource managers are limited only by the application developer's imagination.

As experience with resource managers has grown over the past 10 years, the interface and functions of resource managers have been continuously refined and improved.

Resource manager architecture

Here is the heart of a resource manager:

create a channel to receive messages
register the pathname with the Process Manager
DO forever
    receive a message
    SWITCH on the type of message
        CASE io_open:
            perform io_open processing
            ENDCASE
        CASE io_read:
            perform io_read processing
            ENDCASE
        CASE io_write:
            perform io_write processing
            ENDCASE
        .   // etc. handle all other messages
        .   // that may occur, performing
        .   // processing as appropriate
    ENDSWITCH
ENDDO

The architecture contains three parts:

  1. A channel has to be created, so that client programs can connect to the resource manager to send it messages.
  2. The pathname (or pathnames) that the resource manager is going to be responsible for is registered with the Process Manager, so that it can resolve open requests for that particular pathname to this resource manager.
  3. Messages are received and processed.

This message-processing structure (the switch/case, above) is required for each and every resource manager. However, we provide a set of convenient library functions to handle this functionality (and other key functionality as well).

Message types

Architecturally, there are two categories of messages that a resource manager will receive:

  • connect messages
  • I/O messages.

A connect message is issued by the client to perform an operation based on a pathname (e.g. an io_open message). This may involve performing operations such as permission checks (does the client have the correct permission to open this device?) and setting up a context for that request.

An I/O message is one that relies upon this context (created between the client and the resource manager) to perform subsequent processing of I/O messages (e.g. io_read).

There are good reasons for this design. It would be inefficient to pass the full pathname for each and every read() request, for example. The io_open handler can also perform tasks that we want done only once (e.g. permission checks), rather than with each I/O message. Also, when the read() has read 4096 bytes from a disk file, there may be another 20 megabytes still waiting to be read. Therefore, the read() function would need to have some context information telling it the position within the file it's reading from.

The resource manager shared library

In a custom embedded system, part of the design effort may be spent writing a resource manager (i.e. there may not be an off-the-shelf driver available for the custom hardware component in the system).

QNX/Neutrino's resource manager shared library makes this task much simpler.

Automatic default message handling

If there are functions that the resource manager doesn't want to handle for some reason (e.g. a digital-to-analog converter doesn't support a function such as lseek(), or the software doesn't require it), the shared library will conveniently supply default actions.

There are two levels of default actions:

  • The first level simply returns ENOSYS to the client application, informing it that that particular function is not supported.
  • The second level (i.e. the iofunc_*() shared library) allows a resource manager to automatically handle various functions.

For more information on default actions, see the section on "Second level default message handling" in this chapter.

open(), dup(), and close()

Another convenient service that the resource manager shared library provides is the automatic handling of dup() messages.

Suppose that the client program executed code that eventually ended up performing:

fd = open ("dev/device", O_RDONLY);
...
fd2 = dup (fd);
...
fd3 = dup (fd);
...
close (fd3);
...
close (fd2);
...
close (fd);

The client would generate an io_open message for the first open(), and then two io_dup messages for the two dup() calls. Then, when the client executed the close() calls, three io_close messages would be generated.

Since the dup() functions generate duplicates of the file descriptors, new context information should not be allocated for each one. When the io_close messages arrive, because no new context has been allocated for each dup(), no release of the memory by each io_close message should occur either! (If it did, the first close would wipe out the context.)

The resource manager shared library provides default handlers that keep track of the open(), dup(), and close() messages and perform work only for the last close (i.e. in the example above, the third io_close message).

Multiple thread handling

One of the salient features of Neutrino is the ability to use threads. By using multiple threads, a resource manager can be structured so that several threads are waiting for messages and then simultaneously handling them.

This thread management is another convenient function provided by the resource manager shared library. Besides keeping track of both the number of threads created and the number of threads waiting, the library also takes care of maintaining the optimal number of threads.

Combine messages

In order to conserve network bandwidth and to provide support for atomic operations, QNX/Neutrino supports combine messages. A combine message is constructed by the client's C library and consists of a number of I/O and/or connect messages packaged together into one.

To support atomic operations, a number of messages must be combined into one bigger message so that the resource manager receives the entire message all in one piece.

For example, the function readblock() allows a thread to atomically perform an lseek() and read() operation. This is done in the client library by combining the io_lseek and io_read messages into one. When the resource manager shared library receives the message, it will process both the io_lseek and io_read messages, effectively making that readblock() function behave atomically.

Combine messages are also useful for the stat() function. A stat() call can be implemented in the client's library as an open(), fstat(), and close(). Instead of generating three separate messages (one for each of the component functions), the library puts them together into one contiguous combine message. This boosts performance, especially over a networked connection, and also simplifies the resource manager, which doesn't need a connect function to handle stat().

The resource manager shared library takes care of the issues associated with breaking out the individual components of the combine message and passing them to the various handler functions supplied. Again, this minimizes the effort associated with writing a resource manager.

Second level default message handling

Because a large number of the messages received by a resource manager deal with a common set of attributes, QNX/Neutrino provides another level of default handling. This second level, called the iofunc_*() shared library, allows a resource manager to handle functions like stat(), chmod(), chown(), lseek(), etc. automatically, without the programmer having to write additional code. As an added benefit, these iofunc_*() default handlers implement the POSIX semantics for the messages, again offloading work from the programmer.

Three main structures need to be considered:

  • context
  • attributes structure
  • mount structure

fig: images/3struct.gif

A resource manager is responsible for three data structures.

 

The first data structure, the context, has already been discussed (see the section on "Message types"). It holds data used on a per-open basis, such as the current position into a file (the lseek() offset).

Because a resource manager may be responsible for more than one device (e.g. Devc.ser may be responsible for /dev/ser1, /dev/ser2, /dev/ser3, etc.), a data structure (called the attributes structure) holds data on a per-device basis. The attributes structure contains such items as the user and group ID of the owner of the device, the last modification time, etc.

For block I/O devices, one more structure is used. This is the mount structure, which contains data items that are global to the entire mount device.

When a number of client programs have opened various devices on a particular resource, the data structures may look like this:

fig: images/clidevs.gif

Multiple clients opening various devices on a resource manager.

 

The iofunc_*() default functions operate on the assumption that the programmer has used the default definitions for the context block and the attributes structures. This is a safe assumption for two reasons:

  1. The default context and attribute structures contain sufficient information for most applications.
  2. If the default structures don't hold enough information, they can be encapsulated within the structures that the programmer has defined.

By definition, the default structures must be the first members of their respective superstructures, allowing clean and simple access to the requisite base members by the iofunc_*() default functions:

fig: images/defaults.gif

If the default context and attribute structures don't hold enough information, they can be encapsulated within programmer-defined structures.

 

The library contains iofunc_*() default handlers for these client functions:

  • chmod()
  • chown()
  • close()
  • devctl()
  • fpathconf()
  • fseek()
  • fstat()
  • lock()
  • lseek()
  • mmap()
  • open()
  • pathconf()
  • stat()
  • utime()

Summary

By supporting pathname space mapping, having a well-defined interface to resource managers, and providing a set of libraries for common resource manager functions, QNX/Neutrino offers the developer unprecedented flexibility and simplicity in developing "drivers" for new hardware - a critical feature for many embedded systems.

© 2022 SWD GmbH (SWD)