Few Clarifications

Following in from the design issue discussed earlier, as we realized we could not completely replace select with epoll, there had to be macros involved.
Fallback options : select - to make sure in the absence of epoll, boa could work with the original configuration.
And hence we considered the option of checking the type of the system, which was just a check on the predefined macros __linux and __APPLE__.

But was this check enough?
Linux has inbuilt epoll support from kernel version 2.5.44.
Is it enough to check the kind of system that is and assume that the feature is supported?
I was not sure. Hence, the double check. Found a piece of code which claimed to check for epoll support across systems [1]. Considered to use it in the epoll implementation to make a choice.


Another realization that came our way was that we needn't change the request structure itself, instead build the code around it. The request structure could remain the same way and function as it always did.

Also, as mentioned in the comment and the previous blog post [2] we do not have to consider altering the setting of the max connections as it is set to RLIMIT_NOFILE. Even though epoll may not have a limit on the maximum number of clients connected, but system limits have to be always accounted for. Also, setting the value to RLIMIT_INFINITY could cause some system dependent issues.


[1] http://www.gnu.org/software/autoconf-archive/ax_have_epoll.html
[2] http://a-voyage-into-the-land-of-boa.blogspot.in/2013/10/first-glance-at-code.html

Choosing a Mode of Execution: A Design Issue

An important aspect to examine prior to diving into the actual implementation is the design issue regarding choice of modes among select, epoll and kqueue.

Several possible solutions exist. We shall consider ones that appear most relevant.

Case 1: 

Replacing select in its entirety with either epoll or kqueue

As a convenience to the user, it makes sense to take the responsibility of choosing the path away from the user; make it in-built. 

Since the choice is only among 2 options, usage of macros would be a convenient solution.^[1]

That's pretty nifty.

As our primary aim is to provide an enhancement to the web server, augmenting extensibility features will be an added advantage. The code can be made generic, adapting a common structure to handle requests through either approach. The function activated can depend on the macro chosen in as per the approach declared previously.

struct kevent {

             uintptr_t       ident;          /* identifier for this event */

             int16_t         filter;         /* filter for event */

             uint16_t        flags;          /* general flags */

             uint32_t        fflags;         /* filter-specific flags */

             intptr_t        data;           /* filter-specific data */

             void            *udata;         /* opaque user data identifier */

     };

typedef union epoll_data {

               void    *ptr;

               int      fd;

               uint32_t u32;

               uint64_t u64;

} epoll_data_t;

struct epoll_event {

               uint32_t     events;    /* Epoll events */

               epoll_data_t data;      /* User data variable */

};

We have provided the corresponding kqueue and epoll structures for reference.

            

Case 2: 

Keeping select implementation, and having a choice between select and epoll/kqueue

This approach is an extension of the above. The user has a choice to either use select or an event mechanism (ie epoll/kqeueue. This choice will be made inherently depending on the type of the system)

Code modules have a design potential.^[2] But they might be an overkill with respect to BOA. 

            

Case 3: 

The choice among select, epoll and kqueue is left to the user.

You might wonder whether this is really advisable. After all, the user must not be bothered with implementation technicality. But, in the future, there may arise an operating system that provides libraries containing both epoll and kqueue.^[3] Perhaps a user extremely particular about performance with the knowledge of the type of traffic expected will also find this useful.

In the actual code, a three-way switch will comfortably handle this circumstance.

Note regarding kFreeBSD: (see link [3])

Apparently you can run both Linux and FreeBSD binaries on it. It supports calls up to Linux version 2.6.18. But, using macros here will result in only epoll being used. (we check for __APPLE__ and not FreeBSD. This operating system will fall under __UNIX__ flag.)

The situation appears thus, having articulated possible solutions. We do hope for your valuable suggestions or additions. They would be most helpful in making an appropriate choice.

[1] Using macros to choose

[2] Explanation of Apache modules

[3] kFreeBSD

Determining what to Change

God grant me the serenity to accept the things I cannot change, the courage to change the things I can, and the wisdom to know the difference.

This wisdom is generally what we may lack at the crucial time when the code is handled for changing. We change things that might have had other impacts, dependencies and as a result of which our entire effort might just have to be redone.

Following in from the previous brief discussion of the various function calls made, we can clearly distinguish between the two groups.

The major goal of the entire project being the transition from the use of select to efficiently make use of the epoll mechanism, the majority of the work involved in playing with socket based calls. It is clear that no initialization functions need any sort of tweaks.

The file or the function that has to be majorly changed is of course select_loop( ) itself!
The file select.c contains the major logic of the select statement which needs to be adapted for epoll.
Also, the processing is done in the form of queues. Three different types of queues are used for processing the requests. These do not require any major changes but for the dependence on their use of the structure request.

Examining dependencies further, we come across a structure - struct request which stores the incoming request details and is further used for processing the actual request. Defined in the globals.h file, it is now marked for change to incorporate the epoll based event structure.

A deluge of pointers awaits as we explore the files request.c and response.c. Mainly used for processing the request structure, they become potential candidates and so does buffer.c which contains functions which help in buffering the data before sending it to the client. To he file, get.c performs a part to get the headers of the incoming request. Reading and writing involves the use of pipes and hence the file, pipe.c is another contender. Inspecting further, it is found that the file cgi.c may need minor tweaks for it uses the file descriptor from the request structure.

Here, we have now enumerated the ones that possibly need to be changed. Now let us look at them in detail so as to understand them better.

First glance at the code

Basically written in C, the code is a simplistic one, with not many surprises(yet).

It is always a good thing to begin from the start, and hence we started looking at the file boa.c. Yes, it is practically the first one to get executed (or rather the binary of it that is).

A couple of interesting observations can be made about it looking at its organization. The file itself does not give any deep insights into the code as such, but describes the structure to the execution.

Following a top-down approach, a few static functions were found, code to set various permissions, making stdin and stdout point to dev/null and the usual parsing of command line arguments followed.

What interested me here is the setting of max_connections to RLIMIT_NOFILE which is one of the limitations we shall improve upon by not using select. Possibly this was the first line of code we knew will be altered when it was spotted.

Glimpses of few of the functions..

There are calls to several functions which basically performed the initial setup the server required to run. The foremost of them being fixup_server_root() making sure the root given in the defines.h file or the one entered via command line (-c option) is valid.
The function, read_config_files, reads the configuration file! Well what else could we expect it to do!
The configuration file must be located in the location folder marked as server root.

Digressing here a little about the config file, it has a flat hierarchy, making it all the more easy to read, understand and use effectively. To mention a few of the directives, the port number and the IP address to listen on are most commonly used. Others specify the file locations of the access and error logs. Document root is another directive that one would want to set.
The file itself is parsed with the use of lex/yacc or a similar generated parser. The function, after reading the config file, sets the appropriate variables as given in the boa.conf file. The structure to store the configuration parameters is in its entirety a marvel. All the possible config parameters are stored in 4 value tuples - Name, Type of the config entry, Boolean value that indicates the presence/absence of the entry, Value of the entry if any. The type of the config entry, being S1A, S0A, or S2A determines the number of arguments it can have.

Returning back to our file, boa.c we have several more functions to be called.
The call open_logs() placed right after the parsing of the config file, checks the values set by the previous function and opens the log files accordingly.
The server socket is created and the initial parameters set by the function create_server_socket(). The socket is bound to the appropriate server family depending on the value given in the configuration file.
As the name suggests, init_signals does the task of setting up various signal handlers. It is then checked if the server is running as root, and if not, a log entry in the error_log appears as a warning.
The function create_common_env() sets up environment variables common to CGI scripts.
To make sure the difference in the implementations wont affect how it works, the function build_needs_escape() escapes characters based on the bit positions available in unsigned long.

Only after all this initial setting up, it forks itself to push into the background if the command line option '-d' is not toggled on which tells the program to not fork.
Immediately after it forks, the timestamp is logged in.

The main loop function: select_loop() which takes the server socket as parameter is finally called. And this just marks the beginning of the voyage.