Monday 30 December 2013

Encore!

Few things are just plain addictive. Like writing blogs. Once you start enjoying it, there is no stopping you. Or as some would agree, doing open source projects! Or for that matter, improving upon previously done ones especially when we do get a chance to do that.

Till now we worked on the boa version 0.94.13 available on the official website, and having a second opportunity to correct the wrong, we decided to go with the development version, 0.94.14rc21. Not sure, if it is the right choice, but that isn't what we are exactly worried about right now!

Its good to be back!

Saturday 14 December 2013

Not just code but beyond

A project, merely taken up maybe as a part of a course or otherwise would be limited to just coding. Making the end product work. Just the end result that matters.
This can lead to a much contrived way of doing things.

A project, done with the aim of software engineering on the other hand would concentrate more on the SE principles and aspects such as group dynamics and documentation.

Having experienced all of that and more, wondering how an open source project would be, we stepped into the world of delights as we ventured to take up the course offered as one of our electives. In it, we were mainly required to work on an open source project and it was our shot at giving something back to the community.
And this is what bloomed out of the entire process. A revelation. A sense of achievement and still the fire to do more, much more.
An open source project in its truest sense gave us the liberty to think and opened our minds to newer dimensions.
Not just the end result, but the journey, the journey mattered more than anything else. The knowledge gained out of the entire process. Not only making us technically sound, but ensuring an all-round development.
Distance  not  Displacement.

A self reference: The bloag : boa log that formed into a full fledged blog.
Creating a Mailing list!
Trying to contact the developers and the ecstasy when we got a reply.
Submitting a patch
Learning so many more things in the journey.
Making a video!
Who would have done that otherwise!


Our very own: Mailing List

oh what fun is porting

Amidst the chaos of testing for performance, I almost forgot to talk about cross-compatibility. Till now, we discussed about epoll instead of select, but this applies only to Linux systems. Epoll wouldn't work on Mac! We needed to ensure platform independence at least to an extent.

Hence, we went forward with implementing kqueue so as to be compatible with the FreeBSD environment. And a rush of conditional directives! We had to make changes in the Makefile too, so as to enable conditional compilation. This was again done for ensuring cross platform working of boa.

epoll and kqueue are similar in many respects. To start with, both are event based and have a control mechanism to add/delete modify file descriptors and events.
Kqueue is much more abstracted and hence general.

After implementing kqueue, we were wondering about the performance comparison between epoll and kqueue. But, it was not exactly the right thing to do since we had different system configurations on our laptops. Performance comparisons are only valid when all the other parameters remain same.
Theoretically, kqueue would do better. In the paper given link to below, more on performance has been discussed.

While going through the implementation of boa, we came across this in a comment "oh what fun is porting" and we agree too!!


Few useful links:
  Paper : Comparing and Evaluating Mechanisms : Page 217
  Select, Epoll, Kqueue
  Epoll



More tests!

By principle, software engineering states that the amount of time dedicated to testing the product should be atleast 40% of the total timeline. Even though we had not planned to do so, we did practice this principle.

Bombarding the servers with 1000 connections concurrently was not as easy as it seemed to be. Having various restrictions on system limits, proved to be a deterrent. Changing the ulimit helped to an extent. But having tested on systems which were not supposed to be used as servers, led to complications.

The number of times our laptops hung and had to be restarted by the hard way is more than our fingers can count. This led us to conclude that we had to find another way of bench marking our server.
This was the effect of huge files on the system, but using smaller files did not exactly make sense, since there was no guarantee that the desired level of concurrency would be maintained. Using commands like netstat helped us to find out the actual number of connections active (ESTABLISHED).
Hence, we knew we had to use larger files so that they would take more time to download.


Over the course of the tests, ab spewed out a multitude of errors few of them being...
         apr_poll: timeout specified has expired
This error would pop up whenever the server took too long to respond. We couldn't find a way to increase the hard-coded timeout value of 30s.
When testing on a Mac, this error usually implied that the system had "crashed" and was no longer responding to any user action.
         apr_socket_recv: connection reset by peer
This error is caused by the server sending a RST signal. It occurred whenever the server did not have sufficient resources to process the client requests.


Large file, High concurrency - exactly the conditions we want, for a  test, but a bad recipe for our machines.
Other options being limited (subjected to the ulimit), we started looking at different directions. Also for ab, we could not find out a way to limit the rate of the request to be fetched as we could do in wget. Again, acting upon the advice of an expert in the field, we changed the mtu.
This made the time taken for a single request longer and hence ensured the level of concurrency to be maintained to an extent. And, one more parameter gets added!

Initially we were benchmarking by only varying the total number of requests and the number of concurrent connections. Now we had two more. One was the mtu and the other, the file size. Instead of taking a specific file of an average size, we decided to vary that too, to test the server.

So we continued our tests, hoping to see the light at the end of the tunnel.

Thursday 12 December 2013

Performance analysis


Now, let us look at things a bit more practically.
After all the expected reality is always different from the actual one! Even though one's definitions of reality itself may vary.

Lets look at the performance of boa. So we decided. Historically, boa has been bench marked with ab and zb. We decided to go ahead with the Apache Benchmark.

A simple command:
   ab -n 1000 -c 500 http://10.0.0.1:12345/file.txt
   ab -n <total number of requests> -c <concurrency level> http://<ip addr>:<port>/<location>
and lots of time, is what we required

Repeatedly performing the same on epoll implementation gave us quite surprising results. No, not surprising to the extent of select performing better, but both of them performing at about the same level.


               Time taken
Concurrency Level

epoll vs select

When the lines coincide so perfectly, you wonder if this was the result expected! But Alas! It was not. We expected epoll to perform better, not just better much better!

But theoretically, we did expect select and epoll to perform just about the same for a lesser number of concurrent connections. Upon consulting an expert for advice on these matters, we were told that a concurrency level of 250 is peanuts!

Planning to try out a concurrency of a 1000 clients we wondered how loaded our machines would be.
So we decided to find out!

Tuesday 10 December 2013

Great Expectations

Performance analysis

Relying on the competence of Apache Benchmark, we decided to run performance tests on our version of Boa implemented with epoll and the original one implemented with select.
Theoretically, we expected to observe the following:

  • With large number of active connections, select may perform better as the possibility of activity on an fd in the fd set is high. Hence traversing the fd set gives rise to a low "miss" rate.
  • Similarly, with low concurrency, select's bit comparison will yield better results than the looping structure and word comparison in epoll.
  • A better performance of epoll arises when there are a large number of inactive connections. This is a scenario that is difficult to simulate.
  • Only a hard-limit (depending on the value specified by RLIMIT_NOFILE) persists for the maximum number of concurrent connections for epoll. For select, it is 1024. Although, in select, some workarounds are possible to change this maximum value by manipulating FD_SETSIZE (system dependent)
  • Within select and epoll, increasing the number of connections while keeping concurrency constant should not yield a varying value. The performance is expected to remain constant.
With this frame of reference in mind, we proceed to perform the tests.
Wish us Luck!

Monday 9 December 2013

Debugging macros


Revisiting C was another by product of working on boa.
It was not that there were complex constructs which we had difficulty comprehending but it was while implementing simple ones did we realize once again the beauty of C.

Especially during playing along with macros!

Macros being really simple in their definition and working, are at the same time most difficult to debug. That's what we felt, when we overlooked the simple macros when we got few errors.
Not compilation ones, they are the simplest to fix!

Upon adding our code and attempting to run boa for the second time, it ran and accepted requests.
Not a bad thing at all or on second thoughts, was it? This was better than the previous time where it would just exit upon attempting to connect.

Enter GDB.
THE most helpful tool. It works wonders. No need of adding extra printf statements everywhere just to check where exactly the program goes wrong.

And guess what?! The point to be pinpointed was, in a macro. A very simple and silly mistake. Perhaps it required a fresher mind to figure out.
Macros perform simple code substitution.
That's what we did. But a simple mapping doesn't always work out.

Simple mapping from the return value of a function call to a value to be set inside a macro.

#define EPOLL_ISSET(args..) {\
int i=0;\
for(i=0; i<count; ++i) \
  if(condition) {\
     return_val = 1; \
     break; \
} \
return_val = 0;\   //always sets return val to 0
}


#define EPOLL_ISSET(args..) {\
int i=0;\
return_val = 0;\   //SHOULD BE HERE
for(i=0; i<count; ++i) \
  if(condition) {\
     return_val = 1; \
     break; \
} \
}

This is what happens when mindless mapping is done. This was between a return statement returning a value from the function and the setting of a variable inside a macro!
The first thought was to write a function for the same. But it was avoided considering the manner in which boa is structured as such. An overhead avoided!

Time saved is time gained.

GDB With macros