Publish 2001 (Prophet speaks on Veteran Rewards)

Prophet Speaks on Veteran Rewards May 4 2001 3:37PM Hail Citizens of Britannia,

As we begin public testing of veteran rewards, I believe this is a good time to stop for a bit and discuss it. If I’m correct, many of you are thinking, “What happened? Why was the system so late in the first place? Why is it so late coming back?” Well, I hope to answer many of those questions in the following paragraphs, as I explain the history of veteran rewards from the beginning to the present. I apologize in advance for the lengthy nature of this document, but as we all know, veteran rewards have been in development for a long time!

I want to preface that explanation with a very short account of my history here at Origin. I started working on UO in September of 2000 as a server programmer, dealing mostly with the back-end server systems of UO. Since then I have touched just about every section of code in one way or another, either through bug fixes or new content addition to all the various servers and both clients. My first encounter with veteran rewards was in November of 2000, so all my information from before that time has been re-constructed from conversations I’ve had with the various developers who have worked on veteran rewards.

“Ok, so what’s so special about veteran rewards and why is it so complicated? After all, you did the Christmas gifts without a problem!” Well, with veteran rewards, we are trying to distribute a limited amount of gifts by account over all shards rather than by the number of established characters per shard. There were several reasons for this - for one thing, storing extra data slows backups down and generally decreases the performance and lifetime of the server. As well, by distributing fewer gifts, we make them more intrinsically valuable to the players.

So everyone involved at the office decided that the idea of distributing gifts by account was great. Right from the start, two significant issues were recognized as problems that would have to be solved. The first was that shards had never talked to each other before - ever! So we were going to have to design and implement a method in which they could trade information, and have this system operate in real time. The second momentous task was the calculation of account age. As it stood, a shard knew how old any one character was, but did not know the amount of billed months an account had. That information was stored in a database deep in the heart of the operations group - a completely different team than the UO live group.

The task of getting the account age to the servers was given to the operations group, and the live team took up the task of getting the servers talking to each other. Development continued for a while as planned, and things appeared to be working out well. In the summer of 2000, problems surfaced in the account age determination method. The operations group suddenly discovered that the method they had designed was hindering operation of one of the back-end servers. They informed us that they would need to redesign their solution. In an effort to keep to the planned release date, it was decided to attempt to solve the account age problem from within the UO Live team. The UO team then started work on a completely different way of calculating the account age. The new method showed promise, and the shards were successfully talking to each other, so rewards were opened to the public for testing. This is about the time I came aboard. The programmer primarily responsible for veteran rewards was moving onto another project and they needed someone to maintain the system. I was given that responsibility.

During the initial testing phase, we received quite a bit of concerned player feedback, which convinced us to re-evaluate the rewards. We pulled rewards from testing and re-designed a number of the reward choices. By this time, veteran rewards was a few months behind schedule, and with factions on the way we were running out of resources to develop and test rewards. Factions were just about ready for launch, so a push was made to finish it. Development on rewards was frozen during this period.

Just before the Christmas holiday, we had a meeting on when we would launch rewards, detailing a plan that would launch the system without interfering with the final developments of Third Dawn. A major push was made to test the system and eliminate any bugs. It was launched on Sonoma, and a major problem was found and fixed. We then proceeded to launch on the rest of the North American and European shards.

I should pause in the story to mention some of the flaws in the system that we only found later. The veteran reward system was initially designed as a simple data storage system. Over time, it has been modified to be specific to veteran rewards. During all these modifications, one of the primary design goals was to have multiple safeguards to prevent a player from ever losing a reward during a backup and not being credited for it. The failure here was that little was implemented to prevent people from getting more rewards than they were supposed to.

This is the point where things really went awry. First of all, the central data server was getting requests for account age and reward information over 200 times a second. The problem was that each of the calls took half a second to compute. In other words, every 60 seconds the server was getting 100 seconds worth of work to do. This meant delays that quickly grew out of hand. People soon had to wait 5, 10, and even 15 minutes before they got to choose rewards. Then, to make matters worse, the server system would run out of memory at times, after which it would halt all work and try to reclaim that memory. This reclaiming period took approximately half an hour. Of course, during this 30 minute period, requests for information kept piling up, making the delays even longer.

So our first pressing issue was that it was taking over 500 ms to calculate the account age of a player. This was aggravated by aggressive information polling on the shards that caused it to “hit” the data server more often than it really needed to. We immediately got to work to come up with solutions as fast as possible. Within a day, we had come up with stopgap solutions to both these problems, and by prime time the next day we were looking much better.

Things appeared to be working fine, except for a trickle of reports from the player base that people were getting extra rewards on the European shards. We quickly investigated these reports. Part of the veteran reward system gives us the ability to see who is taking what reward, and how many rewards any one account has taken. Our generated reports told us that everything was fine. Little did we know that the server was telling us this because of a flaw in the program that was corrupting the data. The next day, the trickle continued, so we looked a bit harder at our reports and checked the raw data. Eventually we found one possible loophole that could have caused a few extra rewards. This was immediately fixed. However, we still hadn’t found the more serious flaw mentioned above. In a few more days, the trickle had become a flood. I just couldn’t believe the server would be lying to me - all the tests we ran showed things were working just fine! At this point, we looked hard at the raw data and uncovered some striking discrepancies in it. It was then that we finally found the problem and pulled rewards.

We stepped back, took stock in what had led us to this point, and discussed what we could do to fix rewards. A fix was found for the flaw but no one wanted to re-launch the rewards system without putting it back on a test center. The problem at this point was that we didn’t have the resources to test rewards, as we were well into the alpha of Third Dawn. In order to keep from skipping around and dividing our focus, it was determined that all development had to be put on hold until after Third Dawn launched.

After the release of Third Dawn, we took another long look at the rewards system. What we found was that too much had been patched into the system since its original design, adding too many unknowns. It was determined that we would clean up the back-end system. Over time, what started as a clean-up and an attempt to add more fail safes turned into an almost complete re-write of the system. The new system makes sure players will always get their allotted number of rewards, no more, no less. We also added more tracking and logging information, and we should be much better equipped to analyze what’s going on in the future.

So that’s pretty much were we stand today. We hope that you will take the time and test our rewards on our test shards. The more people we have trying out the system, the more confident we will feel that we indeed have a stable and robust system.

Yours truly, Christian “Prophet” Lassonde Lead Programmer Ultima Online