The Multiverse of Threshold - The Universe Goes Boom and is Reborn

[Aristotle]

In a game about roleplay and story, world altering events can be an exciting and fun opportunity. That's what we intend to make out of this recent cataclysm. This will be a long post with a lot of information, and I invite you to carefully make your way through it all.

What Happened?

On Friday morning, the game crashed. I rebooted it and soon it crashed again. This was very odd so I rebooted the machine. It didn't want to come back up. After attempting a few things on my own, and working along with 3 members of our team, I hired an outside expert Linux/Networking contractor to assist. After many hours of trying numerous things, we just couldn't get the hard drive to boot nor were we able to get other servers to recognize it so we could copy off the data.

Unfortunately, something has apparently been going wrong with the hard drive for quite some time. That is why we've had a number of weird, unexpected crashes over the last year or so. It also compromised the integrity of many of our backups. Even though they were copied off the machine, many of them were corrupt.

We will be shipping the hard drive off to have data recovery attempted. This sort of thing can be extremely expensive and also doesn't have great odds of success. So we are proceeding for now with the expectation that we won't be able to recover the data - even though we are going to keep trying.

Could This Happen Again?

No. Perhaps the most brutal element of this is that we've been in the process of moving Threshold away from server colocation and onto much more reliable, higher performance cloud based server hosting. Remember when we did a bunch of DNS changing a few weeks back? That was part of the preparation for this.

Threshold is now running on Amazon's cloud servers and will run there from this point forward. It has multiple redundancies, data is backed up every few hours to a separate location, and is copied completely offsite regularly. We were able to get this going in less than 24 hours because a lot of the preparation had already been done. It is truly heartbreaking that this was not completed sooner. We've been so focused on finishing the deity/demigod storyline that we simply hadn't had the time to complete the transfer process. Extremely unfortunate.

What Have We Lost?

Ultimately, it looks like we lost about 6 months of character data, and quite a bit more of game data/code/content. We will be able to piece together some of the game data from partial restores of corrupt backups, offline copies of some work, etc. but much of it is completely lost and will have to be redone from scratch.

While we only lost 6 months of character data, there are many things related to your character that are stored separately from your character so they can be modified without your character being online. Threshold was coded long before databases were common or efficient. A lot of clan and religion data is stored this way, for example.

Absolutely No Registration Data Was Lost.

We keep that stored completely offsite and it is backed up in various ways. It will take some time to get it translated back into the game, but you will not have lost any of your registration status, tiers, etc. Many autoloads need to be recoded and getting houses and all house rooms back in place will be labor intensive.

But again, we are going to take this as an opportunity to improve things and come out the other side better than ever.

We are recoding the way the game tracks your registration total, and it should give you a way to check your reg total in game rather than having to mail us.

We are recoding the autoload submission process, which will make the autoload coding process faster by offloading some of the work to the submission template itself. This should make autoload creation much more efficient going forward.

What About the Current Registration Special?

We should have everything back in place for that in the next day or so. We will send out the reg special items, enable the Special TC Shop, and re-process all of those registrations.

What About the Story?

I am addressing this before character data/level questions because ultimately this is the most important thing. Levels come and go but the world story and roleplay is what is truly memorable about Threshold.

There will be an HRPT on Monday, November 14 during MNF (8pm EST to midnight). The Gods will share a lot of information about what has happened to the world. Much of how the Aether is changed will be revealed during this HRPT.

The short version is this:

The planet upon which Threshold exists, Primordiax, was the crux of a cataclysmic cosmic event. There was an incredible concentration of Void Energy in a single dense location (the Sludge Analyzer), and the universe tried to maintain equilibrium by balancing it with Creative Energy.

The Fountain Infuser was completed and that Void Energy was suddenly obliterated. It turns out destroying the essence if nihility is not a good idea. The result of destroying negative energy, while simultaneously having no outlet for its counterbalancing positive energy, was a catastrophe. In an attempt to vent this positive energy, a being of creation, Aryoch, opened an infinite number of wormholes to other dimensions, mixing matter and energy between alternate universes. Some universes received more matter and energy than they could contain, destroying them in the process.

You now reside in one of those universes, with people from innumerable different versions of Primordiax. They have different memories, different backgrounds, different origins, and different fates. Many of the people you thought you knew, may now have a completely different life history, guild, clan, race, or even gender.

What caused the Void Sludge in the first place? When all that Creative Energy was drawn into one place, what happened elsewhere? Was anything wrong (deliberately, accidentally, or incidentally) with the research upon which the Sludge Analyzer and Fountain Infuser were based? Were either of these devices sabotaged? How will you proceed? How will old feuds be impacted? Are the gods the same? Do they have the same rivals? What is Aryoch? And what of this threat: the Void? How can it be fought when it is clear that destroying it is to destroy oneself?


What Will Happen to Characters?

If we are able to restore the data off the hard drive, then we will be fine and we will restore all character data. But if not, there is no perfect solution and no way to fully restore characters. People play so differently, borg different amounts, there's just no reliable way to simulate that. We also can't just have people email us their best guess at their level, because there's no way to verify that, many people have no idea, honest players will guess unfairly low, dishonest players will estimate high, and the process would be so labor intensive it would take hundreds of hours.

We will code a boost based on current character level and then run the game with a large XP bonus% with advancement and training coin costs drastically reduced for the rest of the year (ending Jan 2 or so). That is really the best way to simulate the majority of lost XP, coin, etc.

Yet again, we will use this boost as a way to try and make something positive out of the situation. We will heavily promote the fact that Threshold is running at a massive XP/coin boost and hope that attracts some new (and old) players to join the game.

How Will Game Development Move Forward?

While there are many things we definitely want to re-code, we don't want to spend the next 6-12 months doing nothing but adding back "old stuff." We will continue with much of our planned new content like demigods, new guild abilities, and other features.

Mixed in with that we will redo many of our Quality of Life fixes and other types of content. Some things may never get re-done (especially if they weren't super popular).

We will put more focus on events over the next few weeks, because they are something we can do right away that is fun and doesn't require restoring tons of code. We were already planning a King's Party, for example, and we will do that along with other fun RP events.

If there's any upside to this, it will be an even greater focus on roleplay and world story than ever.

How Can I Help?

Many people have asked us how they can help. THANK YOU SO MUCH. I have always said community is the best thing about Threshold, and as we work our way through this that will be proven once again.

The best way you can help is to stay positive! Focus on the positive things we can take away from this. I truly believe this situation has created a very interesting and fun roleplay opportunity that can provide unique storylines for many years to come. This is now part of the world history of the game in a very real way.

Please don't obsess too much over lost levels, xp, or other numerical accomplishments. I'm not saying those things don't matter, because I know they do. They are part of the fun of Threshold - its more "gamey" side. But in the grand scheme of things, they are not as important as the roleplay or the story. They are not as important as the relationships and friendships in the game. None of the truly important things have been lost, and we actually have a cool opportunity to create fun gameplay from this.

There will be more details posted on specifics as soon as they are available. Please be patient. You know that we always communicate with you all and that won't change.

We Will Get Through This For the Better

I just want to stress once again that we will get through this and the game will be better than ever as a result. We've improved our server setup which will give us superior performance and reliability. We've improved our backup systems which will prevent anything like this from happening again. We've built a world story around this event that will provide interesting roleplay and character development. That which does not kill us makes us stronger.

Other Questions and Answers

I will add other Q&A to this post as necessary.

[Aristotle]

I wanted to make a second post that specifically lists the positives that will come out of this.

• Awesome world story and roleplay opportunities. I am actually extremely excited for the roleplay that will come out of this. We've already seen characters taking this situation and running with it, and we expect there will be a lot more of that. This also gives us a great opportunity to do some crazy things that wouldn't have made sense before. It fits in with the deity/demigod story we were already doing, but now we can really run with it.
  
  
• Better Connection to Threshold. Amazon has far superior redundant network pathing and backbone access than our colocation facility. Your connection to Threshold will be far more reliable and faster than ever.
  
  
• Better registration level tracking. It will be easier for you to check your registration total and easier for us to process registrations. This will also make it easier for us to create more registration tiers and give people more things for their reg total.
  
  
• Better autoload/custom item submission process. Our new system will make it easier for you to submit your proposals and faster for us to get them in game.
  
  
• Server Performance and Reliability will now be dramatically superior. On AWS we now have access to multiple redundant servers that can scale up almost immediately upon need.
  
  
• Server Performance #2: The AWS servers use SSDs (solid state drives) which are much faster than traditional HDDs (hard disk drives) like we've always used in the past. Threshold's code is very disk intensive so this is a huge performance gain.
  
  
• Data Integrity: We are now making automatic backups of character data every few hours and offloading it to Amazon's Simple Storage Service (S3). We are now also periodically downloading a full copy of the game as well. This is in addition to the fact that Amazon keeps their own backups: "Amazon S3 provides durable infrastructure to store important data and is designed for durability of 99.999999999% of objects. Your data is redundantly stored across multiple facilities and multiple devices in each facility."
  
  
• Fewer Surprise Reboots/Crashes: while we have really never had a huge problem here, there have been more of these than usual over the last year or so. We now know this was related to the hard drive slowly failing. That problem has been eliminated.
  
  
• It will bring us even closer together as a community. Like I said in the previous post, that which does not kill us makes us stronger. Many of you don't remember the only long interruption of Threshold in it's history. The game was down for about a month in 1997 due to the sysadmin at Chelmsford Online putting our server into single user mode and not realizing it. Players worked that into their RP and it was super cool.

[Aristotle]

Examples of some Craziness We Can Do:

---

CRAZY REMORT


If you'd like your character to be reborn as a different guild, heritage, race, religion, gender, stats, or anything like that, send an email to threshold@frogdice.com explaining why.

This is even available to clerics (and will be available after the Monday, 11/14 HRPT that changes up the pantheon).

Subject: Crazy Remort - CharacterName

Body:

Current name / gender / race / guild / religion (if any) / clan (if any)

What will be changed: (list)

Why you want to do this.

We won't approve all of them (we don't want a crazy imbalance or too much chaos), but we will approve a few. Please don't be mad if we don't approve yours. These have to kinda tickle our fancy or seem oddly interesting/appropriate to get approved. If we approve too many, it won't be interesting.

Note: You'll be starting over at level 0. But with accelerated XP and lowered advancement/training costs, now is the perfect time.

PLEASE DO NOT COPY AND PASTE THE TEMPLATE FROM HERE DIRECTLY INTO YOUR EMAIL. Many people's email programs preserve background colors when you do that and then the email is really hard to read. Copy it to a text file or just type it out manually please.

---

Random Insanity Throughout November and December

Since most people will be underleveled even after a boost, we can do a lot of super insane things over the next few months without unbalancing or trivializing the game. This includes:

• Massive XP Boost weekends. 2x? Pshaw. How about 4x or 5x?
  
  
• Massive Artifact Drop Rate Increases.
  
  
• Scather, Guardian Shield, etc. are unique right? Yes, of course. But what about Scather from another universe? There are a lot of crazy things like that we can make happen in the next few months.

[Aristotle]

Update on registration related things:

1) Armory and RP Storage should be correct in game. See previous posts about that for more details. If you do not have the correct amount of storage, please use the new regissue command to report it.

2) Houses: We now have the houseissue command to report issues with house rooms, wrong house, missing house, etc.

3) Registration rewards from the last 6 months. I was able to download the transaction data from the last 6 months into a spreadsheet and parse through it. I should be able to process the Coin/XP/TCs portion of *most* regs from the last 6 months. Unfortunately, since people sometimes left critical information off the reg (like character name), this won't be 100% but I think it is close enough that it is worth the time to parse through all this data.

4) Autoloads: We are working on a whole new system for submitting autoload weapon and clothing proposals. It may actually be something that is more fun in game as well. This is complicated and will require a lot of design and testing. The upside is it will make for a better system forever so that's a good thing.

5) Autoload buffs, extra powers, extra spams, etc.: Same thing as autoloads above. We are improving the system and when that is in place we will work with everyone to get their autoloads spruced up.

6) Permanent Reg Special Rewards like +gem slot tokens, +maxhp, etc.: These need to be recoded from scratch. Once we do that, it shouldn't take too long to figure out who should have received what, and we will send them out.

7) Current Reg Special Rewards: We should get all these sent out early this week so you can enjoy the reg special. Thank you!

[Qenton]

Examples of some Craziness We Can Do:

---

CRAZY REMORT

This is going to get fun.

[Kerrida]

This is going to get fun.

Absolutely agree Qenton. I am very, very eager to see how this all plays out.

[indi]

This is going to get fun.

I think this is the part that I'm most excited about.

I'm really sorry that this happened, Ari. It looks like you've carved a good way ahead, and I hope things go well from here on out.

[Harutigus]

This is probably one of the coolest things that I've seen happen in Thresh honestly.

The whole world is wide open, and even had Admin WANTED to do something like this, it allows everyone to kind of reset. I know as a perfect example with a stale and lame RP line with Eodan, that this is an awesome second chance for him.

I also know that all of the hard work they are putting into getting everyone fixed so a huge thank you!

[Zaralia]

0

[Aristotle]

The reason I am asking is that I made a remort request

Anything related to your character should be discussed privately by email. We don't want to derail public threads and most of it is private information that you wouldn't want us talking about here anyway.