BukGet Infrastructure Expansion

Sun 29 June 2014 Posted by Steven McGrath Category BukGet

tags: python  bukget  minecraft  Comments

 

NOTE: Cross-posted from BukGet's news page.

In the past we have run into issues with the API simply keeping up the demand, causing us to expand out to a second server some years ago to help distribute the load. More recently however, the architecture that we have been priding ourselves in being simple and efficient bit us in the butt when the US server went off-line for an extended period of time. Unfortunately for us, this was beyond our control, as the hosting provider that was donating us the gear ran into problems and their entire facility went off-line. We scrambled to being up a temporary master so that generations could continue as normal, however we had to develop a better model for handling this.

Our initial thought was to try to take advantage of the situation and get the community to help us raise the capitol to run our own gear, our way. Unfortunately reality set in pretty quickly when our rally cries were either deleted on Reddit, or generally ignored. A few companies however did respond to offer us some more donated gear and gratis hosting in order to keep us up and running. As it seems that this model is the only model that the community wants to support, we eventually conceited defeat on trying to get our own gear and started communicating with some of the various hosting providers that had contacted us.

With this in mind, we started to engage several companies and then quickly narrowed the list down from there. We wanted to still keep the overall environment small so that it's manageable by our 2-man (lets be honest, David does most of the ops stuff now) operations team. We also made some small, but fundamental changes to how we will be handling the data, generations, etc. as well. As a result, we went from 2 servers to 6, and we will also be looking into making some other changes down the road to help us react quickly and effectively to any future changes.

The first change we did was make the decision to have dedicated generation servers. In the past, all generation was done on the Dallas API server, and while this never cause any issues, we were putting too many eggs in one basket. We also figured that is 1 generation server is good, it might make sense to have a backup server in-case the first one goes down for any reason (like in-case, you know, our provider goes down).

The second change is that we now will have 2 servers in each geographic region. Each server will also be hosted by a different provider. This means if something goes down, that whole region wont be be (hopefully) significantly impacted.

Thirdly, we decided to expand out the MongoDB instances a touch, making sure that the API servers are all identical (and now slaved) to the current active master generator.

Lastly, while the city-based naming convention worked well for us, it often created some confusion. So we are switching up to a state/country model with each region also rolled up into it's respective regional record. This means that to a large degree, we have added some more granularity to how we are referencing the our systems, and also help developers when troubleshooting issues with their code.

In general our goal here is to try to make the BukGet platform generally more stable for everyone and provide the needed resources to grow. We have already talked internally about starting to support more servers than just Bukkit, however some of these changes were needed first to really bring everything up to speed before we could start working down those paths.

Comments


BukGet API3 gets Plugin & File IDs.

Mon 27 January 2014 Posted by Steven McGrath Category BukGet

tags: python  bukget  minecraft  Comments

 

BukGet has been updated with the curse_id and file_id attributes for plugins and versions respectively. These ids are considered non-changing within the Curse systems and are there for plugin devs and plugin management devs to wire into for something more concrete than the slug names.

What intended next is to ...

BukGet gets a new Dallas Server

Mon 06 January 2014 Posted by Steven McGrath Category BukGet

tags: python  bukget  minecraft  api  Comments

 

After the fallout of the ShardHost debacle, generously stepped up to fill the void. To be honest a lot of this had happened back in November, however due to my day job, the holidays, and some general laziness (I would like to play the occasional game as well you know ...

BukGet Goes Multi-Server

Sat 01 June 2013 Posted by Steven McGrath Category BukGet

tags: python  bukget  minecraft  api  Comments

 

Over the past few weeks a lot of work has been done to the API code to be able to support multiple servers. I am happy to say that the new 2-server configuration has been running stably without any significant issues. No changes need to be made to any existing ...

BukGet Gets a Status Page

Tue 14 May 2013 Posted by Steven McGrath Category BukGet

tags: python  bukget  minecraft  api  Comments

 

A couple of things have been happening after the API server started to melt down. The first thing is that we have migrated the primary host to a Shard Hosting server. The guys at Shard were quite happy to provide us a box that is more that appropriately sized for ...

Page 1 / 4 »