[BB], 2019 in review

Another two years (given no 2018 post), another set of milestones reached. Like the 2017 review we’re going to look at what’s happened with more of a high-level overview as opposed to granular in-depth looks. We’re also going to pretend that some of the 2018 changes happened during the 2019 session just… because.

Core BBase Changes

BBase the last few years again has been the target of the big updates. Things such as tighter discord integration, adding an absolute metric ton of new cosmetics for the equipment system and hell, even the equipment system itself receiving quite the update. Of the two years, we can see in the git history that 2019 clearly received more updates as obligations from ourselves here at [BB] became more manageable as well certain things reaching completion.

One thing that didn’t happen was a replacement for the steam group bonus. Due to time constraints it was easier to implement a system within discord for a bonus instead which has led to further developments, such as Betty Bot and other realtime information being exposed to the average player. Feedback on such things has only been good so we’ll look to bring more of this in the near future I think.

Beyond that BBase is in a mature position where we can pretty much build anything around it without having to really make sweeping, large changes cursing ourselves for the lack of foresight.

Platinum is now monthly (unless you hold legacy Platinum)

This finally happened. My original plan of making an a-la carte system was really too complex for the average person just wanting to donate and be done with it but $15 one time-donations just don’t cover server bills. As a result Platinum became $8 a month of which we’ve seen some moderate success.

The next step is to get Gold ($5) out alongside branding (which will be attached to the title-card system). I appreciate some may feel that the new Platinum model isn’t as good value as $15 for life and I would prefer to keep it that way – but it just wasn’t sustainable. Along with our goal to make it so donation effects don’t become pay-to-win it means we miss out on a market (commonly known as whales) in the micro-transaction world.

We are however open to ideas on what people may like to see – so let us know if there’s something you feel we’re missing out on!

Trading / Marketplace

We finally released it! Instead of just having users trade items / cubes willy-nilly, we implemented a marketplace system that allows users to place items for sale without having to actually be on the server. This seems to have overall been successful although not without some teething issues, such as costs being too high at launch and so on.

One thing we’re currently investigating is private market listings aimed at specific people / groups of people (to incentivise trading groups) and also a “black market” listing that may appear every so often with rare / no longer obtainable items for large amounts. We’re not sure what the appetite is for such things, but proof-of-concepting exists to help discover if it’s worth our time or not.

Server Health / Maintenance

I don’t think we’ve had any major outage over the past few years beyond Windows Update sometimes taking longer, just because. The infrastructure changes have been instrumental in keeping us online at times and completely secure even when new exploits were discovered.

We have had to extend what we protect and cover – mostly down to the fact that we should have been doing so in the first place and we’ve had no excuse with the new setup to not be doing this.

Trouble in Terrorist Town is back!

I’m not going to cover too much here as the previous blog post mostly covers it. Turns out 1 month = 2 years.

Surf

Surf has gone through some interesting developments over the past few years. At the moment most of the dev time is focused on a few larger features – namely split runs and the recording feature for timetrials. Both of these are fairly finnicky as the gamemode was never designed for either of these (with the latter having the potential to grow in size in a nasty way) so the groundwork is carefully being lain for these.

The harder server also got rebranded from “hard” to “Tier 2-6” for two reasons. First off – people except those whom become true enthusiasts rarely enjoy anything too difficult (I blame the state of gaming today) and it was putting players off.

Secondly is that there are maps we don’t add to the easier server (Tier 1-3) because there are stages and maps that just don’t mesh with giving newer players a fun or interesting time. An executive decision was made that the 2-6 server can receive maps that are interesting / challenging, as long as they’re T2 and above and we’ve actually seem some success in rising playercounts on the harder server. Nothing spectacular in terms of player counts, but large enough that we’re seeing players on it daily most of the time.

The last few years have shown me with the thousand of players (yes thousands) we get per month that Surf is a niché gamemode; T2-6 is even more niche and the ‘conversion’ rate of keeping players as a result is incredibly low. More work will continue to be done on hard though these days I tend to believe unless we cater to a more hardcore crowd it’s unlikely T2-6 will ever be as busy as T1-3.

Escape

So we released Escape in a beta form with a much better map, but a more “die and you can retry immediately” sort of form. It seems players like playing it at first, but frustration happens after a few deaths and then they leave. Even with larger rewards on offer it’s not enough to keep players enticed.

It’s likely we have a winner on our hands here but we need to pace players at how often they get to play Escape, which I think the previous lobby system actually did a decent job at; though if it’s the right method to go about this is hard to say.

Go Fish

During August I got ill. Bored, I took up a community request to launch a Go Fish server. This isn’t easy because most of the source code is either from 2008 (so pre-2012 Gmod update) or has been poorly updated on the workshop.

After a few days I had a modern version of it up and running with all exploits fixed. So I suppose we have a Go Fish server now.

In future: NPC / Boss fights and some RPG elements.

Overall…

We’re in a pretty decent spot right now in terms of community. In terms of GMod as a whole, I think we’re currently in a rough spot. Using publicly available tools we can see that the overall population of GMod has gone down over the past year and coupled with the closing of the Facepunch Forums I think has only added further fuel to the fire.

S&Box is nowhere to be seen and it’s getting more and more awkward to provide for newer players given that CS:S content is no longer shipped automatically with GMod (or at least, that game-pack combo is long gone). I think 2020 / 2021 will be interesting to see how the GMod community lasts and how that impacts us as a community.

Forum Post

TTT: Back from the dead, but what took so long?

Just over 2 years since it went down, Trouble in Terrorist Town is back at [BB]! But seriously, what took so long?

The answer is fairly layered with the most obvious response being “lack of time, other priorities” and so on but it actually extends beyond this and can be broken down in to three fairly core abstractions.

Backups are Important.

First of all, when we had that big server maintenance every effort was done to back up all the things that otherwise weren’t backed up or not deemed critical whilst also racing against a clock. One thing we had to cover that TTT actually missed out on was backing up of the server.cfg & maps, meaning that there was a fairly stringent time cost to simply setting that back up. We weren’t so much starting from scratch in this area but our configuration set us apart from the crowd and dialling that back in is a case of trying to remember how it was. Furthermore TTT had been neglected with players responding accordingly (i.e. no players) so we didn’t see it as a priority.

Player tastes change

Secondly is how the community player-base moved on: here are [BB] we tend to do one thing very well whilst the rest gets neglected. It’s hard to tell sometimes if we’re following the players or if they’re following us (or even both) but typically there’s a correlation with how much work gets done on specific features vs just working on something for the hell of it.

An example of this is Surf, it was set up originally as arena-surf as that’s what I figured would be the most popular (and in 2009 that was often what was played the mode of the surf-types). Over time this evolved to a rudimentary skill-surf system which is now a full blown system due to external influences bringing players our way. Many of you that join us here at [BB] probably aren’t even aware that at times we were leading in one gamemode at some point in history. Be it RP, TTT, Deathrun (or these days Surf) and generally that’s when we’ve fleshed it out further and further.

What we have started to recognise is that players (old and new) are after another outlet and we have the perfect setup for that, so it’s time TTT gets dusted off.

A tiny point on player popularity

Something which we have to caveat the above with is for many years (2014 onwards) there were (and to a lesser degree still are) issues with servers appearing in the browser list. While this blog isn’t the time or place for it all you need to know is that for a while you wouldn’t (and even today under certain conditions) see every server available to you due to geographical filtering affecting this. This was pretty demoralising and impacted our work as we tried to work out why we haemorrhaged players.

Time is money.

Finally, the development cost. TTT around 2015 onwards started going through some rapid developments with the release of the net system. Although the net system had been released a few years earlier, the original author of TTT had seemingly faded in to the background and a community shift was looking to update and modernize it which started happening at a very rapid rate. The original edits to the [BB] TTT gamemode hadn’t been done a good job at keeping modified code clean and clear from the core TTT implementations so updates were hard. The lack of a decent diff tool at the time as well made this all the trickier.

Over time I caught up and the last major update I’d done to the TTT gamemode had been in April of 2016. Surf at this point had become incredibly popular and in August of the same year I’d gotten a new job which required my full attention so TTT more or less just fell to the wayside.

So what changed, why work on TTT?

Over time I’ve had many people come to me asking for something more combative and engaging. This tends to fall in to the category of roleplay (lets not go there) or TTT, with TTT often being suggested by old and new players alike. Development by the community has slowed on TTT meaning now is fairly opportune to go in, marry up our versions and make this work. It also lets us repair some of our errors with the benefit of hindsight.

And so what was spat out is still a clusterfuck in TTT terms but in terms of maintainability and compatibility with BBase as it stands today is so much better to the extent that we can add new features and fixes without totally breaking it or pulling our hair out. Massive shout out to those in the TTT community that added convenience hooks all over – it’s great to have standard hooks to call and manipulate stuff with.

Last night when we all played showed me that people still enjoy TTT and especially our flavour with the small enhancements. Maybe it’s the start of something big again with TTT, who knows!

Forum Post

Radio Silence

We haven’t really used this blog since mid 2018 and in some ways it’s hard to pin down exactly why; it’s more a case of I think we somewhat evolved from a blogging format similar to that of why the forums get used less and Discord is more our central point of contact.

In many ways this sucks – for both us at [BB] and for yourselves within the community because blogs are fantastic at conveying huge chunks of information with perspective that doesn’t quite fit with the short message style of Discord or the more information-seeking discussion type of a forum. Sure your opinion may still pop up and you want to give feedback for it but that’s exactly why we link blog posts to the forum.

Another notable change that happened internally at [BB] is we went from grander, loftier targets to more internal, quarterly / monthly rolling releases. While this wasn’t explicitly decided – I feel Kaiden made the right call here in effectively asking for less – but more direct and attainable content that we know we can get out the door. What this means is we have a content pipeline while bigger features quietly get worked on in the background. However what this has meant in terms of relaying this information to you is new features being added on a rolling basis without any sort of change-log or an easy explanation on how to access these changes / features.

What this overall tells me though is our current model of communication sucks. There is some communication, everyone seems to get a slightly different frame of the picture as opposed to everyone getting the same uniform message. For some things (e.g. gameplay) this can be trickier and needs to be worked on but for general information and why something is done as such – we can do better.

Going forwards I’m going to try and bring the blog back in to relevance. It gives people a chance to peek behind the curtains and for ourselves on the dev team to better relay and explain our position and ideas on what we’re doing and why we’re doing it.

Forum Post

Surfing SyncMeter explained

Syncmeter (or the little doodad shown below) was recently added to Surf and it seems to have generally caused a fair bit of confusion as to what it is, how it works and the entire point of it.

Hopefully we can clear some of the confusion up and explain where it came from, what it measures and why. Most of all however I want to explain why sync doesn’t really matter unless you’re really pushing for some top times.

Where Sync came from

Sync was originally requested here and initially I was somewhat confused by the idea of it and at the time somewhat even dismissed it as a fairly useless feature for Surf. In some ways this still holds true due to a number of factors; mostly due to how everyone calculates sync is slightly different. Sync originally came from bhop servers where it’s far more important to ensure you’re syncing your air strafes with your mouse as one wrong movement can end up with you falling drastically short.

The point of sync

As briefly explained above, its’ primary purpose is to ensure you’re pressing the right keys alongside the right mouse movements. In Surf this extends to making sure you’re not wasting more time air-strafing around corners than you need to be or that you’re not air strafing while going straight (more on this below, as it’s rather contentious).

So where does the confusion lie?

Various players have seen this new sync box pop up and are questioning – what’s the purpose of it? Do I need to have a high sync for any reason? Why does it calculate differently to xyz server I’ve played on? Is it awesome I have a high sync?

Sync Explained

Sync at its core works to calculate a score that when you’re in the air and your mouse is moving say, left, you’re also pressing A (and also the case for right and D). These airstrafes and how you manipulate them can be a significant difference in getting a meh velocity and an awesome velocity. It can even dictate how and where you land on a ramp as you’re flying in.

What sync cannot do however, is tell you what you’re doing wrong or realistically, if you’re even doing anything wrong at all. 

Even at the deepest level, all sync ensures you’re doing is that if you’ve moved your mouse in a certain direction with an intent to turn, you’ve pressed the corresponding button to do so. Airstrafing is usually fairly forgiving in this regard with Surf so unless you’re a new player (of which Sync helps you visualise what’s going on better) or a top-tier player (and even then) sync actually kind of does nothing for you.

Score Calculation

From today there’s three different methods to calculate Sync scores now implemented. Two which are available from the UI in game, the third requires you to manually change it via the console.

  1. Sync Mode 1 (or, tickbox unchecked in the F1 menu / bb_surf_sync_scoremode 0 command) – This mode allows you to hold down A or D while going straight in the air and this will contribute to your overall score as a ‘good’ thing. As mentioned before some scoring systems discount this behaviour because in theory, you should never be holding A or D when going straight at all. However due to how Source physics work there is no actual detrimental effect from this.
  2. Sync Mode 2 (or, tickbox checked in the F1 menu / bb_surf_sync_scoremode 1 command) – This mode works similar to above, however it won’t credit you with points while holding A / D going in the air. Somewhat more accurate in this regard as it won’t raise the overall sync % of the run including however you’re effectively telling the system to ignore the above scenario
  3. Sync Mode 3 (bb_surf_sync_scoremode 2 command) – This mode punishes your overall score for holding A / D while not turning in the air. The reason this mode is somewhat more locked away is for newer players – it may give them the image they’re doing potentially worse than what they are because they don’t understand the logistics behind this mode.

Why three modes?

When I was looking all this stuff up I was finding that most servers were running a variant of a few plugins or rolling their own and everyone by virtue was then creating their own scoring system for such a feature. I’d like to think there’s flexibility in how you can surf in general; so I’ve added these three modes to replicate the most common scoring systems.

In conclusion

You don’t need sync. If you’re able to grasp the basics of Surf then you really don’t need sync. If you’re new or you’re really trying to perfect your style and timetrials with little-to-no room for error, then it’s useful; but not the be all and end all. Oh and one more thing – we don’t record it serverside at all. It’s purely a measure for your eyes only.

[BB], 2017 in review

For most of us New Years is on the hinge of opening up. With that said lets review [BB] in 2017! This year the review will be geared more towards core [BB] as a whole as opposed to looking at gamemodes individually. Some of this will still occur but we’ll be primarily ignoring or summarising minor events, if at all.

Forums

This year we got a new forum design! With how [BB] had exploded in usage again we decided it was time to actually get ourselves a proper design instead of using the default XenForo skin. Kaiden was tasked with finding us at least a strong base design we could work off. It can be quite time consuming to come up with a strong palette, design etc and so Kaiden tottered off to find some shortlisted designs. We weren’t able to find any that quite tickled us the right way so we were able to settle on a base design and to modify it from there; which is the forum skin we have today!

In general the Skin works far nicer for us, especially as instead of having categories flowing down towards the bottom of the page, we instead have multiple elements per page. It nicely compacts for unregistered / guest users and spreads out nicely for those who are! We’ve seen a very small uptake in the forums as a result of the changed design which has been a nice net benefit. Design however isn’t the only thing which create participation so in the new year we’ll be looking to somehow incentivise players with using the forums more.

And before people think I’ve forgotten it – we also had a new banner created for us by Ender! It’s a pure vector image create in SVG meaning that it’s fairly flexible where we can deploy it and keep it looking sharp. Sadly GMod itself doesnt directly support SVG, but then again it’s never been great with non vmt / vtfs as a whole.

Ban Page

More of an extension to the previous point – we finally got the ban page prettified up with a few little design secrets in there – again courtesy of Ender. Under the hood the web aspect of the punishment system uses Aegis. More on this in the future.

Escape

Last year we alluded to Escape possibly coming out this year. It ended up being pushed back but not without just cause! Our original design of Escape used a map with a predefined amount of actual cubes. Now the pattern of the cubes, which ones were trapped and so on we randomised each time but there was a key flaw. If you could get to the very edge of the ‘arena’ you could wait (hopefully in the right position) and hop on to the escape cube as it did its rounds on a predefined path.

Now the game itself wasn’t particular easy. Play tests have shown that it’s actually fairly hilarious and good to play with a nice low chance but again this was a concern for us. It limited the field of play quite strongly for those that got used to the game and the limited area to squeeze through we foresaw as being problematic.

With this in mind Killermon has been working on it for the past year and I can say design-wise, we now have potentially unlimited cubes for the player to go in any direction with! Now on a more technical level we probably won’t allow for this (at least, not without further testing) even with the heavy levels of occlusion going on to make sure the server doesn’t get totally destroyed. The lack of an arena edge does create bit of a logistical issue on how the player(s) escape – but that’s for us to solve!

With any luck in the new year we’ll have something up and running so you can all finally enter the Cube.

Surf

Surf in terms of groundbreaking design was somewhat stagnant this year. Various things we wanted to accomplish with it ultimately got put on the backburner for one thing or another. Maps were added, maps were removed with some minor adjustments here and there. The biggest features we added were autohop zones and a bonus-lite system as well as the release of the solorun feature. We also standardised the seasonal rewards both in the rewards issued and how many players are rewarded.

Primarily we spent this year refining some of the backend of Surf and fixing outstanding bugs (spectator keyboard anyone?). With 2018 we should see some additional Surf features finally come online that players have been requesting.

Climb

Climb (for those who didn’t see it) was a test to see if it had enough substance for players to enjoy. For those questioning where Climb has been – it’s not been left out in the cold or forgotten about!

There’s a concern with it at the moment in terms of balancing – ie do we balance based on giving the maps their own times, gravity and such. There’s also a strong disparity between maps of a good quality and maps that just make no sense other than they want to give you the ultimate carpal tunnel. Climb is one of the older games that came along with Counterstrike where it’s done the odd thing and not really ever been standardised.

Other gamemodes such as Deathrun, Surf, Gungame and so on all developed their own sub-genres and rulesets whereas Climb just… didn’t. As a result of this we’re trying to work out how to give you the best experience and make it feel rewarding as opposed to just.. jumping simulator 2018.

BBase

Last year I said it was unusual for BBase to be the highlight of a blog post – yet this year it’s a similar story. A whole load of stuff has found its way in to BBase, including but not limited to – Achievements, Crafting, Statistics Tracking / Personal Bests, Item upgrades (full painting), Item outfits, MVP enhancements and so on. Work on these systems continues to streamline it both for ourselves and you, the players and as a result much of this has taken up our time. Almost all of these systems have initially been accepted with open arms; some requiring further tweaks but as a whole I think everyone agrees it was time well spent.

It’s worth mentioning that we did have an unfortunate casualty this year – the group system. Due to a GMod update changing some internal core Steam functions we’ve lost the functionality (or at least, the method to best do it) for the time being. It is being looked in to and we want to get it back ASAP however due to a lack of the C++ API Steam provides by both myself and Killermon, some further research is needed.

Trading

In the works. We’re still having to work out how we plan to solve various issues and to keep it as painless for both yourselves and us. What we don’t want is the great Steam trading disaster for when that first released.

Server Upgrades

While not quite BBase, this year due to unforeseen consequences we changed the entire core of how our servers work. While there were initial teething issues and configuration changes, we can safely say we’ve seen an entirely net benefit from these changes. Having the ability to redirect players to different IPs, mitigate unfortunate events and generally not having to worry about Windows shitting out has all been a refreshing breath of air. We may have older harder powering [BB] but we’ve gave it a new lease of life – and one that means when it finally becomes time to move we can move it without issue in a virtual sense, as opposed to a metal sense.

It was a long time coming to do this and it all proved to be very valuable experience.  A stressful weekend but I would do it again.

Honourable Mentions

Roleplay

Again, this year we had queries about it. Unless there’s one hell of an audience for it that are willing to play – the answer from the last yearly review still stands. Apologies to those that want their fix – it’s just super hard to justify at the moment. If people want to put together a loose wishlist of what they feel would make it fun with a [BB] twist, feel free to do so. The only requirement is you dont make x amount of fucking pointless jobs because why do people even do that.

Trouble in Terrorist Town

For the eagle-eyed, this has been missing since the server infrastructure change. It’ll be back! Probably within the next month or so.

Discord

Yeah, we use Discord now. Apparently cool kids don’t like Mumble or something.

All in all, Discord has proven to be a decent platform to use and it seems players of todays age prefer it more to the old style forums that were previously more common in the past. So we’ll maintain our current Discord for as long as required.

Deathrun

Minor fixes / changes here and there. We now offer a Weekend Madness bonus event for you to earn extra cubes with!

Overall

I’m happy to say that all in all various systems that have since taken root have been effective. Though some were initially a bit slow to come to fruition more people feel they are able to participate easier in [BB] activities. The goal for us now is to try and ensure you all have these activities (and more!) to enjoy and to have fun with your friends.

We hope you’ve enjoyed your time here at [BB] so far and continue to do so. From all of us here at [BB], happy new year.

Upcoming Feature: Autohop Zones

So Autohop is coming, but with a twist that is to try and give the best of both worlds to everyone; both users of this system and users who won’t dare touch it.

We’ve been discussing this system for a good few months, even more so since this post was made to further push the autohop discussion. In fact Admins have actually had access to a full autohop system that was useable anywhere in the map – however they weren’t able to record times with this system. Some users also got to test it fairly early on as it wasn’t entirely locked down correctly (oops).

However, more about the zones themselves

Firstly, Autohop Zones can be enabled through the F1 Menu, as below.

 

 

 

 

 

However each time the server changes level or you rejoin the server you’ll need to recheck the option each time. In general we don’t want to overly encourage this system for players. We want you to become better, to excel and to improve. Forever using this system won’t help you with that so the general idea is eventually, hopefully, you’ll just surf and completely forget you even had it turned on!

Furthermore with this system enabled you are no longer eligible for time-trial records. This was quite a hefty discussion we had internally but we came to the agreement that realistically we tend to avoid bhop heavy maps and the system isn’t deployed on most maps, therefore to keep the status quo – for now is the intention. You are however still completely eligible for Cubes without any negative impact. Therefore if you’re attempting to farm Cubes you’re now able to do so a little bit faster with somewhat more effectiveness.

For those that are looking to use this system, here’s a few locations (and an example of how they look) –

 

 

 

 

For those that have no interest in using such a system – don’t worry! You don’t ever see these zones, your records cannot be taken by these users and there’s no change to your Surf experience. For those that are using this system – other users aren’t able to tell other than the fact that you don’t set records. However as you have the ability to disable this feature anyhow it’s no sure-fire way of saying if a user is using this system or not.

If you’re eagerly anticipating the launch of this system, it’ll come around the same time as the Halloween event! And if you’re a player who is having trouble completing maps due to hopping and the like; I do suggest you use this. We’re trying to create a system that helps you in an educational sense as opposed to one that you can hop and then turn your brain off.

 

 

 

 

 

Maintenance is hard and what we learned, Pt 2.

Apologies for the day break between these two posts. I couldn’t quite find how I wanted to convey what I wanted to easily, so I gave myself an extra day.

The new setup we have hasn’t changed the hardware in any way, shape or form; bar the fact that we can now access the full 2TB of the disks we have instead of the 1TB due to the Raid 1 configuration we had at the time (for clarity, when we first brought this server online years ago; it was built with 1TB discs which have both in time failed, replaced with 2TB). Operating System-wise, we were previously running Windows Server 2008 R2 which proved to be an interesting learning experience. In essence it comes with your standard consumer windows features but with a few extra features out of the box.

For the time (and experience) this was the logical and the best option we had.  It gave a familiar environment with a neat GUI and most importantly gave us the ability to run and install Garrysmod Servers. Back in 2008 GMod could only run on Windows, no OSX or Linux options were available (well, Linux via WINE sure but this was hardly optimal) and over time I’ve come to learn about Windows own various shortcomings whilst also branching out. A really glaring issue for example is the fact that Windows handles all networking on Core 0. This means if you get some sort of Denial of Service, Windows can be overwhelmed simply by just throwing enough data for it to crunch regardless of the fact we had a 1Gbit pipe (at the time, your average attack was 30Mb – 250Mb). In a time where Denial of Service attacks were still treat primarily with disdain and nullroutes by Datacentres, this proved to be a glaring hole; the only fixes were to use a linux based OS which was capable of dealing with it better or installing a hardware firewall at a greater cost. Given the time, cost, experience and finances of those involved it was decided that the only option was to ride it out.

However it’s not all negativity. Windows Server 2008 R2 served us from 2009 until September 2017, an entire 8 years. 8 years in tech time is an incredibly long time and there’s been some interesting leaps and bounds in the server space. Cloud computing! Virtual machine prominence! Containers!11!

An impromptu meeting between various [BB] staff members was held on where we would go, how we would upgrade. While some people prefer to maintain the status quo and stick with what they know – we aim to upgrade to be bigger, better and easier. The less we have to worry about backups, redundancy and so on the better! So our plan became this

  1. The host machine will change from Windows Server 2008 R2 to Debian 9 Stretch.
    • This gives us all of the benefits of Linux with little to no downsides.
    • For example, Linux and Debian are both open, malleable systems. Open source means more eyes on it which in tandem means bugs and security issues get patched incredibly quick, often within hours of their announcements (as opposed to waiting for Patch Tuesday for Microsoft).
    • Linux is incredible with some of the networking features that we plan to leverage, especially in terms of creating greater access the most popular servers.
  2. We’ll deploy virtual machines for applications that require them. So for our Garrysmod servers, we deploy a virtual machine via KVM / QEMU that run Windows Server 2016
    • Running it in a Virtual Machine allows us to take snapshots of the OS, meaning we can rollback the entire VM if there’s a bad update, if something goes wrong
    • It also means if we swap to new hardware, we can toss it straight over without having to reinstall everything, reconfigure and so on (for the most part at least).
    • We can use manage and filter the connection between Linux and Windows. Denial of Service attack that isn’t being picked up by the Datacentre firewall? We can filter it with greater effectiveness and block it from ever hitting the Windows VM.

I’m paraphrasing most of the benefits and keeping it short. It’s something that realistically could be fleshed out more for those interested in its own blogpost. The key point I’m trying to bring across is this is fantastically better. It gives us that desired redundancy we’ve been after for a long time and neat bits of control.


When the server was initially configured, there were some misunderstandings initially from myself on how the networking actually occurred. On the second day; this is what took up the majority of the time (in total, about 2-3 hours) to configure and have working as intended. Once this aspect was sorted and I was able to connect in externally to both the Linux aspect and the Windows aspect at the same time it meant the actual workload was complete. The next few hours were based around re-uploading all Player data (130MB compressed down to 20MB) and re-configuring the software. The last remaining 3 hours of work ended up being a case of waiting on a license to be transferred and re-activated correctly so we could get the updates we needed to have everything running as previously. To remedy the fact that this was taking much longer than expected –  a manual, more crash-prone system was set up which allowed us to resume service and allow players to get back online and playing their favourite servers! Thankfully we only had to run the less-stable state for a few hours before we swapped it over.

All in all, if we ignore the trials and tribulations of the first day, it seems this would have taken about 6-8 hours to complete given the best case scenario, well within the estimated time-frame initially allocated. As shown though wrenches can be thrown in to the plan which delay these motions and make things not so much harder, but instead just plain longer. If I had to give advice to anyone doing this in future, it’d probably be these points –

  • Compartmentalisation is good. The more your systems can be spread apart (without intense fracturing / splitting) the better. This means you only need to move exactly what needs to be moved and no more.
    • An example of this is the Forums, Donation server and so on remained online without any issue due to the fact we don’t put everything on a single server.
  • Smaller input actions for bigger processing actions is ideal. A 200MB .iso is far less likely to fail compared to a 5.4GB iso, especially on a tentative connection.
  • Double, triple, quadruple check you have everything you need. Even if you expect a system to carry it over as part of its database – double check this again to ensure that this is the case.
    • A few minor configuration files were lost in the process as a result of me anticipating they were packed up elsewhere. Turns out this wasn’t the case.
    • Thankfully, that is all that was lost.
  • In the event things are taking longer than expected even though the processes themselves are realistically straightforwards and without issue, look how to deal with it faster and better instead of driving yourself insane with the same problem over and over.

Hopefully this post and the previous were of interest to those that read these. I’m hoping to hit the sweet spot between being technical yet conveying that technicality to those who may have less experience in these matters. While we’re still not 100% up and running with a couple of servers, this should be done before the week is over. Thanks to everyone for bearing with us while we did this work and for holding on while we do the last few tweaks!

Maintenance is hard and what we learned, Pt 1.

So the most recent weekend (2017/9/29-30) was quite interesting! The servers were down for approximately 27 hours with the physical machine down for about 16 hours. So what went wrong and why did we need to do the work we did?

In terms of  being up-front about one of the causes which fuelled this; we here at [BB] have been fairly silent on a breach that we experienced roughly around May. For those that are perceptive and keep track of the general infosec environment (or maybe you were just flat out affected by this), we were struck by a variant of WannaCry. This variant instead of encrypting and holding our files ransom; instead installed a bitcoin miner on the server and turned it into a zombie machine that would respond to commands from a remote service.

Just to clear one thing up – No actual data loss / data breach occurred. No data was stolen.  We hold no personal details or sensitive details (such as payment / paypal details) on marruuk (Texas) (And even then, we only ever hold tokens about information, we never see your CC / personal details ever).

What happened was part of an automated attack with no direct interest in [BB] itself and thankfully due to the odd behaviour this particular variant caused; my suspicions were roused fairly quickly and we went into triage mode, attempting to isolate exactly what had happened and ways to detect it. Unfortunately due to the nature of how it was embedded in the system we weren’t able to clean the machine in a way we knew for sure meant the infection was gone. However the fact that it was receiving instructions on Port 80 (standard website port) meant we could block it from receiving instructions.  Some time passes and we tentatively unlocked port 80 with what seemed to be no adverse effect and throughout summer we allowed it to remain as is. We always held a plan in place to wipe the server – once a machine has been compromised; you cannot guarantee you’ve fully cleaned it up, so going nuclear is the only option you have.

What brought this plan forwards was the fact that the server started accepting instruction again in September. However this time due to the nature of the instructions and what they were doing – I was actually able to discern what the cause was – a poorly documented system of Microsofts’ designed to assist in automation and administration. I removed the offending entries and all issues subsided immediately.

Fantastic!

However this wasn’t good enough – we couldn’t guarantee that we had totally eradicated any potential backdoor or problems. The decision was made to take the server down, wipe it and improve our underlying system. As of the current time writing this, we’ve seen a complete success and we’ve been able to restore everything to normality fairly quickly (give or take a few lesser played servers + the API).

But why were the servers down for so long, I thought you assured 12 hours was the maximum?

It’s a well known fact I, alongside Kaiden and Santamon are all English, living in the United Kingdom.  None of us have physical access to the machine so instead we need to remote in or instead, use a KVM over IP (basically a remote screen, mouse and keyboard etc over the internet). I had queried the datacentre to mount on the KVM an install image of an operating system to cut down on the time and fiddling of other methods we could do this by. In theory the entire process to get this part done should have taken no longer than an hour and that’s being overly generous.

However the .iso image kept unmounting itself; killing the installation constantly. I’d have to put in a ticket in, attempt to resume the installation and if that was impossible – restart from scratch. Some people define insanity as doing the same thing over and over again and expecting different results and this was certainly it. The network speeds were abhorrently slow and it seemed to unmount the .iso what seemed to be no more than 30 minutes. I had been trying since roughly 10am to get this process done and by the time I had decided to take matters in to my own hands, it was about 9:30pm. Not a bad thing that I’d taken an entire day off from my usual day-to-day activities to see this through.

The image that the datacentre was attempting to used was about 5.4GB in size; streamed from a remote location. I decided screw it – we can make this work better, we have to! Through the KVM; I manually set a 200mb .iso of the OS. It was far lighter and nowhere near complete however it gave me enough of what was required to see the job through. 20 minutes later and we had a working OS. It took another couple of hours to get the initial networking put together and by 2:30, or maybe 3am Marruuk was finally alive. Not ready to start hosting servers, but she was alive.

To be continued in Pt. 2 where we elaborate on the new setup, a few other reasons as to why we did it, somewhat of how we did it and the overall conclusion.

Item refunding (and why it isn’t happening, sort of, kinda, in a way?)

Alternative Title: How Cubes and Crates work in [BB] and why cube refunding for items isn’t really a thing, but maybe it will be.

As players recently have noticed they’ve been acruing items over time, a common request has come up – can we please do something with these items (ie, refunding for a % value) please?!

Well, the answer to this is – sort of, maybe, kinda,  yes…? But no.

From a gameplay point of view, Cubes are actually quite important in the ecosystem of things, even if players don’t initially realise it (and this will be a larger scale of why many other games implement this feature for what seems entirely redundant in some cases). Currency in a game is an easy way to express progression; without having to actually display anything other than an arbitrary number going up. User completes action A; they receive X currency reward. User complates action B; they receive further X currency reward. In mobile games this is actually used heavily to try and tap in to that part of the brain which likes to gamble and so on, but more on this another time.

Within the [BB] ecosystem Cubes allow for a sense of progression, a journey. Once you’ve reached so far in that journey you’re able to purchase a cosmetic to display how awesome you are! That item then becomes part of the next leg of your journey as you continue to farm for more Cubes to kit yourself out – for the bigger; the better and flashier looks. For a while Cubes were the only way to gain items – so if you weren’t good at the task given to you: you basically gained nothing. 

This sucks – so how can we improve this? Random crate drops!

Crate dropping systems are in some way predatory (but that’s a blog post for another day) however they fill an interesting niche that both conflicts yet compliments the Cube system. Within [BB] every day or so you’ll gain a crate when you spawn. For many players this is often the first time they load in on a fresh day – but technically every death (assuming the internal cooldown isn’t active for that player) can trigger a crate drop. Within these crates is a sample of items – often new sets mixed alongside some older sets with high quality, high sought after items alongside what many items will consider to be junk – or low value items. Better yet – you can often get a version of the item which is ‘unusual’ which has special effects and has an approximate 2% value to appear.

To move things along a bit, we end up hitting a wall. We have items now being conjured out of thin air (as opposed to being traded for Cubes) meaning that over time players are going to accrue a large amount of items that they don’t really want. And there’s nothing worse than forcing something on a player which they really do not want. This brings us to the current predicament where players are looking for solutions that others have implemented and are going ‘why not do this!’

Remember how I said that the Cube and Crate drop system somewhat compliment each other but also fight each other? By giving players items for free; we’re diminishing the value of Cubes – especially so when we’re offering special effect versions that can only be uncrated. With this Cubes are now less attractive albeit still serving a purpose of progression and allowing players to get an item directly as opposed to rolling the dice. However, what if we allowed them to refund these items for % values?

Well, this is where things go haywire. If we start allowing for items to be turned into a direct Cube value it completely destroys any value Cubes have. Under the right conditions and drops players could actually earn far more from a refund than they would playing the actual games themselves. The fact that players are suggesting that items be turned into Cubes shows that the player considers Cubes to at least have value; possibly (and understandably, it’s not their job to) not seeing the long term consequences. What players want is a useful way to take items and morph them into something else to be useful, this is where the crafting system comes in.

Essentially with crafting you’ll be destroying items you never use / want which will allow you to build up to making various items. Internally there’s some debate as to if we should release the demolition system first (ie the destruction of items) to allow you to gather reagents – but this doesn’t really solve the issue long term because all we’re doing is replacing one item that actually has a use at that time to an item that literally just sits there (think the Halloween candy or the xmas charcoal). So all I can say really is watch this space. We’ve heard you loud and clear – we just need to see how we can make this a priority and the best way of making it at least somewhat interesting (and decent). The last thing we want is to release something that simply doesn’t work well – with a system like the painting system we’re able to get away with iterating over a few updates (which is what happened). With something that affects items and Cubes in such a way we’re only going to get a single shot at getting this right which means we need to put the work into this in the right way.

That said however, there is a use-case currently where item refunding is valid.

As of the moment there’s no easy way to preview items (no idea why we haven’t stuck this in yet, in theory we probably have all the systems in there, they just need to be connected up) so often players are having to experiment, query other players or simply take that risk which is unfair to the end player. With that in mind, what I’m happy to do from today (until we add a feature to automate this) is any item that you submit a forum post for in the dev section (something like Item Refund – Item Name) within two days of purchasing; we’ll refund you the Cube amount for 100% of the value and remove the item from your inventory. Though we do ask that you don’t decide to bulk-buy items and then choose to refund them because we do reserve that right if you’re doing it too often to simply deny the refund.

To sum it all up – we know there’s problems with the Store (including things not listed here) and the obscene amount of items players are totalling up at this point in time – but in this case it’s not entirely straightforwards to craft a solution due to the delicacy of it all and we’re working on it!