Scott’s Blog

Systems Admin for the Indie Dev

April 26, 2016
/ / /

devotion_to_duty

Hello again! It’s been a long time since I’ve posted anything, but I’d like to let you know that Jake and I are both still alive and well, heads down on development of our upcoming RPG, Ethereal Legends. We’ve still got a good amount of work to do on that before it’s released, but I recently took some time off from game dev to reconfigure our network infrastructure, and I felt as though my experience could be helpful for any other indie devs looking to start their own game companies.  This post isn’t aimed at being a nitty-gritty tutorial on exactly how to configure your network, but is more directed at being a high-level overview of some of the choices we made when building our network, and why. So here we go!

I have a bit of prior experience with this subject, as systems administration for small business is my day job, outside of game development. I think systems administration is something a lot of new indie devs simply don’t think about. Chances are that a lot of starting indie devs are sitting at home, where they don’t worry about static IP addressing. They don’t have physical servers or a proper source control setup. They’ve never had a reason to connect two machines together with anything other than Dropbox or Google Drive. And while this is a perfectly fine way to do things when you’re working by yourself, it becomes extremely cumbersome when you start scaling and working with other people. Eventually, you need some way to centralize, organize, and secure your files, in a way that is easily accessible to the people you work with.

Even though Systems Administration is my day job, and I love doing what I do, SysAdmin for me has always been a “means to an end.” I don’t really build networks or servers because I enjoy it (even though I truly do); instead I taught myself how to do all these things out of necessity. I neeeded a website for the games I was building. I needed a shared file server with source control, in order to more easily perform backups. I needed a local domain to securely authenticate users in order to work with others on those same files. I needed an FTP server to deliver the game download to players over the internet. These were all things that I felt we absolutely needed at Soverance, and while there are many ways to accomplish these tasks, I wanted something that was robust, fully featured, and able to scale, in the event Soverance was able to grow from the tiny operation it is today and into the true gamedev company I want it to become.

Our main Workstation setup - Jake's desk and our MediaServer.

Our main Workstation setup – Jake’s desk and our MediaServer. That’s my office on the right, and our VR room on the left.

To accomplish these goals, we turned to Microsoft and the Windows platform. In addition to my previous experience with Microsoft, the main reason we chose to go with MS is because of their BizSpark program. If you’ve not heard of the program before, it’s basically like a “hands-off” startup accelerator – they give you three years to basically go hog-wild with Microsoft software and services, free of charge. That includes a significant amount of credits for Azure cloud, as well as license keys for numerous versions of Windows, Server, Office, SQL Server, and whatever else MS makes that you might find useful. As an fledgling indie game company, this type of support is invaluable. Yes, you can put a monetary cost on what they give you, and it’d be into the tens of thousands of dollars… but if you put it to use, you can learn a lot of valuable knowledge about how your company will operate in the future.  Either way, if you’re starting a game studio (or any company, really), I highly recommend checking out the MS BizSpark program.

After we gained access to the software, we had to plan out our network and determine our requirements. How many users would we have? What services did we need? What hardware did we need to deliver those services with high availability and decent performance? What was our budget? Our requirements were essentially as follows:

  • 4 users; an Administrator and 3 standard users (myself, jake, and our MediaServer).
  • Must be able to add more users.
  • All users need secure access to a large (2TB+) shared data repository.
  • Secured Perforce source control for our Unreal Engine projects
  • A public-facing website
  • An anonymous FTP server for game downloads
  • Daily backups to a local external drive.
  • Stable, always-on private domain email.
  • Cannot overload any one system with too many processes
  • Minimal budget, less than $2000
  • All under one roof, so we can centrally manage updates, applications, and authentication.

The last requirement was the most difficult, but luckily BizSpark has provided us with everything we needed. Now let’s jump down into some of the basics of Windows Server administration.

First, to get the user administration we want, we must run a Windows Server role known as Active Directory. Without going real in depth, Active Directory (AD) is basically the equivalent of a master control room for your domain network; you’ll use AD to create and manage users, permissions, and security policies for every device connected to your network. It’s also an integral part of your internal DNS infrastructure, allowing you to refer to computers with their names instead of their IP addresses. Microsoft recommends as best practice that you run Active Directory using two servers; Primary and Secondary Domain Controllers. They’ll replicate data between themselves, as well as help spread the load from queries to the AD and DNS services so you’re not overloading one machine. Setting up two servers as Domain Controllers also takes care of the file sharing requirement, as you can have one of the servers pull double duty as the file server.

You’ll need to budget for a third server to run Windows Server Update Services (WSUS). This box will generally be authoritative over every other Windows machine on your local network, in regards to updates. A vital part of any MS network, once it’s set up it’ll just run in the background and update your machines when appropriate. I set the machines on my network to update once per week, at an early morning hour when I know we’re not using the network.

serverbank

Mmm servers….

For anything public-facing, like the website you’re reading this on or the game’s FTP server, we needed a different solution if we were going to maintain high-availability and security. I wasn’t exactly keen on letting people hit our office’s public IP when visiting the website, as it opens us up to some security concerns, and I didn’t want to have to worry about the website or game being down because our power or internet went out here in the building. That’s where Microsoft’s Azure cloud platform comes in. Once again, Microsoft recommends that you run not just one, but two virtual machines in what they call an “Availability Set”, which helps to limit the effects of hardware failures, network outages, and power interruptions inside the Azure datacenter. Through the BizSpark program, you receive $150 per month in Azure credits, which is more than enough to run two VMs full time all month.

The real trick came when it was time to connect the Azure VMs to the local domain. Since they’re not physically in the building, creating a secure connection requires a VPN device, to create an always-on VPN connection between your Azure virtual network and your private local network. We’ve fortunately got one, an old Netgear FVS124G, which you can probably find somewhere for less than $100. I was surprised that this particular VPN device worked with Azure, but sure enough, it did. There’s documentation for some of the more common firewalls (like Cisco/Juniper/FortiGate stuff), but basically nothing on Netgear. Trial and error did the trick, but I lost about a day messing with it. Either way, once you’ve established the VPN connection between your local domain and Azure, you can join them to your domain just as any other locally connected machine. This obfuscates the fact that they’re run at all in the cloud, allowing me to access them from my dev workstation just as I would any of the local machines.

Finally, we needed stable email, and for that we really only had two choices:  Google for Work or Office 365.  We tried Office 365 first, mostly because I wanted to stay within the Microsoft ecosystem. Unfortunately, one of the biggest features for me was the Single Sign On feature, which would have allowed us to manage the Office 365 accounts using our local Active Directory, and users could sign in using their AD credentials. Unfortunately, I ran into some issues setting this up due to my Azure and Office 365 accounts not wanting to play nicely with each other, and after many hours on the line with O365 support, I decided to just drop it and move to Google. We’ve been on Google for Work for a number of years before all this, so we were already familiar with it’s management. Unfortunately, it’s now the only service we use that isn’t under the same roof, which kind of annoys me but isn’t a problem.

When all said and done, our network diagram looks like this:

network-map

As you can see, we’re pretty simple; we’ve honestly got more horsepower than necessary for a company of our size. Let’s go over the purpose of each device in the diagram, above.

  • Netgear FVS124G VPN Firewall
    • Everything starts here! It’s a crappy firewall that I got for free, but it’s gigabit VPN and it works just fine, so whatever. We plan to replace it with something better one day.
  • Scott
    • My workstation. Windows 10, i7-6700k Skylake, 32 GB RAM, Nvidia GTX 970
  • Jake
    • Jake’s workstation. Windows 10, i5 something, 8 GB RAM, Nvidia GTX 980  (we know, Jake needs more RAM)
  • Media Server
    • Another Windows 10 machine, but it mostly just runs our TV. AMD Phenom II X4, 16 GB RAM, Nvidia GTX 750 TI
  • SOV-DC1
    • Dell PowerEdge 2950, 2 Xeon X5365 processors, 16 GB RAM; 75 GB Raid 1; 2 TB Raid 1
    • Primary Domain Controller
    • File Shares
    • Perforce repository
  • SOV-DC2
    • Dell PowerEdge 1950, 2 Xeon 5160 processors, 4 GB RAM; 75 GB Raid 1
    • Secondary Domain Controller
  • SOV-IIS
    • Dell PowerEdge 1950, 2 Xeon 5160 processors, 4 GB RAM; 75 GB Raid 1
    • WSUS
  • SOV-Game
    • Dell PowerEdge 1950, 2 Xeon 5160 processors, 4 GB RAM; 75 GB Raid 1
    • Internal game server, for testing multiplayer. We usually leave it offline, since we’re not currently testing multiplayer in Ethereal.
    • SQL-Server Instance (unused, thought we might use it for something one day)
  • SOV-WEB
    • Azure VM, 1 core, 3.5 GB RAM
    • soverance.com website files
    • ftp.soverance.com, anonymous FTP for the Ethereal Legends game launcher to update client installs.
  • SOV-AzureDC
    • Azure VM, 1 core, 3.5 GB RAM
    • Replica Domain Controller
    • MySQL instance for soverance.com

As you can see, we’ve got a pretty solid network. We found a great deal on the servers themselves, getting the used 2950 donated to us by a friend, and finding the used 1950’s on CraigsList for $50 each. Spent a few hundred bucks putting the server bank together, as far as buying the rack, rails, and associated items (like the PSU and monitor tray). I was actually amazed that we spent more money on the rack itself than we spent on the servers… but hey, what are you gonna do?

Soverance server bank

More than enough space left in our rack, if we need to expand. Yes, that’s a Kinect v2 on the tripod.

Altogether I think we came out with something pretty nice. There’s still a few upgrades I’d like to make… for example, I’d like to get a large NAS to use as our backup device, instead of the crappy little external drive I’m using now, which could store regular and complete images of each machine’s system state. Definitely want to get a better VPN firewall, preferably one that has wifi. Our local servers all have 15k rpm SAS drives as their OS drives, and while that’s decent enough for our purposes, I wouldn’t mind replacing them with SSDs like our workstations have. We’ve also got enough drive slots left over in the 2950 that we could add more storage if necessary.

Probably the next upgrade we’ll do, though, is buying a portable air conditioning unit. The room we put the servers in easily rose in temperature anywhere from 10 to 15 degrees, maybe more. Not a big deal when we turn the air on… there’s a lot of ventilation and the A/C gets cold. But we don’t always leave the A/C running, and one day we plan to move the servers into a smaller closet, so we’ll need the cooling.

In the end, I’m super happy with how our network has matured here at Soverance. We’ve got the infrastructure in place now to build some really cool things. Hopefully this post has helped inspire some other indie devs that know that they too can build great networks to help facilitate the success of their teams.

Now it’s time to get back to game development!

Read More

#IndieDEV Podcast Episode 07

April 21, 2016
/ / /

Seventh episode of the couch collaboration efforts of Ron, Scott, Jake, and D’metria. This week we are talking about how awesome the day is, being an entrepreneur, and more…

Be sure to like and subscribe, let us know what you think and what you want to hear us talk about in future episodes!

-Hosted by The Indie Cluster and sponsored by Soverance Studios

Read More

Endless Reach 2.4.1 is now Open Source

January 9, 2016
/ / /

# EndlessReach
Source code for the last Unity version of Endless Reach, dated November 2014.

// version 2.4.1 – November 2014
// Soverance Studios
// www.soverance.com

githubLink

This repository contains the source code for the final version of Endless Reach, built by Soverance Studios. The last version update was version 2.4.1, released in November of 2014. This application was built in Unity 4 and programmed in C#, and as it will never again be updated, the project is now entirely open source. No assets are included, this is only the source code.

Endless Reach is a hardcore shoot ‘em up that puts you in the shoes of a lone space drone pilot, laying claim to the uncharted galaxy known as the Reach.

Send your starfighter drone out into the unknown, and use it to battle your way through the alien hordes and make the regions safe for colonization. The future of humanity depends on you!

Endless Reach was originally prototyped for mobile devices, and released for Android and Windows Phone devices. Oculus Rift support was added shortly afterward, and the VR supported version was released to Oculus Share. We also ran promotional campaigns for Endless Reach on both Steam Greenlight and the Square Enix Collective, though both were ultimately unsuccessful. The Greenlight page has since been removed, but the remaining informative links persist:

Main Website: http://soverance.com/endlessreach/

Google Play: https://play.google.com/store/apps/details?id=com.Soverance.EndlessReach&hl=en

Windows Phone: https://www.microsoft.com/en-us/store/apps/endless-reach/9nblggh0k1tz

Oculus Share: https://share.oculus.com/app/endless-reach

Square Enix Collective: http://collective.square-enix.com/projects/54/endless-reach

Reddit: https://www.reddit.com/r/EndlessReach/

Endless Reach contains some basic shoot ’em up logic with the addition of a few new things, namely the Overload and Boost features. Online leaderboards are also incorporated, and they are powered a public Google Spreadsheet. There’s saving and loading, as well as a “world hub” called the Nautilus that essentially serves as a fancy level select menu.

[KEY FEATURES]

  • 10 action-packed levels set in beautiful spacescapes
  • Amazing electronic soundtrack by Shiny Baubles
  • High powered particle effects and visual stimuli
  • Power-up chaining and limit break mode
  • 5 enemy types, 3 turret types, and a different boss in every stage!
  • Missile targeting via head tracking!

[PLATFORMS]

note that all platforms other than VR are no longer in development. VR ONLY!

  • Complete Version History is found on soverance.com.
  • Windows PC – VR [Oculus Rift]
  • Android
  • Windows Phone

[INSTALLATION INSTRUCTIONS]

Download from Official Website: http://soverance.com/endlessreach/

[CONTROLS]

Endless Reach supports the Xbox 360 gamepad ONLY

  • Head Tracking – Lock On Target [look at target to lock]
  • Left Analog – Drone movement
  • Left Stick Down (L3) – Unused
  • Right Analog – Unused
  • Right Stick Down (R3) – Unused
  • Directional pad – Menu movement
  • A – Fire
  • X – Purge [when prompted]
  • B – Missile [when locked on target]
  • Y – Unused
  • Left Trigger – Boost [hold to continue]
  • Right Trigger – Unused
  • Left Shoulder – Toggle FPS counter
  • Right Shoulder – Unused
  • Start – Pause / Unpause
  • Back – Reset HMD Orientation

[FIRE MODES]

Collect BLUE orbs to advance fire mode to the next level.

* Standard [0.4 second fire rate, single shot, blue effect]
* Plus [0.2 second fire rate, single shot, green effect]
* Multi [0.1 second fire rate, triple shot loose, yellow effect]
* Cannon [0.06 second fire rate, double shot strong, red effect]
* Limit Break [20 second duration, 0.03 second fire rate, single shot strong, purple effect + shield]

[LIMIT BREAK]

* When entering a Limit Break, a timer will appear on the HUD. This timer indicates the remaining duration before the ship begins to overload.
* You can reset the timer, thereby extending the duration of a Limit Break, by simply collecting another blue power orb.
* The drone takes no damage during limit break.
* When the Limit timer reaches zero, the drone will begin to Overload.

[OVERLOADING]

* The drone becomes unable to move while overloading.
* Overloading is a sort of quick time event, in which the player will be prompted to press button “X” in order to safely purge the overloading energy.
* A successful purge will send a large shockwave to destroy all enemies within a large radius, and deal damage to bosses as 25% of their maximum HP.
* Failure to purge at the appropriate time will destroy the drone.

[BOOST]

Hold the Left Trigger to increase movement speed. Initial boost shockwave will destroy all enemies in a small radius. The shockwave does no damage to bosses. Boost power is limited to a meter on the HUD. When the meter is depleted, the boost feature will no longer be available. Collect orange orbs to refill the boost meter.

[LOCK-ON TARGETING]

Using the Oculus Rift HMD, certain objects can be targeted simply by looking directly at them. When locked on, a targeting reticle will appear around the object, and the drone will notify the pilot of it’s locked on status via a sound effect. The following objects are currently targetable:

* Turrets
* Certain portions of Bosses

[HOMING MISSILES]

Once locked on to a target, press “B” to fire a homing missile of charged energy. The drone can charge an infinite number of missiles, but they can only be fired while locked on to a target.

[The NAUTILUS]

The Nautilus is the pilot’s vessel, used to explore the Reach, and it serves as the starfighter drone’s dock. Since sectors of the Reach are currently too dangerous to send humans to, the Alliance has chosen to deploy these starfighter drones to eradicate hostile alien life in the area, making the Reach safe for colonization.

[Ethereal]

Ethereal is the A.I. companion on board the Nautilus. He is your psychic link to your starfighter drone, Aurgus, and will be your sole source of information concerning mission objectives and status. Listen to what he has to say, as the voice of Ethereal provides valuable insight into playing the game!

[Galaxy Map]

Activate the Galaxy Map by pressing “A” when in range of the pedestal. A black hole will appear to let you know you are in range. Use the Galaxy Map to select the system you wish to explore. Choosing a system will notify your intent to Ethereal, who will open a portal in the staging area. Entering the portal will link your mind to his, and together you will launch the Aurgus to the chosen planetary system.

* You can hit “B” at any time to close the Galaxy Map

[ONLINE LEADERBOARDS]

View the Leaderboards at the official website

Leaderboards are separated by each level, and high scores are level-independent. Pilots will be prompted at the status screen to enter a name if their score beats one of the level’s Universal High Scores.

* Only ten (10) Universal High Scores are allowed per level.
* Names can only be 15 characters long.

[LEVEL DETAILS]

Level Name (Music Track) [Boss Name]

0 – Main Menu (Song For You, Milkman)
1 – Lumoria Nebula (Thunder Muscle) [Rex’Aern]
2 – Celestial Plains (Lightstabber) [Celestial Archon]
3 – The Spiral Downs (House-Step) [The Watcher]
4 – Coral Dreamscape (Power Moves!!! v2)
5 – The Drudgeon Edge (This One’s For You)
6 – Cirrus Straight (Aphrodisia)
7 – Visceral Downfall (Hocus Pocus)
8 – Centurion Dust Fields (It’s a Mega Dub Style)
9 – The Shinkansen Spiral (They Come)
10 – Omega Terminal (Fatima’s Funk)

LEVEL LAYOUT AND ENEMY LIST

1: (Ra’Aern)/ Turret_01 – Boss 01 – Aern – (Easy Boss)
2: (Ra’Aern)/ Turret_01 – Boss 02 – Celestial Archon – (Easy Boss+1)
3: (Disarray Probe)/Turret_02 -Boss 03 – The Watcher – (Easy Boss+1)
4: (Disarray Probe)/Turret_02 -Boss 04 – Harbinger of Chaos – (Intermediate Boss)
5: (Centipede Minion)/Turret_03 -Boss 05 – Wyvern – (Intermediate Boss)
6: (Centipede Minion)/Turret_03 -Boss06 – Dragon Worm – (Hard Boss)
7: (All minions + turrets) -Boss07- The Hammer – (Very Tough)
8: (All minions + turrets) -Boss08- Karasue – (Very Tough +1)
9: (All minions + turrets) -Boss09- Guardian of The Reach – (Incredibly Tough)
10:THE GAUNTLET – CONCURRENT BOSS FIGHTS ( 02,04,06,07,08,09) 10 – Omega – (Impossible to Gauge)

Read More

#IndieDEV Podcast Episode 05

October 15, 2015
/ / /

Fifth episode of the in and out collaboration efforts of Ron, Scott, and important passer-byers. This week we are live from SIEGEcon 2015 talking about winning in the game industry, what you missed, and more…

-Hosted by The Indie Cluster and sponsored by Soverance Studios-

Read More

Soverance Studios – Alpharetta Office

May 30, 2015
/ / /

Oh man! It’s been nearly two months since I’ve made a post, and though that makes it look like we’re not doing anything… it couldn’t be further from the truth!

I’m excited to finally announce the establishment of our first office facility, located in Alpharetta, Georgia, just a few miles north of Atlanta. This is a huge opportunity for Soverance Studios, as we’ve never had an office to work out of before. As I’ve been the sole proprietor of Soverance Studios since it’s founding, I’ve always simply run the business out of my home. I’m excited to say that both of those details no longer apply!

twitter15Along with our new office, I am extremely pleased to announce that Jacob Corle has signed on to help me achieve my vision of what Soverance Studios can bring to virtual reality. A graduate of the Art Institute of Atlanta, Jacob’s passion for gaming and technology, along with his skill in game programming, make him a formidable addition to start the Soverance team with.

Our new place is 550 square feet of industrial zoned warehouse space, provided to us by Unique Sports Products. In exchange for my services with network administration and Epicor ERP implementation, the good folks at Unique are allowing us to use this space free of charge. That’s right, we don’t have to pay a dime for our rent or utilities! We definitely got lucky there.

It’s a great, symbiotic relationship we have with Unique Sports. We provide them with solid and secure network infrastructure and maintenance for the entire building, including the deployment and management of their ERP solutions, as well as 24 hour support. In exchange, we’re provided with the time and the facility to build our own products, without the pressure of having to seek funding. In addition, I believe we’ll be able to integrate our technology into their product line, creating all new sports simulators for virtual reality. Unique has 40 years worth of experience in the industry, and has some major brand power that we’d be able to leverage with our new approaches to virtual reality content. Not to mention they’ve already got the channels to sell such a product to nearly all of the largest sporting goods retailers on the planet.

Our main Workstation setup - Jake's desk and our MediaServer.

Our main Workstation setup – Jake’s desk and our MediaServer.

Now that we’ve got ourselves settled into the office and begun getting into our groove, we’ve also had the time to team up with some local Atlanta indie talent and get recording a brand new podcast, #IndieDEV (pronounced “hashtag indie dev”). Brainchild of Ron Jones, the podcast is still in it’s infancy, only two episodes in, but they’ve both been a real joy to be a part of and I’m glad we had the opportunity to host both of them at our new facility. We had some excellent conversations about being an indie in the local Atlanta game dev scene, and got to chat a bit about some really cool games with really cool people. You can check out our latest episode below. We’re hoping to make this a more regular thing, so check back often!

I’m super excited about what’s to come for Soverance Studios. I hope you are as well! If you’d like to keep up with what we’re doing, we’d love to have you subscribe to our newsletter! You can always follow me on Twitter @Soverance, as well. ‘Til next time!

Read More

The Future of Endless Reach

March 13, 2015
/ / /

oc_7

Recently I’ve been feeling the need to get back on developing Endless Reach, as things have kind of stalled on this project while I’ve been developing a map for Unreal Tournament. I have a lot of love for Endless Reach, and I do believe it can be a really innovative game; helping bring the shoot ’em up genre to VR.

I believe the future of Soverance Studios will be built with Unreal Engine, and all of my projects going forward will utilize that software. Despite that vision, I still prototyped Endless Reach on mobile with Unity 4, and developed the VR Beta as we all know it today using the Unity engine. It’d be a fair amount of work to rebuild in Unreal, but I am up to the challenge and believe it would be a worthwhile investment.

Coral Dreamscape Gameplay (Level 4)

Coral Dreamscape Gameplay (Level 4)

With that in mind, I’d also like to begin adding support to the game for the recently announced SteamVR headset, as well as support Samsung’s GearVR. This will obviously cost money to obtain the SDKs and dev kits, money that I do not necessarily have at this time.

To help fund this vision, I’m mulling around the idea of launching a Kickstarter campaign for Endless Reach. It would likely be asking for a very small amount, maybe $2500 or so, to increase the campaign’s chance of success. All of this funding would go directly to the game’s development and the purchase of new hardware (specifically, SteamVR dev kits) in order to release for those devices. This Kickstarter campaign would also coincide with a reintroduction of the game on Steam Greenlight, after last year’s unsuccessful attempt.

I’m well aware that Endless Reach has never had the best following, which could make some difficulty for a successful Kickstarter. A few months ago in the peak of marketing Endless Reach, I managed to get a few thousand people to download and try the game, but most of the feedback I received told me it simply wasn’t that fun or intuitive, and I feel that I can do much better.

Nautilus Cryo Room - The "hub" of Endless Reach.

Nautilus Cryo Room – The “hub” of Endless Reach.

I still hope to make my vision of Endless Reach into a reality, and I am now acutely aware the current VR beta does not do justice to that vision. I hope to make it the best game I can, bringing a beloved genre to virtual reality and providing many hours of shoot ’em up fun.

Please, if there’s any interest in seeing an improved version of Endless Reach, I’d love to hear your thoughts! If you’d contribute to a Kickstarter campaign, it’d love to know that too! Show some love for the game, and I’ll make sure you get the best shoot ’em up VR has to offer.  If you’re a developer and would like to help realize a VR shmup, then definitely send me an email! [scott.mccutchen@soverance.com]  Any feedback you may have can be directed to the Endless Reach subreddit, or follow me on Twitter @Soverance.

Read More