Welease Wodewick!!!

Things have appeared quiet over the last few weeks, but in reality I have been hard at work in a special kind of hell normally reserved for Dev-Ops type people.

I was going to be using my time fixing bugs and getting Ancient Armies ready for the next stage of development. However, I kind of got myself side-tracked….

I had always known that my development environment lacked one very crucial piece of functionality…

The ability to create reproducible and traceable builds along with their associated releases.

Up until now, if I wanted a release, I would fire up Visual Studio and hit build. Next I would run the installer for Ancient Armies, then away I went.

However, this process had a number of issues:

  • I could not reliably reproduce an exact build or release from the past.
  • I didn’t really know what code changes were in my releases.
  • Ancient Armies itself lacked any formal version numbering – which exacerbated point number 2 above.

All of these issues are critical. If one does not know which code changes are in a particular version of the game, then one cannot test the game.

To resolve the above, I decided that I would create an automated build and release management process. After a little research and a personal recommendation, I decided to integrate a tool called TeamCity into my development environment.

What does TeamCity actually do?

A build from within the TeamCity tool.

A build from within the TeamCity tool.

It allows me to configure and manage the build and release process. With it, I can build a particular version of Ancient Armies, automatically obfuscate the binaries, create installers, run automated tests and then deploy the installers to my NAS system!

That’s a bit of a mouthful!

The best part is that it all happens automatically the minute I commit anything to my remote master. It’s like magic! Write some code, commit it, wait around a minute or two, then miraculously I have a new build directory created on my NAS drive complete with an obfuscated and unobfuscated installer – all this without lifting a finger! 😎

Having the ability to be able to instantly recreate any version of Ancient Armies cannot be underestimated and will be especially useful during the Beta phase of development.

TeamCity also integrates exceedingly well with my JIRA issue tracking system:

The TeamCity dashboard in JIRA.

The TeamCity dashboard in JIRA.

In the above screenshot I can instantly see that I have built 20 versions of Ancient Armies, of which the last two builds went well. I can even click on the individual blocks to get detailed information with regard to each build.

The key principal with these builds and releases is that they are versioned. In the above screenshot, the current version is build number 20. This allows traceability through issues, code and the final finished components.

TeamCity also integrates tightly with JIRA issues:

The new Team City fields in Ancient Armies' JIRA issues.

The new Team City fields within Ancient Armies’ JIRA issues.

In this case, I can see that this fix (AA-230) went into build number versions 10 or later. This means that I now know precisely which versions of Ancient Armies have this fix in! And more importantly, which versions do not!

The new functionality also allows me to kick off a build from here, or to initiate one when the issue is closed or resolved.

All of this was simply not possible before.

I can even view the specific builds that incorporate code changes from the issue being viewed:

The TeamCity tab shows all the builds that incorporate changes from the parent issue.

The TeamCity tab of the JIRA issue shows all the builds that incorporate changes for that issue.

The versioning created by the build process automatically flows through to the installers and then on to the final components. The screenshot below shows the file properties for the Ancient Armies game executable:

File properties now display the file version number from the build.

File properties of the game components now display the file version number from the build.

From the screenshot above I can instantly see that this version of the game was created by build number 20!

Prior to the introduction of this release system, I would have had no idea which version this executable was, or which fixes were incorporated into it – all versions of the game were statically fixed to version 1.0.0!

In addition, the game itself is now aware of its new auto-generated version number:

The about window from the Ancient Armies game.

The about window from the Ancient Armies game.

Just by viewing the About Window for any part of Ancient Armies I can immediately see the dynamic version number for that component. In this case it originated from build number 20.

When it comes to testing, any issues found can now be reported against a very specific version number. This number can then be used to determine which changes went into that release so as to target any diagnostic work.

This all sounds like really good stuff, but there was a rather large fly in the ointment…

It turned out that the installer technology that I was using for my game could not be built by MSBuild. As a result it could not be incorporated into the build and release process. This is a huge flaw. It’s fine having an automated build and release process, but if you cannot build and release the installers for the game, you are pretty much wasting your time!

I was a little shocked. I was using Microsoft technology in my installer, so the last thing I expected to be doing was re-writing it due to future incompatibilities.

There was a hint of trouble ahead from within Visual Studio 2015 itself. Its complete lack of native support for the installer technology that I was using should have rang a few alarm bells…

I searched high and low for a replacement technology. I had found that most commercial installers can cost upwards of £500 and were usually priced in the thousands! Far too rich for my pocket.

In the end I did find an open source technology, one that is even used by Microsoft itself: The WiX Toolset.

This toolset is a double edged sword. It is exceedingly powerful – much more powerful than the technology that I was using, but alas very complex and lacking any kind of wizard or UI design tools. Everything is done by creating low level XML code  – not exactly very friendly!

I found myself continuously banging my head on the wall trying to bend the WiX Toolset to my needs. Even the simplest of tasks devoured a disproportionate amount of my time. In addition, the whole toolset follows a multitude of hidden and non-obvious rules, all of which are set to trip-up the unwary.

I think it would be fair to say that the WiX learning curve was exceedingly steep and incredibly precipitous. In retrospect I might have picked a simpler toolset like Inno, but alas, we are where we are…

With much persistence and having written nearly 400 lines of XML code, I now have a world class installer for Ancient Armies. More importantly, this installer can now be built from the command line, which means that it can be incorporated into the automated build process.

Over the next few screenshots I will take you through the differences between the old and the new installers. The old one is on the left, whilst the new one is on the right.

First up, the introduction page:

The introduction page from both installers.

The introduction page from both installers. (Click for larger version)

Straight away one can see that the new installer on the right looks much more like a standard installer. In addition, and rather importantly, it automatically picks up the build version number from the build process. Here we can see that it is the installer for version 1.0.0.20 of Ancient Armies. This build labelling was simply not possible with the older installer on the left.

Incidentally, the rather alarming legalese text at the bottom of the old installer is not mine, it’s automatically incorporated by the installer itself with no option to turn it off.

Next up is the licensing page:

The licensing page from both installers.

The licensing page from both installers. (Click for a larger version)

There isn’t much of a difference here other than the ability to print the license agreement on the new one. The new one’s title bar also indicates the specific version of Ancient Armies that is to be installed.

Clicking the next button results in the options screen:

The options page from both installers.

The options page from both installers. (Click for a larger version)

Here the differences are very pronounced. The new installer on the right is much more advanced in that it allows one to pick exactly which components are to be installed. In this case, the Main Game (not optional), the Editors and/or Telematics.

The older installer is an all or nothing affair – you cannot choose your components.

Clicking the next button takes you to the pre-install screen:

The confirm installation page from both installers.

The confirm installation page from both installers. (Click for a larger version)

Here, both installers are very similar, though the new one on the right makes it absolutely clear that you are about to perform an install and it warns you that Admin privileges will be requested by the icon shown on the button.

Clicking next takes one to the installation itself:

The progress page of both installers.

The progress page of both installers. (Click for a larger version)

The new installer on the right wins again. Its progress information not only shows a progress bar (not currently visible), but it will also tell you what it is doing at any particular time. Alas, it runs so quickly that I couldn’t capture this. In addition, its progress bar takes third party installations like Direct-X into account.

The old installer on the left provides no real feedback other than progress and even this is flawed as it always hangs at the end whilst Direct-X and other third party software is being installed.

Once the installations are completed one is taken to the final screen:

The final page from both installers.

The final page from both installers.

Note that the new installer on the right tells you precisely which version of Ancient Armies you have just installed. This is simply not possible with the older one on the left.

In addition, if one looks at the Windows Applications Management Window, one is now presented with the version number there too:

Windows Applications now shows the correct and dynamic version number of the game.

Windows Applications now show the correct and dynamic version number of the game.

As a result one is never in doubt as to precisely which version of the game is installed.

Writing the new installer using the WiX toolset took an inordinate amount of time and effort. Despite the frustrations, I think the end result was worth it.

I now have a modern contemporary installer that can pretty much do anything.

However, the real story here is that I can now produce automated builds and releases from end to end with dynamic version numbers incorporated into all the finished components. This new capability should not be underestimated!

Had the old installer technology worked with the build process, the whole build and release process would have been configured in a matter of hours. Instead it took me around two weeks of continuous tweaking, two weeks that could have been spent coding.

This is the one downside of home development, one has to wear many hats and be multi-skilled. These additional hats can and often do take one away from the core software development effort. The only silver lining, is that from a professional standpoint it does make me a well rounded developer.

Whilst I’ll look back at this period of development work as not particularly pleasant, at least it is now done and dusted. Time to look forward to some real game development!

Laters

RobP

The Cores Cannea Take No More!

I’ve been away for a while as I have been rather distracted…

Do you know what it is yet?

Do you know what it is yet?

For those that don’t know, these are Virtual Reality (VR) goggles – in this case the HTC Vive. The more observant will just be able to discern the Star Trek Enterprise’s holo-deck!

I firmly believe that VR is the future. This is first generation tech, yet it puts one firmly inside the games and other realities that one plays. In the case above, I actually get placed inside of the holo-deck and it all just looks real – like you are really there! The feeling of presence is immense.

These kind of technical revolutions only happen once or twice a life-time, hence why I have been rather distracted by the whole thing. VR is quite literally a game changer and will eventually affect all aspects of life – its applications are not just limited to gaming…

Anyways onto Ancient Armies. What have I been up to? The image below provides the first clue:

Can you spot the subtle difference?

Can you spot the subtle difference?

For those of a technical persuasion you may have noticed the Commit hashes (6bb0638 and 6815d6e). What do these mean?

In my case, it is an outward indicator that my backend source control system has changed!

Once upon a time I used to use Perforce, but I have now switched to using GIT.

The old and the new!

The old and the new!

GIT has many advantages, but here are a few:

  • Faster, a lot faster…
  • Allows me to easily work offline – especially useful for my laptop.
  • Each machine has a copy of the repository, thus reducing the risk of losing both the source code and history.
  • Simple and easy branching – this makes coding experimental features a lot safer.
  • The Master repository is on an always-on fully raided NAS drive.
  • Switching between machines is very easy unlike the error prone Perforce.
  • Better Visual Studio integration.

The move involved a little juggling as I wanted to retain all my source code history. However, thanks to the wonders of the internet I have managed to achieve just that!

In terms of changes to Ancient Armies itself, the Line of Sight (LoS) system got a major overhaul, as have a few other subsystems.

Why the overhaul?

The original LoS system was designed to run asynchronously in the background whilst the game was running. This system ran very well where a game turn had enough real time in it to allow the LoS system to perform all its calculations at a high enough frequency.

However, I ran into three issues with this system:

  1. Not very scalable. The more units on the field, the more time it required to run its calculations. Given that a turn duration was more or less fixed, this could mean that the frequency of updates to the various units’ LoS would decrease as the unit count increased.
  2. Time compression reduced its effectiveness. The time compression feature can actually reduce the amount of time that a game turn plays out in. This impacts how many LoS calculations that the system can carry out. The higher the compression, the smaller the number of calculations run and hence the lower the fidelity.
  3. Chronos. The arrival of the new Chronos subsystem meant that all game calculations are now carried out prior to a turn. These calculations are relatively quick, which meant that even for a 1:1 time compression, the LoS Calculations never quite got  enough time to perform as often as was needed.

LoS is complicated in Ancient Armies. This is for a variety of reasons, but includes such things as the lack of coarse hexes that other games have, uniquely and variably sized units and also terrain that supported curves.

To its credit, the LoS system has been highly optimised over time, so for what it actually does it is relatively quick. But alas, for our needs, it was not quick enough.

How to get around this?

I decided to take a two pronged approach:

  1. Firstly, I would alter the code to auto-detect and fully take advantage of all the processor cores that are available on a user’s PC.
  2. Secondly, I would instigate an auto-scaling algorithm that would automatically reduce or increase the fidelity of the LoS system based on the user’s machine’s capabilities and the size of the battle being played out.

Of the two approaches, the first was by far the most difficult. My code would need to be able to run in multiple threads, in parallel, on all the cores of the user’s CPU.

Achieving this whilst maintaining data integrity and synchronisation across all the processes turned out to be a non trivial task.

Luckily, the .net platform can help out here, but alas, you need to be on at least version 4.0 to take advantage of the facilities on offer. However, up until now I could never get Ancient Armies to work on .net 4.0…

One day of hard graft later and I can report that Ancient Armies is now running against version 4.61 – the latest available to me 😎

In all, I have been working solidly all weekend to get this new and improved system up and running. Here are the fruits of my labour:

This is Ancient Armies on my gaming machine. As you can see, it successfully detected all 8 cores and spread the load between them! :)

This is Ancient Armies on my gaming machine. As you can see, it successfully detected all 8 cores and spread the load between them! (The peaks are from a turn’s LoS calculations) 🙂

It doesn’t look like much, but to have the ability to detect the number of processor cores and spread the work out amongst them took a lot of effort.

The upshot of these changes are that Ancient Armies can now fully take advantage of modern machine hardware architectures and will scale itself appropriately.

Even in terms of LoS calculations, the system can now detect how well it is doing (or not) and automatically scale the fidelity of the LoS system based on the hardware and battle size.

This means that the system will now always be quick, regardless of hardware configuration. The new system will also work well across all time compressions and battle sizes too! In fact the system has never been so fast or so accurate! 🙂

To provide further options for the user, I have also added an options screen where they can, if they wish, override the automatic settings for the LoS System:

The new Line Of Sight options screen.

The new Line Of Sight options screen.

The new options screen also provides plenty of information for each setting so that the user is empowered to tweak their systems to their liking. That said, the Auto mode is so good, that for most people, I would just leave it there!

That’s it for this week!

Hopefully VR won’t prove to be too distracting….

Laters

RobP

The Development Environment!

Ancient Armies - now in the new and improved .net 3.5!!!

Ancient Armies – now in the new and improved .net 3.5 framework!!! (Click for full-sized image)

This week saw a fair bit of work, of which the biggest was the upgrade to my development tools and environment.

Sorry for the fairly techie nature of this post, but maybe it will help other home developers with picking the tools that they need for some serious home development! 🙂

Ancient Armies used to be coded using Visual Studio 2005, with the game targeting the rather antiquated .net 2.0 framework. Both of these are very old and lack many features that would really aid Ancient Armies development going forward.

In terms of tech, the key thing that I wanted access to was a Microsoft technology called LINQ that can only be found in .net 3.0 and later. Alas, this version of .net is not supported by Visual Studio 2005 (it is simply too old).

The primary catalyst that drove forward the upgrades to my tooling was the release of the rather excellent free development environment from Microsoft: Visual Studio 2015 – Community Edition. This edition is far richer and far more capable than my old Visual Studio 2005 environment. Plus, it also gives me access to the .net 4.6 frameworks and all earlier versions down to 2.0.

I decided that whilst upgrading the development environment, I might as well upgrade my entire toolset – including my source control system from Perforce.

This latter tool is very important as it looks after my source code and keeps it safe. My version of Perforce was very old, so it would be quite a jump to upgrade it to the much more modern Perforce Helix.

For me, the upgrade to the new source control tool was by far the highest risk. A fair bit of time was spent backing stuff up to make sure that I had somewhere to go if things went wrong. Luckily, in the event, the upgrades went very smoothly.

So what tools do I use in my development environment? And how are they connected?

My fully integrated development environment!

My fully integrated development environment!

The above diagram shows the tools that I use. They are all free or very, very cheap 🙂 I will provide links later.

This system is extremely powerful and is a full end-to-end solution from requirements and documentation, through to task management and then ultimately onto the source code.

Having an integrated system like this enables me to see how much progress I have made and to keep track of this progress, without really having to do much in the way of extra work.

It also helps me manage the project and keep track of any issues or design decisions.

The best thing about the above tooling is that it is all scalable. A very handy feature, should Ancient Armies sell well!

Here is the detail on the tools that are used:

Microsoft Visual Studio 2015 Community Edition

Visual Studio 2015! - (Community Edition) - This is where I do all my coding!

Visual Studio 2015! – (Community Edition) – This is where I do all my coding! 

https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx

This development environment is completely free and can be used for commercial use, provided that there is only one of you. So in my case, it fits my purposes very well indeed.

I cannot begin to describe what an improvement this is over my rather old Visual Studio 2005 system. The result of this upgrade should be increased productivity and the provision of access to much more modern technology.

Perforce Helix

Perforce Helix! - My source code configuration management system.

Perforce Helix! – My source code configuration management system.

http://www.perforce.com/downloads/helix

Source control is one of the most important aspects of software development. You want to be able to work on your code from multiple places, as well as being able to compare, rollback and submit changes to that code.

In addition, the source control system keeps your code nice and safe. This is very important as your code is an asset that can quite literally represent years of invested effort!

Without a source control system, development would be very difficult indeed.

This tool is available for free for teams with up to 20 people. That plus the fact that it is much better than SVN and suits my development processes much better than GIT, makes it a no-brainer.

Atlassian FishEye

FishEye! - My source code integration and management system.

FishEye! – My source code integration and management system.

https://www.atlassian.com/software/fisheye/overview

FishEye forms the bridge between Perforce and JIRA. It allows me to check in source code within Visual Studio and have that code automatically allocated to a specific JIRA task.

In addition it provides many more source code forensic tools, which enhance and supplement those available in Perforce.

The cost for this system is $10 usd for one year of support for 10 users. You can still use it after that year, but it cannot be reinstalled, nor upgraded if outside of the period. $10 a year seems like a good price to me 🙂

Atlassian JIRA

JIRA! - My task management and bug tracking system!

JIRA! – My task management and bug tracking system!

https://www.atlassian.com/software/jira

JIRA forms the hub of my efforts. Without it, I would have no idea what tasks I currently have to do, nor would I know how far through those tasks I am.

This sounds like an optional tool, but when one is working on a large project like Ancient Armies, one needs a means to log tasks and bugs so that they are not forgotten and to enable them to be planned properly.

JIRA also provides me with estimation metrics, both actual and predicted as well as providing direct linkages to my requirements, documentation and source code.

The screenshot below shows the actual source code changes related to the above task. From here I can even view the code changes themselves – a very handy feature when one is investigating previous changes to the system.

Source code integration with JIRA!

Source code integration with JIRA!

The cost of this system is exactly the same as for Fisheye. An absolute steal in my opinion!

Atlassian Confluence

Confluence! - My document and requirements management system. This is a view of my design document for Ancient Armies' new directory structure.

Confluence! – My document and requirements management system. This is a view of my design document for Ancient Armies’ new directory structure (I’ll discuss this in a later post – honest!).

https://www.atlassian.com/software/confluence

Confluence is my documentation and requirements tracking system. All project documents are stored here and are ‘live’, in that they can be linked to various JIRA tasks and issues.

Having all of one’s document in one place cannot be underestimated. Having the ability to link them to JIRA tasks and issues is just priceless. For example, one could create a requirements document and link each requirement to a JIRA issue. As the issues are worked on, the document updates itself in real-time to show the current progress. Overall, pretty awesome!

A screenshot of my requirements from very early on in the project. Each requirement is connected to a JIRA issue.

A screenshot from Confluence of my mapping requirements from very early on in the project. Each requirement is connected to a JIRA issue. This enables me to instantly see the state of the project.

Anyways, that’s it for the Ancient Armies development environment tour.

As a result of these upgrades, I am now in a much stronger position. Ancient Armies now has access to much more modern tech, which should increase its shelf life, plus I have access to many programming constructs that were simply not available to me prior to the upgrade!

That’s it for this week.

Laters

RobP