Playtesting and Analytics – When to Pivot and Why

One lesson I continue to learn the more I make games is that the idea you begin with is only an arbitrary starting position, and what your product becomes will be so different to its original incarnation, for good reason. A game without feedback and iteration is like Sherlock without Watson – capable, but missing the connections it needs to succeed. Testing and iteration is essential in the game dev equation, because you can’t accommodate for every player right off the bat. Great games take time and attention to play testers, and cleverly observing the behaviour of your players will empower you to tighten your design and experience in the right ways

Controls were easily the first and most contentious thing about our first and second playlists. Initially, the lack of item rotation was problematic for many people, or the ability to return objects that were stuck after you picked them up. Then, when we opened up the play testing to more people, some outright couldn’t play our game because they couldn’t invert the Y axis – lesson learned for next time.

The next issue was that we weren’t communicating the controls to the player. Many players had no idea that scrolling the mouse (when we used Keyboard controls) would rotate the object. This spoke to a larger issue that we didn’t communicate our controls at all. Playtesters had no idea these mechanics existed and actually asked for them to be implemented next time. The clear remedy here was to implement a HUD system that would contextually display the relevant controls. For example, picking up an item will display placement, return and rotation controls.  After explaining the controls through this UI, players understood how to manipulate the objects in the ways they were expecting, and we never encountered the same problem anymore.

While that feedback was easy to pivot on and implement, we didn’t get to fix everything that wasn’t great with our game. My lecturer recommended that to really hone in on our intended aesthetic, we should add a packing system, where the objects you don’t pack won’t move with you between stages. I wasn’t sure about this until I played our game during testing. Unfortunately, implementing both mechanics was too much for our team with the time we had, but to make sure that people understood the kind of message our game was trying to send, we added an age at the beginning of each of the levels to communicate the chronology of each of the houses. While not perfect, I think it is a good compromise.

We also implemented Unity’s Analytics Services into our game. We used the events to extract playtime and per-stage playtime, as well as which objects the player selected first, how many objects they selected, and whether the player used particular tools within the game. With this information we can get a broad idea of how long people engaged with the game, whether they interacted with all available objects However, we found Unity’s Analytics would take way too long to send when the player quits the game, so the game would appear to hang (and during a playtest we just need people playing at all times), and the data would take too long to update on the website for us to check if it sent at all. Furthermore, Unity’s Analytics website is very unwieldily to use – trying to visualise the information that did send took too long to be usable, and custom event variables are difficult to derive enough information from to be effective without a way to visualise them. Some data is viewable in the Event Manager tab, but that information doesn’t carry times, dates, or other information for me to plot when this data was captured outside of the visualiser. Overall, Unity Analytics were problematic and didn’t help us improve the project at all, or in time to be effective. It’s a shame because this information would have provided insight into how long players spend working on their houses, whether players moved all of the available objects, which stages they played, and  what they did when they started the game. The analytics are useful in theory – and I will try it again in another project – but I’ll do so cautiously.

Like in every project, there are always more changes that could be made to improve the work you do, but not all of them are feasible with the time and budget you have. The key is pivoting your testing is to find a solution that solves the problem within the restrictions you have, and making use of every avenue of feedback you have.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s