Category Archives: All

Spring 2014 Recap

In Spring 2014, my final semester at the University of Hawaiʻi at Mānoa, I completed my undergraduate project, worked as a teaching assistant for an introductory cybersecurity course, and completed my undergraduate degree.

1. ICS Security Assistant: ICS 425, Cybersecurity and Ethics I

As a teaching assistant to Professor Barbara Endicott-Popovsky of the University of Washington, I supported the teaching of the course by posting and grading assignments, responding to student concerns, and managing student peer evaluations and weekly discussions of current events in cybersecurity. I had taken this course one year ago in the Spring 2013 semester, which prepared me well to communicate the course material to students this year.

Meetings with the professor to plot the course of the lessons, appraise my performance, and address student concerns occurred once a week, usually by phone.

This was a distance learning course, so I did not have set office hours. All discussion and communication with students occurred through email or forums managed on Laulima, a collaboration system for the University of Hawaii that is built on the Sakai platform.

2. ICS Undergraduate Project, Part 3: Spring 2014

As in the previous semester, my responsibilities as a member of the Collaborative Software Development Laboratory were to maintain the documentation for the Ubuntu, CentOS, and Vagrant installation methods, and to maintain the Vagrant provisioning script and its supporting files.

Resolving Compatibility Issues for Makahiki Dependencies

The release of pip 1.5 on New Year’s Day 2014 temporarily broke the Vagrant provisioning script (the bash shell script mentioned in previous posts). Makahiki was, and as of August 2014 still is, dependent on several external, unverified Python packages, including the libraries services for Django, PIL, NewRelic, Sphinx, and Markdown.

After changing the provisioning script to use setuptools-0.8 for compatibility with pip 1.5.x, I ran through the Ubuntu virtual machine installation process several times to determine which pip-installed packages needed to be flagged with “allow-external” or –allow-unverified” to be installed. The pull request describing the proposed changes can be seen here.

The development team decided that it was simpler to remain on the last known compatible version, pip 1.4.1, so I reverted the changes in my code and changed the documentation to install that older version instead (see GitHub pull request 592). However, the modified pip commands were needed to pass the CSDL’s Travis CI testing, which built the software using the newest version of pip (see GitHub pull request 603).

In early January, the sequence of commands used to install the Python Software Collections Library (SCL) stopped installing Python 2.7.3, and stopped installing correctly on i386-architecture CentOS virtual machines. This was the result of the maintainer of the SCL switching to Python 2.7.5 and ending support for i386 architectures.

At the time, Makahiki on CentOS required Python 2.7.3. To enforce this version requirement, I revised the installation process to compile and install Python 2.7.3 from source code after installing development packages. This was based on my previous work on the now-deprecated Python installation scripts for Ubuntu and CentOS, which had also compiled and installed Python 2.7.3 from source code. These changes were applied to Makahiki in March 2014 (see GitHub issue 586).

Towards the end of January, I was also a Smart Grid Game administrator for the 2014 Kukui Cup, a sustainability and alternative energy educational game supported by the Makahiki and WattDepot software. As an administrator, I was responsible for evaluating and scoring submissions for the activities and questions which students completed and submitted through the Makahiki Smart Grid Game module.

In late February, we were able to remove the VirtualBox version warning from the documentation after tests with Virtualbox 4.3.8 demonstrated that the bugs caused by VirtualBox 4.3.0 had been fixed (see GitHub pull request 603).

Though some work was done on the alternate configuration user interface, I spent March through May working on completing the data analysis and undergraduate thesis writeup for my Honors Upper Division project.

Conclusion of the Honors Upper Division Project

My undergraduate research was presented at the 2014 Spring Symposium, held on May 8, 2014. The undergraduate thesis was submitted to the Computer Science department and the Honors Department. Links to full versions of each document are provided below:

The main insight gained from the usability testing was that the Vagrant installation method that I had developed represented a significant time savings over the manual installation method, cutting over an hour from the average setup and configuration time. This was accomplished by using a single, default Ubuntu 12.04 LTS virtual machine as the environment for Makahiki, which enabled the installation of software packages and replacement of configuration files to be automated in Vagrant. This reduced the need for OS-specific installation steps, since all users of a Vagrant installation would be installing Makahiki onto the same virtual machine. The usability testing also helped to indicate areas of the documentation which were difficult to understand or likely to cause user error, and these areas were cleaned up after the study had been finished.

I received my B.S. in Computer Science with Honors from the University of Hawaiʻi at Mānoa, graduating on May 17.

3. The George Washington University

After a successful application to The George Washington University’s Cybersecurity in Computer Science master’s degree program, I am preparing to begin my studies at the end of this month. I was fortunate to be awarded a CyberCorps scholarship, and I am looking forward to experiencing the unique opportunities for academic and professional development that are available at the university.

My Undergraduate Project So Far, Part 2: Fall 2013

This post documents the progress I have made on my undergraduate project over the last four months, with Dr. Philip Johnson of the Collaborative Software Development Laboratory as my advisor. It is a continuation of Part 1.

September 2013: WattDepot Backup Script

September was spent working on a shell script and cron job to back up the WattDepot database, and documenting the process for restoring it to a new WattDepot installation. The script (viewable here) dumped WattDepot’s local PostgreSQL database, created a monthly backup of it if it was the first day of the month, created a daily backup, and deleted any backups more than seven days old.

During September, I also worked on the first drafts of the usability survey that would accompany my usability tests. The study had been redesigned to center around Vagrant and another usability enhancement by the Collaborative Software Development Laboratory’s Carleton Moore, called the Smart Grid Game Designer (SGG Designer).

October 2013: Vagrant Maintenance Updates and Experiment Redesigns

In October I discovered that VirtualBox 4.3.* versions were incompatible with the Vagrant virtual machine that I had been using because they had been packaged with an older version of the VirtualBox Guest Additions (4.2.0). The fix I implemented required the user to install the third-party vbguest plugin, which automatically attempts to upgrade all Vagrant machines to the version of the Guest Additions currently installed on their system. After this was complete, the user had to run the Vagrant installation script once again after editing some configuration files.

Since this solution was more complicated in the short term than continuing to rely on the last compatible version of VirtualBox, documentation was added telling users not to use versions newer than 4.2.18. The documentation for Vagrant virtual machine configuration was also updated to reflect changes to the behavior of the vagrant up command since Vagrant 1.3.5.

In October, I also conducted informal “friends-and-family” testing of my study design. The original plan had been to have each subject complete the Vagrant installation process in one virtual machine, then configure a Makahiki competition using the Smart Grid Game Designer in a second machine. It took one user approximately two hours to get through the entire process, which led me to break the test into two separate tests, each one being taken by a different group of subjects. The first test would now cover the Vagrant setup process, and the second test would now cover the SGG Designer. Documentation for Vagrant was simplified and revised based on feedback that described which parts were too complicated.

Last of all, I implemented a partial fix for a Makahiki bug that allowed mixed-case usernames to be used even though their users could not use them to log in due to a Django bug. The patch changed the handling of bulk user uploads to automatically convert mixed-case usernames to lowercase.

October – November 2013: IRB Submission and Django 1.6 Upgrades

The second half of October saw a long back-and-forth dialogue with the university’s Institutional Review Board that eventually resulted in the exemption of my study from full review, even though the delays forced me to move the study to January. My UROP research funding was approved once the IRB approved my exempt status, but I am still waiting on the actual disbursement of funds.

In November, Yongwen Xu of the CSDL upgraded Makahiki to use Django 1.6. I uncovered and fixed some minor bugs related to Vagrant functionality, but otherwise my part of the project was mostly unaffected by the upgrade.

December 2013: User Interface Work Resumes

Towards the end of the semester, I was finally able to resume work on the new configuration interface I had originally planned to implement to improve Makahiki’s usability. Some basic features like user uploads and team and group editing have been implemented in the Makahiki widgets on this page, but round settings and challenge settings are still not included here.

Current state of the simplified configuration interface.

The current state of the simplified configuration interface. Only a few settings widgets are currently implemented.

In the next semester I will be conducting my study, completing my final research report and presentation for the Spring Symposium, hopefully finishing work on the configuration interface, and preparing to graduate with my B.S. in Computer Science.

My Undergraduate Project So Far, Part 1: Spring and Summer 2013

Over the last year, I have been working on an undergraduate research project for Dr. Philip Johnson, head of the Collaborative Software Development Laboratory (CSDL). This is the first part of an end-of-year recap of my current progress. Next semester I hope to maintain a more regular update schedule.

January – May 2013: Research Proposal

In the Spring 2013 semester, I completed a research proposal for usability testing of new features that I planned to implement in the Makahiki software and presented my work at the 2013 Spring Proposal Conference. The goal of the research was to develop and test additional functionality for Makahiki that made it easier for system administrators and developers to install and configure. In addition to a successful research proposal, I was approved for funding from the University of Hawaii’s Undergraduate Research Opportunities Program (UROP).

The initial research proposal produced for my Honors 495 research writing course created a 28-page report, a 15-minute presentation, and a 48-inch-by-36-inch poster. PDF versions of these files can be viewed below.

Spring proposal conference 2013 poster.

My poster for the 2013 Spring Proposal conference. Click to view a full-size image (3.56 MB).

June 2013: SGSEAM Conference Paper

My first task for the undergraduate project was to assist with the data analysis for a conference paper on the “Serious Game Stakeholder Experience Assessment Method” (SGSEAM) under development by Dr. Johnson’s Collaborative Software Development Laboratory. This paper was published in October 2013 as “SGSEAM: Assessing serious game frameworks from a stakeholder experience perspective” [citation].

One source of usability testing data for this paper was the surveys that my classmates and I had filled out in ICS 691 when we tested out the Makahiki administrative interface. Having been lucky enough to avoid some of the more extreme problems when I installed and set up Makahiki as a student, it was interesting to be able to take a higher-level look at the data in order to identify trends in what we had seen as usability problems.

July 2013: Script Development

An analysis of the problems that my classmates and I had reported when we installed local instances of Makahiki in Spring 2013 indicated that many of us had wanted the Makahiki installation process to be faster or more automated. Dr. Johnson suggested that this could be fixed by creating an automated script-based installation for Makahiki.

This shifted the focus away from the Makahiki serious game setup feature which was part of the initial proposal, but allowed me to work on a more feasible virtualization-based solution to problems with the initial setup of Makahiki.

Two solutions were eventually attempted: automated Python scripts for Red Hat Enterprise Linux / CentOS 6.4 systems and Ubuntu 12.04.* systems, and a Vagrant installation with a provisioning script. The initial plan was to support all of these methods.

The Ubuntu Python script implementation used a single master script to run other sub-scripts based on the command-line arguments that it was issued. The scripts installed Makahiki dependencies, initialized and updated the database, and cleaned up files downloaded during the installation. This turned out to be somewhat difficult to port over to CentOS due to the differences in the CentOS 6.4 base installation of Python (2.6.6) and the minimum version needed by Makahiki (2.7.3).

The solution that I eventually worked out was to create an additional script for CentOS that completed an altinstall of Python 2.7.3. With an altinstall, Makahiki would run as long as a Python 2.7.3 environment had been initialized in the shell that makahiki was running in. However, to initialize this environment, the user had to remember to run the command "scl enable python27 bash" manually in each new shell. In addition, sudo commands ignored the altinstall and ran as if they were in the original shell with Python 2.6.6.

During script development, I found that Ubuntu 12.04.* and CentOS 6.4 systems’ default repositories used the wrong version of libmemcached, so compiling and installing the correct version had to be built into the scripts as well.

Due to the version-specific commands and dependency issues, Dr. Johnson and the other CSDL members advised me to focus on the Vagrant installation, as the OS-specific scripts were becoming unwieldy and would likely be difficult to maintain in the future. The scripts were removed from the project, but I was able to add the CentOS installation process to the Makahiki local Unix installation documentation.

The Python scripts are no longer being maintained but can be viewed on GitHub.

The Vagrant solution involved a single Ubuntu 12.04 LTS virtual machine downloaded from Vagrant’s servers. This machine was provisioned using a shell script that installed the necessary packages. Unlike the Ubuntu and CentOS Python scripts, which had been intended for possible use on non-virtualized installations, the Vagrant script could also overwrite configuration files with Makahiki-specific ones: it would always be running on a virtual machine, and always be overwriting the exact same default files.

Installing Makahiki on a Vagrant virtual machine on Windows was simpler than the process of installing Makahiki directly onto a Windows machine, so my next task was to document the configuration of an appropriate development environment for a Windows host working with Makahiki on Vagrant.

August 2013: Vagrant and Eclipse

In August, I documented the process for using a Vagrant installation of Makahiki with an installation of Eclipse on the host machine. The resulting environment used the PyDev and Remote Systems Explorer plugins to enable Python and SSH connection support, and installed XML/HTML/Javascript editors to assist with the process of web development for Makahiki.

The PyDev remote debugging server.

The remote debugger works well enough, but I wonder if it is really worth bringing down your firewall.

The PyDev remote debugger was blocked by Windows Firewall even when a rule was explicitly created to allow its designated port to pass through, and only worked when the firewall was fully deactivated. The remote debugger instructions were added to the documentation, but with a warning that using the debugger would likely create a security risk.

As the Fall semester began, I shifted my focus away from direct Vagrant development and worked on getting my study proposal approved by the university’s Institutional Review Board.

This summary of the progress of my undergraduate project continues in Part 2.

Greyhats: Fall 2013 Recap

The Greyhats club is a Registered Independent Organization at the University of Hawaii at Manoa. It was founded in 2010 by five students who have since graduated. I have been a member of this club since Spring 2011, and have previously served as its Treasurer (Spring 2012) and Vice President (Fall 2012, Spring 2013).

The purpose of the Greyhats club is to educate its members in cybersecurity-related topics and to train for the annual Collegiate Cyberdefense Competition that happens every March. In the past, the Greyhats participated in the virtual regional, and I was on the CCDC 2012 team which placed 3rd. I was also a member of the CCDC 2013 team, which unfortunately did not place.

Though I prepared some basic Unix command tutorials this semester, I have stepped back from preparing instructional content for the club as my responsibilities for my undergraduate project have increased and I approach my target graduation date next semester. It is in the best interests of the club to have more active leadership that will be enrolled at the university for at least the next two semesters, so I did not run for President or Vice President. I am currently the Treasurer, a position which is largely ceremonial because the club did not do fundraising or apply for grants.

At the beginning of the semester, I configured a Google Groups forum for use as a discussion board for the club. This enabled us to move some discussions out of the mailing list to a format more suitable for longer conversations.

As in Spring 2013, I continued to serve as de facto Secretary by announcing meeting times, preparing meeting minutes, and distributing them to the mailing list. I also took up the management of the club’s new cloud file storage service account, uploading meeting minutes, tutorials, and presentations. I organized and uploaded documents from past CCDC events in 2011, 2012, and 2013 so that other members could use them to train for the 2014 CCDC. We have run into several situations this year where only one or two of us had some document that a former member had sent around. Creating an archive of tutorials and competition materials is a way for the club to try to preserve its institutional knowledge in a more organized way as members graduate.

Po’oihe Cyber Security Exercise

In July 2013, I attended planning meetings and contributed injects (competition exercises) for the Po’oihe Cyber Range’s first annual Cyber Security Exercise as a member of the White Team. The actual Po’oihe exercise took place from August 2-4, 2013 at the University of Hawaii at Manoa. The Blue Team played the role of the IT department for a Hawaii bank.

Po'oihe participation certificate.

Certificate of participation (click for large image).

Teams

The Cyber Security Exercise followed a structure of simulated cybersecurity exercises that divided competitors, support staff, and judges into five categories.

The Five Teams You Meet at a Cybersecurity Exercise

  • Blue Team
    A cybersecurity competition has multiple Blue Teams, each one made up of people from a specific company, school, or other organization. Pooihe had a mix of businesses, schools, and Armed Forces teams. Each team has a Team Captain and Team Co-Captain who were responsible for communicating with the White Team.
  • Red Team
    The Red Team is a team of cyber security professionals who work to disrupt the operation of the computers that the Blue Team is responsible for. Once the team gains unauthorized access to a system, they can add and delete users, deface web pages, bring down critical services, plant keyloggers and other listening tools, or whatever else is needed for the exercise. Once a Blue Team detects that the Red Team has gained access to their system, they are usually required to file an incident report with the White Team that describes the impact of the Red Team attack, the evidence for the attack, and any steps taken to mitigate the attack.
  • White Team
    The White Team serves as the judges for the event, scoring the performance of each Blue Team and enforcing the rules of the competition. At the Po’oihe exercise, the White Team also designed the exercises, or “injects” for the event.
  • Black Team
    The Black Team is in charge of setting up the competition environment (equipment, rooms, etc.) in advance of the competition.
  • Gold Team
    The Gold Team acts out the roles of the management and staff of the fictional company in all communications with the Blue Teams.

Injects

Each inject consists of a task (such as creating a security policy, setting up a web server, configuring a firewall, etc.), a time deadline, and a point value. Each inject is scored according to a separate document that defines how it will be scored. Injects are delivered to each Blue Team as printouts or by email at specific times during the competition. Each Blue Team’s total score on all injects is used to determine the overall winner of the exercise.

Injects are judged by the White Team during or after the competition. Teams will lose points for an inject if their results are incomplete, incorrect, or otherwise not sufficient to meet the criteria for the exercise. The overall score of a team for all of its submitted injects determines its ranking.

Further reading about the Po’oihe exercise

Developing a Serious Game Application Part 2 – A Project Postmortem

Project Status and Future Development Goals

The project’s Github project page can be found at jtakayama.github.io/ics691-setupbooster/.

a25_landing_page

Among other issues, there were problems replacing the logo.

In many respects, this project did not go as planned. Several tasks ran overtime, several other long-term projects competed for my attention, and even when modifications to Makahiki did partially work, it was a challenge to figure out where exactly some components of each page were being loaded from.

a25_new_theme_blue

Above: The new theme uses blue and gray to match the logo, and to go for a more technological-looking background. However, theme changes do not seem to apply to Settings pages.


In terms of cosmetic changes, I switched the default theme to a customized blue-gray-white theme, created a custom set of default teams, users, and profiles to be used to initialize the database, and reworked the landing page and About page to describe SetupBooster instead of the Kukui Cup.
A gray page.

Above: The new “About” page.


So far, I have been unable to implement the article-creation and article-editing workflows. Non-functional widgets for creating articles, viewing a user’s average rating over all articles, and viewing user status information were created and are located on the user profile page.

The article creation widget.

The article widget, currently not functional.


The article creation widget currently writes an entry to the database, but there is no function implemented to retrieve the entry, and it does not have a corresponding editable, static page file.
rendering_glitch
The average_rating and player_status widgets are currently rendering in the same position due to a bug resulting from a conflict with a default Makahiki widget of priority 3 that has been disabled; the average_rating widget has priority 3 and the player_status widget has priority 4.

The primary future implementation goal is getting the article model to function correctly, since it is the central piece of the site. Once article pages can be rendered correctly for viewing and editing, the rest of the modules should be easy to integrate. Assuming each article can be identified by a unique ID and a revision number, creating view_article and edit_article widgets and passing them some variables to retrieve page content should be relatively easy.

Lesson 1: Gamification Benefits From a Clear End Goal

An issue I ran into when designing content for my software guides site “game” was that a site that exists to host user-generated content potentially indefinitely can be difficult to gamify. Competitions like the Kukui Cup or gamified personal challenges like World Without Oil ask players to make commitments and carry out actions within a specific time period. This allows the game designer to structure the experience by introducing levels, scaling difficulty, multiple rounds of prize giveaways, and constructing a narrative (if applicable). Other gamified concepts (fitness goal monitoring sites, social sites that measure TV viewing) can work for an indefinite amount of time because they can be driven by the user’s desire to achieve something or earn recognition for something of personal value. The gamified system can provide structure and incentives for the user, but in those situations, the user sets the timetable.

The new user groupings.

In SetupBooster, the teams don’t really serve as a means of organizing competitors against each other: they just distinguish between different types of users. In theory this could be done without teams completely (since Makahiki has superusers and non-superusers) but using the teams functionality could allow for additional “teams,” like temporarily banned users or more limited kinds of administrators.

In the case of the wiki / document-sharing site I planned for SetupBooster to eventually become, the issue is that the wiki model runs indefinitely, but individual users don’t really have a personal stake in the site unless they become popular or become administrators – both things that are determined by the community, not the software. A training game might help people get over the initial hurdles of editing a wiki. However, beyond this point, having users define the content of the wiki makes it difficult to create defined categories of prizes or activities.

In this situation, I think that only thing the wiki can reward are generalized measurements of activity: article creation, adding editors, making lots of comments, editing frequently, and similar statistics. Absent some kind of external prize, the wiki game eventually will become a long grind like parody games Progress Quest or Achievement Unlocked, with new badges and recognitions doled out every so many completed tasks. Arguably the content of the wiki could be the major selling point if the writers are good enough, but if that is the case, than the gamification really isn’t making much difference beyond lowering the barrier to entry.

Lesson 2: Don’t Work Alone If You Can Help It

As I stated in earlier posts, I had originally planned to work alone because I thought the final project would be more similar to my upcoming undergraduate project. (In retrospect, knowing what I know now about the process of modifying Makahiki, that would probably have been even more difficult.) All conflicts over scheduling and creative vision aside, the main reason to work in teams if you can – especially for students with heavy courseloads, long work hours, or other responsibilities – is that having more people working on the project increases the odds that at least one person will be able to work on something on any given night. It was too often the case that I was unable to put in much work on the project for days at a time. The tendency in such cases is to prioritize the things that are due earlier, which resulted in a Github commit pattern that bunched up noticeably just before major deadlines.

Teammates, if they care about deadlines, will pester each other to get the project done on schedule: this was the case with another team project I was working on for a cybersecurity class last week, and I believe it to be true of functional team projects in general. For more complex software projects, working alone and being responsible for the entire design of the system makes planning more difficult than for team projects, where each person can take a piece of the structure and work on it. On the other hand, breaking up a project can cause implementation problems if the different pieces don’t have the same gameplay style, access shared data in different and incompatible ways, or just end up not working because one piece of the puzzle is missing or late. Though I was working alone, I could work knowing that, for better or worse, all the pieces of the system were my responsibility to modify and I didn’t need to meet with someone to get part of the problem fixed.

Lesson 3: Documenting Your Code Is Valuable (even if you don’t have to)

Though Github’s Project Pages feature turned out to be easy to pick up, the process of documenting the vague flowcharts and lists I had created in earlier phases of the project was not. Being forced to describe the content of the game in detail caused me to realize some potential limitations of applying gamification to a wiki-like site, and to try to think more carefully about what kinds of integration I might be able to create between the tutorial game and the wiki interface. More importantly, it probably resulted in me having more of a product to show for my efforts than if I had just continued being stuck on resolving lower-level issues with the article model. Though in both cases I did not accomplish most of my initial implementation goals, having documentation to provide a blueprint for future progress increases the odds that I might be able to continue working on the project or come back to it later, rather than abandoning it because I ultimately had no idea where it would be going after the course was finished. Though there were several issues I was unable to resolve, having documented the plan for the final application refocused me on the larger goals of the project.