Dec 282013
 

Now that we have the CS Department’s GitLab server set up, and CS-140 Lab 1 is rewritten and tested using the new server, I’ve started to think about how to automate my interactions with the server. I had already  written some Bash scripts to interact with the Bitbucket server to get student code, convert it to PDF files, and put it back on the server after grading. Those scripts should still work fine with GitLab, since it’s just git on a different server.

One thing that I had not been able to automate previously is the step of issuing a pull request for students to merge my grading branch into their repository. This was not too much of an issue when there were only 6 students in the summer class (so only 3 repositories per lab assignment), but it was going to take more time with ~48 students in the spring class. While reading RSS feeds, I came across a post mentioning the GitLab API. This could be the solution to my problems! And there’s a Python module for the API! I had already been writing Python scripts to make my grading easier, and had been starting to rewrite my Bash scripts in Python.

I started playing with the GitLab API in Python, and had managed to create a merge request (GitLab’s term for pull request.) I had also noticed that you could create GitLab accounts through the API. This seemed like something I should pursue – creating ~48 accounts per semester seemed like something that should be automated.

Since I intended to post my code on Github, one of the first issues I had to address is how to avoid publishing my private token for GitLab. I could have put in a dummy token before pushing my code, but I would have to remember to do that before every time I committed my code. The solution to this issue was solved through the use of the .gitignore file. If I put my token into a file, then I could add a line to my .gitignore file so that it would not be committed.

# Private GitLab Token - not to be stored in repository #
########################################################
gitlabtoken.txt

Then I could just read the token out of the file, and use that string.

# Get my private GitLab token
# stored in a file so that I can .gitignore the file
token = open('gitlabtoken.txt').readline().strip()

After importing the pyapi-gitlab module, I could use that token, along with the server’s URL to create a GitLab object. Notice, that I had to turn ssl verification off, since we only have a self-signed certificate.

# Create a GitLab object
# For our server, verify_ssl has to be False, since we have a self-signed certificate
git = gitlab.Gitlab(GITLAB_URL, token, verify_ssl=False)

Creating a user account is pretty simple using the API:

# Create the account  
success = git.createuser(name, username, password, email)

The returned success value is a boolean — either it worked, or it failed (but you can’t tell why…).

One thing that’s a bit odd about the createuser call, is that you have set a password for the user, but the notification email to the doesn’t include the password. (If you create a user account from the web interface, it generates a random password, includes it in the notification email to the user, and requires the user to change their password when first logging in.) And, the password you set doesn’t seem to work either!

So, I’m just telling the students that they should use the “Forgot Password” link to have a password reset email sent to them, and then proceed from there. (If this is ever fixed, I’ll have to generated a random password.)

Getting the class list as a CSV file from the Blackboard Grade Center is pretty easy, and the first three rows contain the student’s last name, first name, and username. I can use those three strings to generate the name, username, and email needed for the createuser API call.

The only challenge with processing the CSV file is that Blackboard puts some strange character at the beginning of the file, so the file has to be opened with utf-8 encoding. (And the header line needs to be thrown away.)

The last thing I wanted to add is a way to have optional verbose output, so that I could see if the user creation was working. (I decided that it should always notify the user if the account creation failed.)  To do this I had to learn two new things about Python: how to parse arguments1, and how to send output to stderr.

I used the argparse module:

import argparse
# Set up to parse arguments
parser = argparse.ArgumentParser()
parser.add_argument('filename', help='Blackboard CSV filename with user information')
parser.add_argument('-v', '--verbose', help='increase output verbosity', action='store_true')
args = parser.parse_args()

and used the verbose argument to determine what to print:

if not success:
    sys.stderr.write('Failed to create acccount for: '+name+ ', '+username+', '+email+'\n') 
elif args.verbose:
    sys.stderr.write('Created account for: '+name+', '+username+', '+email+'\n')

Full code is on Github here.

  1. I already knew how to do simple argument parsing, but I wanted to learn how to deal with optional arguments.
Dec 102013
 

Chad Day recently completed the installation of our new GitLab server (read about it here and here.) This project was precipitated by some issues I had been having in trying to teach the use of git earlier in our curriculum. I had been having the CS-401 Software Development Process students use git and github in their FOSS projects, but it was difficult for them seeing git for the first time and expecting them to use it intensively in a project in the same semester. They had asked a number of times, “Why don’t you teach this in an earlier course?”

So, I decided to try using it in the first programming course – CS-140 Introduction to Programming. While they don’t do any large projects in CS-140, they do work on their weekly labs as Pair Programming. Using git for the collaboration aspect (so they don’t have to keep emailing versions back-and-forth to each other) and as a way to submit their completed lab assignments to me (so I only have to receive one copy of the assignment per pair) seemed to make a lot of sense. In addition, I had attended a workshop at CCSCNE 2013 entitled “Git on the Cloud” which provided a methodology to do just that, which had encouraged me further.

The “Git on the Cloud” workshop suggested using Bitbucket, since it allows an unlimited number of private repositories.1 I’m very willing (in fact, I often require) students to make their code for senior-level projects public with an open source license. But private is important for coursework at the freshman level.

On the other hand, the “Git on the Cloud” methodology involved using a single repository per student, and a different branch for each assignment(!)  In other words, whenever you change branches/assignments, all of your other code goes away, and is replaced with the code for the current assignment.

After discussing this with Chad and Dillon Murphy, we decided that this was too confusing, and gives students an incorrect idea of how git should be used. Also, it would only work in pairs if the students worked in the same pair throughout the semester, and I like to have my students switch partners for each lab. So, I wrote my lab instructions using Bitbucket, but one repository per assignment, per pair.

I tried it out during my summer 2013 section of CS-140. It was a nice testbed — with only 6 students it was not too difficult to work out the bugs in the procedures. In another post I’ll explain how I had the students use the repository, and how I processed the repositories for grading (including the scripts that I wrote with some help from Stoney Jackson on a train ride from Providence to Philadelphia.)

The problem came when I decided to try it with my CS-135 Programming for Non-CS Majors class in Fall 2014. Soon after we started the first lab — the git lab — the student who had also been in the summer class could not add her lab partner as a collaborator to her Bitbucket repository. After a bit of investigation, we determined that while Bitbucket allows unlimited private repositories, you can have at most 5 collaborators — per account, not per repository — without paying. That had not been a constraint with a class of 6, but it certainly was with a class of 24, and it would just get worse as the students progressed to other courses.

At this point, I gave up on git for the semester and started looking for alternatives. I had used Gitolite in the past, which had worked well but had no web interface. I wanted something more like Github, and came across GitLab. I added installing GitLab to Chad and Dillon’s project of building a number of new servers for the department (see here and here.)

Once Chad had finished the GitLab install and worked out the kinks, we decided to test it by running through the CS-140 Lab 1 using the new server. I quickly updated the lab assignment to refer to a repository on our GitLab server, set up the repository on the new server, and had Dillon and Chad work their way through the lab to look for problems. They found a few typos, and a number of places where I had not replaced all the references to Bitbucket with GitLab, but otherwise it worked.

We had only one puzzling issue — Dillon was able to push changes to a repository he should not have had sufficient privileges to modify. It turns out that (not surprisingly) if you are a GitLab adminstrator, you have the ability to push to any repository on the server.

I’m looking forward to testing the server on a larger scale with 48 students in CS-140 starting in January.

  1. I’m aware that students can get multiple private repositories from GitHub, but that requires contacting GitHub and asking. The Bitbucket option just seemed easier.