Oct 092014
 

I wanted to post an update to my post Sabbatical Reading List.

To better keep track of my progress with my reading, I’ve started tagging the books on my LibraryThing account in different categories:

Sep 082014
 

Summer is over, and it’s time to really get to work on my Sabbatical project.

I did do some work this summer – I’ve read 3 of the books on my Sabbatical Reading List (and added a few more to the list) and I’ve finally de-lurked on the OpenMRS developer mailing list and in some of the online meetings, and I’ve made a decision to convert all of my course materials to Markdown (the better to track changes on GitHub – see a future post). But, it’s all been pretty passive.

So, the Friday before the Labor Day Weekend, I decided it was time to get back to the “develop code in OpenMRS” part of the project.

Since it had been a couple of months since I had set up  my development environment and tried to build the OpenMRS code, I decided that starting over from scratch (mostly) would not be a bad idea. Here is what I did:

  1. Read OpenMRS’s Getting Started as a Developer wiki page. I had already set up my OpenMRS ID, and signed up for the developer mailing list. I already had a GitHub account, as well.
  2. Read Development ProcessfromOpenMRS Developers’ Guide, and got the code:
    1. Fork openmrs-core repository.
    2. Clone my fork onto my computer.
    3. Set the OpenMRS repository as my upstream remote so that I can pull changes from the main project into my local working copy.
  3. Set up development environment based on Get Set Up from the OpenMRS Developers’ Guide. I chose the section on Manual Installation because Iwantto be able to develop code fortheOpenMRS core application. I followed the general outline of this section, but went about some of the software installations differently.
    1. Install MySQL. Because I am setting up my development environment on a Mac, I installed MySQL using Homebrew
    2. Install Maven using Homebrew
    3. Install Git using Homebrew (actually, I already had Git installed, but I made sure it was up-to-date)
    4. BuildtheOpenMRS code:
      cd openmrs-core
      maven clean install
    5. RuntheOpenMRS web app through the jetty server:
      cd webapp
      mvn jetty:run
    6. The first time you run the web app, it will take you through the Setup wizard
  4. Set up Eclipse. I already had Eclipse installed, but I made sure that my version was up-to-date. Eclipse had been updated to a new major version (Luna) since the last version I installed (Kepler). 1
  5. Git IDE Integration: Since EGit is already installed in current versions of Eclipse, and I’ve already forked and cloned the repository, I really only needed to do the To import as a Maven project section, to get the the projected into Eclipse.
  6. Build the OpenMRS code under Eclipse. I followed the steps in the following section:
    1. How to run the build
    2. How to run Junit
    3. How to Run Web Application

Now that I have a working environment that builds and runs, the next step is to choose a ticket to work on.

  1. In the past, I had just downloaded the newest version, and replaced what I already had installed. I figured that there must be a way to do the update without having to reinstall and reconfigure all of my plugins. It turns out that you can add the release’s repository to the Available Software Sites: http://wiki.eclipse.org/FAQ_How_do_I_upgrade_Eclipse%3F
Jun 182014
 

I have been lurking in the OpenMRS project for the last 6 months or so. I have read wiki pages, installed the development environment, cloned the repository and built the code, and listened in on a number of OpenMRS weekly developer meetings.

As I begin my sabbatical, I realized that it was time to finally introduce myself to the project members. So here’s what I posted to the OpenMRS Developers mailing list, and to OpenMRS Talk:

My name is Karl Wurst and I am a Professor of Computer Science at Worcester State University in Worcester, MA, USA (about 80 km west of Boston.)

Our university has recently created a concentration in Software Development for our Computer Science majors, and I am one of the primary instructors for the courses in this concentration. I am currently on sabbatical (no teaching responsibilities) from June through December 2014 and my plan is to participate in OpenMRS to improve my somewhat outdated Software Engineering skills.

I have installed the development environment, built the openmrs-core code, and now I will begin looking for tickets that I can work on. I am excited that the 1.10 beta release is imminent, and hope that I can be of some help in that sprint. I am also very interested in the development, testing, integration, and release processes as a way of seeing “real-life” examples of many of the tools and technologies that I have been reading about, but not had any hands-on experience with.

I am also part of the Foss2Serve/POSSE (foss2serve.org) group that is encouraging faculty to have students participate in Humanitarian FOSS projects as part of their coursework, and have been doing that primarily with our senior project course with varying amounts of success. I would like to have my students participate in OpenMRS beginning with the Spring 2015 semester (January through May 2015.) I want to get familiar with the project myself, first, so that I can direct them.

I also want to use OpenMRS for examples in our courses on software process and management, and testing and QA. We also have an installed server instance that we hope to use for the Health Informatics course that we teach for our Nursing students so that they can get some hands-on time with an EMR system.

I’ve already learned a lot just by exploring and listening. I’m looking
forward to learning even more by contributing.

A note to my students: Introducing yourself to a new group of people is hard, even for faculty members! I have put this introduction off for a while. I may be a Professor, but these people are real experts – they do this stuff all the time, and many of them do it for a living! But, as I’ve often found, once I forced myself to write my introduction and pressed the send button, I’ve gotten back only helpful, welcoming responses. Open Source communities really are welcoming groups that are genuinely happy to have you join, want your help, and will help you succeed. You’ll see…

Jun 062014
 

I have begun my reading list for my sabbatical. It includes general software development/software engineering books, as well as books specifically aimed at the next two courses I will be teaching in our Software Development Concentration:

  • CS-348 Software Process Management (Spring 2015)
  • CS-443 Software Quality Assurance and Testing (Fall 2015)

Here is my sabbatical reading list on my LibraryThing account. I will update it as I find other books, or have other books suggested to me.

Jun 062014
 

I am spending the Fall 2014 semester on sabbatical1. This is the proposal I submitted to request my sabbatical leave:

I will use my sabbatical to become more expert in the area of Software Development/Software Engineering. The Computer Science Department recently created a Concentration in Software Development, which has expanded our course offerings in this area from two courses to four courses. This expansion does not simply expand the number of hours we spend on Software Development topics, but adds many topics that we have not been covering at all. Many of these topics are outside the expertise of any of the faculty in the department.

I am the only member of the Computer Science Department who has worked as a professional software developer, but have not worked professionally in that field for over 20 years. Many of the current processes, techniques, and tools were not in use at that time. The members of the department have worked to learn these new skills so that we can teach them to our students, but have only an academic/theoretical knowledge of many of them – we lack the practical experience of using these skills in a professional environment.

During my sabbatical, I will learn the processes, techniques, and tools of modern software development, and apply them in a professional context by working as a full-time (but unpaid) developer within an open source software project. I will work with Dr. Heidi Ellis at Western New England University and Dr. Gregory Hislop at Drexel University to get the academic perspective on how to teach these skills to undergraduate students, and to take advantage of their experience working with, and their contacts within, open source projects.

Drs. Ellis and Hislop are both well-known software engineering researchers and software engineering education researchers. They have been on the forefront of work to help students develop professional software engineering skills by working with open source software projects. They have a particular interest in having students work within projects with a humanitarian aspect. I have done some work with them in this area over the last 4 years, but have not had the time to work exclusively and intensively on developing these skills myself. In addition, Dr. Hislop served on the committee that developed the SE 2004 software engineering curriculum for the ACM and IEEE, and is currently serving on the committee that is updating those standards. I have already tapped his expertise in developing our new curriculum, and plan to do so again as we develop the new courses in the curriculum.

Proposal
  1. Develop a list of processes, techniques, skills, and tools that are necessary for modern software development. Thislistwill bedevelopedin consultationwithDrs. EllisandHislop, by reviewingtheSLOs of our newly approved Software Development Concentration courses, and by reviewing the SE 2004 Curriculum and any publicly released drafts of the new ACM/IEEE Software Engineering curriculum. This list will include,atminimum:
    • Agile development processes
    • Automated build environments
    • Automated test environments
    • Version control systems
    • Software architectures
    • Design patterns
    • Requirements elicitation
    • Software licensing and intellectual property
    • Project planning and estimation
    • Risk management
    • Analysis techniques
    • Test planning, strategies and techniques
    • Test coverage
    • Code reviews
    • Quality assurance
    • Project and team management
  2. Select an appropriate humanitarian open source project to participate in. The project will be one which
    • Allows me to experience the full range of processes, techniques, skills and tools from the list above. (Or as many as possible.)
    • Allows me to use tool and language skills I already possess to minimize the number of new tools and languages I need to learn.
    • I can continue to use with students in the Software Development Capstone course, and with other courses in the concentration.
    • At this point, the two projects that seem most likely for my participation are:
      OpenMRS (http://openmrs.org/)
      “The global OpenMRS community works together to build the world’s leading open source enterprise electronic medical record system platform.
      We’ve come together to specifically respond to those actively building and managing health systems in the developing world, where AIDS, tuberculosis, and malaria afflict the lives of millions of people.
      Our mission is to improve health care delivery in resource-constrained environments by coordinating a global community to create and support this software.”
      Ushahidi (http://www.ushahidi.com/)
      “We are a non-profit tech company that specializes in developing free and open source software for information collection, visualization and interactive mapping. We build tools for democratizing information, increasing transparency and lowering the barriers for individuals to share their stories.
      “Ushahidi”, which means “testimony” in Swahili, was a website that was initially developed to map reports of violence in Kenya after the post-election fallout at the beginning of 2008. Since then, the name “Ushahidi” has come to represent the people behind the “Ushahidi Platform”. Our roots are in the collaboration of Kenyan citizen journalists during a time of crisis. The original website was used to map incidents of violence and peace efforts throughout the country based on reports submitted via the web and mobile phones. This website had 45,000 users in Kenya, and was the catalyst for us realizing there was a need for a platform based on it, which could be used by others around the world.”
  3. Participate in the selected project. Iwillparticipate in the selected project on a full-time (unpaid) basis, contributing to the project in whatever ways I can including:
    • Participation in planning and design meetings
    • Writing code
    • Testing
    • Writing documentation
    • Helping with support
  4. Participate in Western New England University course. I would like to observe or help teach a software engineering course at WNEU so that I can see what pedagogy is used in the course, and adapt it to our own courses.
  5. Blog about my experiences. I will write about my sabbatical experiences on my blog (http://blog.karl.w-sts.com/). This will allow me to document and reflect on what I am learning and how I can use it in our own courses.
Preparation for this Sabbatical
  • Participant in POSSE (Professors’ Open Source Summer Experience) in 2010, 2011, and 2013 – A workshop designed to prepare faculty to support students working within open source software projects. The summer 2013 workshop group is continuing to work together over the 2013-2014 academic year.
  • Participant in SoftHum (Software for Humanity) workshop in 2011 – A workshop with faculty working to design materials for use in their courses.
  • One of the organizers of Teaching Open Source Symposium in 2012
  • Participant in OpenFE Materials Sprint in 2013 – A workshop to develop materials for teaching POSSE workshops
Relevant Courses
  • Introduction to Programming
  • Software Construction, Design and Architecture
  • Software Process Management
  • Software Quality Assurance and Testing
  • Software Development Capstone
  • Software Development Process
Benefits to the Computer Science Department and to the University

The department will benefit by having a faculty member who has relevant professional experience in software development, with contacts within at least one open source software project, and with the academic experience of applying that experience to courses. I will be able to use that experience and those contacts in order to give students the opportunity to gain valuable practical experience working within a large software project on the same types of tasks and using the same kinds of skills that they will be expected to use in their professional careers.

The University will benefit through a strengthened Computer Science program, by having a higher profile in the software development world, and being recognized as an organization that has donated a semester’s-worth of work of a full-time faculty member to further the mission of a humanitarian project.

There have been some changes since I wrote the proposal and it was approved:

  1. IchoseOpenMRS as the project that I will be working within.WhileUshahidi seems like a very interesting project,OpenMRS seemed to fit better with my goals for the following reasons:
    • It is written primarily in Java, which is the language that we use most in our Computer Science courses. (Ushahidi is written primarily in PHP, which we do not teach at all.)
    • OpenMRS seems to have a more “formal” software development process and tooling, which I think covers more of the topics on the list of what I want to learn.
    • We can use OpenMRS as a tool in the Health Informatics course that we teach for the Nursing majors, to provide an Electronic Medical Record system for the Nursing students to try out.
  2. I participated in the Teaching Open Source Symposium at SIGCSE 2014 and in POSSE 2014.
  3. I increased my use of Open Source tools both in my own work, and introduced them in our first-year courses.
  4. Students in my Spring 2014 capstone course did some work within the OpenMRS project.
  5. I started attending OpenMRS online meetings to familiarize myself with the project.

I will be writing more about sabbatical as the summer and the semester move along.

  1. For those not familiar with the ways of academia, a sabbatical is a paid leave awarded every 7 years to allow a faculty member time for research or updating skills.
May 282014
 

cue Ozzy Osbourne laughter…

This blog is coming to you direct from Amtrak Northeast Regional Train 95, where Stoney Jackson and I are on our way to POSSE 2014 at Drexel University in Philadelphia, PA. This is becoming an annual tradition for us.

So, why is it called the Coding Train? Because we are spending the 5 hour train ride writing code!

When we did this for the first time last year, we worked on the code for the grading scripts that I had started writing in bash (https://github.com/kwurst/grading-scripts/tree/bash-version). Stoney started adding error checking, and then a Python version – neither of which he finished, but we learned a lot about how GitHub works for collaborative development.

This year we discussed a number of options for what project we would sprint on (after we spent a lot of time on professor-talk about curricula, and courses, and learning outcomes, and assessment) but we ended up back on the same project. This time our starting point was the Python conversion of the original scripts that I had started in December, and which I had just begun to refactor this month (https://github.com/kwurst/grading-scripts/tree/master).

Stoney has been doing some serious refactoring on the code, adding one major new feature: a JSON configuration file so that I don’t need 15 different scripts – just different configuration files to pass to a single, more general script. He’s also undertaken a major cleanup of the code, and added the project’s first unit test!

I, on the other hand, have been installing tools that Stoney suggested – git flow and git bash prompt, and in the process having to debug my Mac’s installation of Homebrew and cleaning up my .bashrc file (being completely ignored by my shell) and my .bash_profile file (full of lots of cruft from previous installs.)

Stoney has just pushed his branch, so now it’s time for me to pull it, and test it on some data on my computer. And we’re almost to Philadelphia, so just in time…

Dec 282013
 

Now that we have the CS Department’s GitLab server set up, and CS-140 Lab 1 is rewritten and tested using the new server, I’ve started to think about how to automate my interactions with the server. I had already  written some Bash scripts to interact with the Bitbucket server to get student code, convert it to PDF files, and put it back on the server after grading. Those scripts should still work fine with GitLab, since it’s just git on a different server.

One thing that I had not been able to automate previously is the step of issuing a pull request for students to merge my grading branch into their repository. This was not too much of an issue when there were only 6 students in the summer class (so only 3 repositories per lab assignment), but it was going to take more time with ~48 students in the spring class. While reading RSS feeds, I came across a post mentioning the GitLab API. This could be the solution to my problems! And there’s a Python module for the API! I had already been writing Python scripts to make my grading easier, and had been starting to rewrite my Bash scripts in Python.

I started playing with the GitLab API in Python, and had managed to create a merge request (GitLab’s term for pull request.) I had also noticed that you could create GitLab accounts through the API. This seemed like something I should pursue – creating ~48 accounts per semester seemed like something that should be automated.

Since I intended to post my code on Github, one of the first issues I had to address is how to avoid publishing my private token for GitLab. I could have put in a dummy token before pushing my code, but I would have to remember to do that before every time I committed my code. The solution to this issue was solved through the use of the .gitignore file. If I put my token into a file, then I could add a line to my .gitignore file so that it would not be committed.

# Private GitLab Token - not to be stored in repository #
########################################################
gitlabtoken.txt

Then I could just read the token out of the file, and use that string.

# Get my private GitLab token
# stored in a file so that I can .gitignore the file
token = open('gitlabtoken.txt').readline().strip()

After importing the pyapi-gitlab module, I could use that token, along with the server’s URL to create a GitLab object. Notice, that I had to turn ssl verification off, since we only have a self-signed certificate.

# Create a GitLab object
# For our server, verify_ssl has to be False, since we have a self-signed certificate
git = gitlab.Gitlab(GITLAB_URL, token, verify_ssl=False)

Creating a user account is pretty simple using the API:

# Create the account  
success = git.createuser(name, username, password, email)

The returned success value is a boolean — either it worked, or it failed (but you can’t tell why…).

One thing that’s a bit odd about the createuser call, is that you have set a password for the user, but the notification email to the doesn’t include the password. (If you create a user account from the web interface, it generates a random password, includes it in the notification email to the user, and requires the user to change their password when first logging in.) And, the password you set doesn’t seem to work either!

So, I’m just telling the students that they should use the “Forgot Password” link to have a password reset email sent to them, and then proceed from there. (If this is ever fixed, I’ll have to generated a random password.)

Getting the class list as a CSV file from the Blackboard Grade Center is pretty easy, and the first three rows contain the student’s last name, first name, and username. I can use those three strings to generate the name, username, and email needed for the createuser API call.

The only challenge with processing the CSV file is that Blackboard puts some strange character at the beginning of the file, so the file has to be opened with utf-8 encoding. (And the header line needs to be thrown away.)

The last thing I wanted to add is a way to have optional verbose output, so that I could see if the user creation was working. (I decided that it should always notify the user if the account creation failed.)  To do this I had to learn two new things about Python: how to parse arguments1, and how to send output to stderr.

I used the argparse module:

import argparse
# Set up to parse arguments
parser = argparse.ArgumentParser()
parser.add_argument('filename', help='Blackboard CSV filename with user information')
parser.add_argument('-v', '--verbose', help='increase output verbosity', action='store_true')
args = parser.parse_args()

and used the verbose argument to determine what to print:

if not success:
    sys.stderr.write('Failed to create acccount for: '+name+ ', '+username+', '+email+'\n') 
elif args.verbose:
    sys.stderr.write('Created account for: '+name+', '+username+', '+email+'\n')

Full code is on Github here.

  1. I already knew how to do simple argument parsing, but I wanted to learn how to deal with optional arguments.
Dec 132013
 

Downloading student assignment files from Blackboard as a single zip file saves a lot of time — you don’t have to individually open each “attempt”, download the file (renaming it in the process, so you don’t keep overwriting the previous file, since they are all named “Homework1.pdf” 😉 ), and then move on to the next one. Instead you get one convenient .zip file that contains all of the assignment files.

Unfortunately, Blackboard does some other things that make your life a bit more difficult. Once you unzip the file, you will find:

  1. The student files are renamed from filename.ext to assignmentname_username_attempt_datetime_filename.ext
  2. A text file is created for each student named assignmentname_username_attempt_datetime.txt even if the student has not entered any text data or comments.

Checking all of the text files to see if they really contain a comment and deleting those that don’t, and renaming all of the assignment files to username.ext so that I can start grading them 1 This process takes 15 minutes or more per assignment, which certainly lowers my enthusiasm for grading.

Today, I decided that I should write some code to automate this task. The time it would take to write the script would be recouped in only a few assignments. I decided to write the script in Python because I could easily see how to do the string manipulations. My shell scripting string manipulations are not as good. I would have to learn how to do the file system manipulations in Python, but I figured that would be relatively simple.

The first step is getting a list of all the files in the directory (leaving out all of the subdirectories)2:

onlyfiles = [ f for f in os.listdir(dir) if os.path.isfile(os.path.join(os.curdir,f)) ]

The next step is filtering that list to get just the .txt files:

txtfiles = [ f for f in onlyfiles if '.txt' in f ]

Then you can search the contents of the textfiles. You’ll notice that there are two characteristic phrases that indicate no text data and no comments. You can just delete the files that contain both of those:

for f in txtfiles:
    file = open(f)
    contents = file.read()
    file.close()
    if 'There are no student comments for this assignment' in contents and \
       'There is no student submission text data for this assignment.' in contents:
        os.remove(f)
        print('Deleted', f)

After refreshing the list of files to be just the remaining files, you can go about renaming the files. They all have _attempt_ embedded in their filename. Then you want to strip off everything up-to-and-including the first underscore, and from the second underscore up to the file extension. Then rename the file.

for f in onlyfiles:
    if '_attempt_' in f:
        first = f.find('_') # location of first underscore
        second = f.find('_',first+1) # location of second underscore
        extension = f[f.rfind('.'):] # get file extension
        newf = f[first+1:second] + extension
        os.rename(f, newf)
        print('Renamed', f, 'to', newf)

There are probably other features I can add, but this works well enough for now. Back to grading…

Full code is on GitHub here.

  1. I may still have to convert some of them to PDFs, if the students have not followed instructions, since I grade them by marking up the PDFs on my iPad. But that’s something I’ll tackle later. For my programming classes, I do that with my grading scripts which are still a work-in-progress.
  2. http://stackoverflow.com/a/3207973
Dec 102013
 

Chad Day recently completed the installation of our new GitLab server (read about it here and here.) This project was precipitated by some issues I had been having in trying to teach the use of git earlier in our curriculum. I had been having the CS-401 Software Development Process students use git and github in their FOSS projects, but it was difficult for them seeing git for the first time and expecting them to use it intensively in a project in the same semester. They had asked a number of times, “Why don’t you teach this in an earlier course?”

So, I decided to try using it in the first programming course – CS-140 Introduction to Programming. While they don’t do any large projects in CS-140, they do work on their weekly labs as Pair Programming. Using git for the collaboration aspect (so they don’t have to keep emailing versions back-and-forth to each other) and as a way to submit their completed lab assignments to me (so I only have to receive one copy of the assignment per pair) seemed to make a lot of sense. In addition, I had attended a workshop at CCSCNE 2013 entitled “Git on the Cloud” which provided a methodology to do just that, which had encouraged me further.

The “Git on the Cloud” workshop suggested using Bitbucket, since it allows an unlimited number of private repositories.1 I’m very willing (in fact, I often require) students to make their code for senior-level projects public with an open source license. But private is important for coursework at the freshman level.

On the other hand, the “Git on the Cloud” methodology involved using a single repository per student, and a different branch for each assignment(!)  In other words, whenever you change branches/assignments, all of your other code goes away, and is replaced with the code for the current assignment.

After discussing this with Chad and Dillon Murphy, we decided that this was too confusing, and gives students an incorrect idea of how git should be used. Also, it would only work in pairs if the students worked in the same pair throughout the semester, and I like to have my students switch partners for each lab. So, I wrote my lab instructions using Bitbucket, but one repository per assignment, per pair.

I tried it out during my summer 2013 section of CS-140. It was a nice testbed — with only 6 students it was not too difficult to work out the bugs in the procedures. In another post I’ll explain how I had the students use the repository, and how I processed the repositories for grading (including the scripts that I wrote with some help from Stoney Jackson on a train ride from Providence to Philadelphia.)

The problem came when I decided to try it with my CS-135 Programming for Non-CS Majors class in Fall 2014. Soon after we started the first lab — the git lab — the student who had also been in the summer class could not add her lab partner as a collaborator to her Bitbucket repository. After a bit of investigation, we determined that while Bitbucket allows unlimited private repositories, you can have at most 5 collaborators — per account, not per repository — without paying. That had not been a constraint with a class of 6, but it certainly was with a class of 24, and it would just get worse as the students progressed to other courses.

At this point, I gave up on git for the semester and started looking for alternatives. I had used Gitolite in the past, which had worked well but had no web interface. I wanted something more like Github, and came across GitLab. I added installing GitLab to Chad and Dillon’s project of building a number of new servers for the department (see here and here.)

Once Chad had finished the GitLab install and worked out the kinks, we decided to test it by running through the CS-140 Lab 1 using the new server. I quickly updated the lab assignment to refer to a repository on our GitLab server, set up the repository on the new server, and had Dillon and Chad work their way through the lab to look for problems. They found a few typos, and a number of places where I had not replaced all the references to Bitbucket with GitLab, but otherwise it worked.

We had only one puzzling issue — Dillon was able to push changes to a repository he should not have had sufficient privileges to modify. It turns out that (not surprisingly) if you are a GitLab adminstrator, you have the ability to push to any repository on the server.

I’m looking forward to testing the server on a larger scale with 48 students in CS-140 starting in January.

  1. I’m aware that students can get multiple private repositories from GitHub, but that requires contacting GitHub and asking. The Bitbucket option just seemed easier.
Dec 022013
 

In my CS-135 Programming for Non-CS Majors class, one of the primary objectives for the students is to learn to work with collections of data in files. I’m always happy when this requires manipulations that can’t be performed with other tools that the students are comfortable with — thus motivating the need to learn to code.

This afternoon in class, students were working in groups on their final projects. Two groups came up against some problems in getting their data into a format that could be easily processed in Python. Both cases involved data that was only available in the form of PDF files.

The old standby of selecting text and pasting it into Excel did not provide nice columns of information. Our second attempt was to export the data as text.

Case 1

In the first case, we got text data that looked like:

Biology 306 N/A 306
Biotechnology 80 26 106
Business Administration 748 N/A 748
Chemistry 141 N/A 141
Communication 245 N/A 245
Communication Sciences & Disorders 218 N/A 218
Community Health 158 N/A 158
Computer Science 116 N/A 116
Criminal Justice 445 N/A 445
Early Childhood Education 80 19 99
Early Childhood Education, Non-Licensure 26 N/A 26

This looked promising – we’ve dealt with one-record-per-line-space-delimited data files in class before. You just need to read a line at a time, and use Python’s string split method to turn it into a list… But — wait! — the first item  is a variable number of words separated by spaces. That will make for some messy lists — they’ll all be of different lengths:

['Communication', '245', 'N/A', '245']
['Communication', 'Sciences', '&', 'Disorders', '218', 'N/A', '218']
['Community', 'Health', '158', 'N/A', '158']

Here’s the solution: Python lists can be indexed from the end using negative indices. So, we can definitely get at the last three values (numbers of majors — undergraduate, graduate, and total). Assuming a list in a variable department, they are at positions department[-3], department[-2], and department[-1] respectively.

But, what about the department name, which may be in multiple list items? Well, we can get it as a sub-list, using list slicing: department[:-3] yields:

['Communication']
['Communication', 'Sciences', '&', 'Disorders']
['Community', 'Health']

All that’s left is to concatenate them together into a single string:

name = ''
for item in department[:-3]:
    name = name + item + ' '

Full code is here: https://gist.github.com/kwurst/7761789

Case 2

In the second case, we got text data that looked like:

Boston    00350000    4368    65.9    15.2    0.8    2.1    15.9    0.1
Boston Collegiate Charter (District)    04490000    34    67.6    32.4    0.0    0.0    0.0    0.0
Boston Day and Evening Academy Charter (District)    04240000    162    13.0    55.6    0.0    6.8    24.7    0.0
Boston Green Academy
Horace Mann Charter School
(District)    04110000    72    70.8    26.4    0.0    1.4    1.4    0.0
Boston Preparatory Charter Public (District)    04160000    27    74.1    11.1    0.0    3.7    11.1    0.0
Bourne    00360000    145    90.3    4.8    0.0    2.1    2.8    0.0
Braintree    00400000    369    95.1    3.3    0.3    0.3    1.1    0.0

Which could be fixed the same way, except for the fact that some of the district names ended up broken across multiple lines. (I’m not sure why this happened, and it turned out that exporting the data in a different way fixed the problem. But I’d already found a solution, so I’m going to document it here…)

Working from the assumption that the district org code always starts with a zero (I know — not a good assumption, but it works in this case…), the solution involves checking for lines with no zero in them and concatenating them together. Then you can treat the lines as in Case 1.

for line in f:
    while line.find('0') == -1:
        line = line + f.readline()

Full code is here: https://gist.github.com/kwurst/7761789

%d bloggers like this: