CSC-103: Taming Big Data

Course Overview

Taming Big Data uses big data and computational science as the theme for learning about computers and computing. It will focus on applications from the sciences and social sciences.

We'll start the course with an overview of how we communicate with a computer and what programming is all about. Then we will move on to look at how we get the computer to manipulate data. By the end of the course students are able to develop programs that manipulate data for problems such as simulation, classification, and financial analysis. They will also be able to prepare data so that it can be processed by existing applications and tools.

The course starts out with a series of relatively small assignments. In the later portion of the term the problems grow in size, in keeping with students' increased ability to handle large amounts of data that are stored in files.

Language & Resources

In this course the bulk of programming is done in Python. Increasingly students who are very interested in the course topic are also encouraged to explore R.

• Python: https://www.python.org/
• R: http://www.r-project.org/

In class, you are required to use our lab iMacs. However, when working on your projects outside of class, you have a choice. If you'd like to continue using our iMacs, feel free! We have three spaces that you can use:

• Olin 107 (where we have class)
• Pasta lab (N104 in the Science & Engineering building)
• CS Resource room (Steinmetz Hall, 209A, just down the hall from my office)

Course Text

• Practical Programming: An Introduction to Computer Science Using Python (PP)
  Jennifer Campbell, Paul Gries, Jason Montojo, Greg Wilson
  ISBN 978-1-934356-27-2

Example Assignment

CSC-103
Fall 2012
Final Programming Assignment

• weather-1960-1969.txt contains Albany daily high temperatures for 1/1/1960 - 12/31/1969.
• weather-2002-2011.txt contains Albany daily high temperatures for 1/1/2002 - 12/31/2011.

19600101,28.4

Your assignment:

• Step 1. Come up with a question you'd like to ask, given temperature data for two 10 year periods that are roughly 40 years apart. It has to be a somewhat complex question. Something like just comparing the average temperature over each of the two 10 year periods would not be acceptable. You must email your proposed problem to me no later than class time on Thursday 11/8/2012. You will get a response no later than midnight that night. You are strongly encouraged to turn this part in early, so that you can get feedback sooner, so that you can start the next part sooner.

• Step 2. You will develop a program that answers the question you have asked. This program must:
  • use both provided files
  • include a function call that will start execution, so that it can be easily run from the linux prompt using just the .py file name
  • provide instructions and/or information to the user about what the program does
  • clearly request any necessary user input (though you are not required to have user input)
  • use at least two different data aggregation mechanisms (list, dictionary, tuple, set)
  • make use of both the date information and the temperature information (for example, the simple problem of computing the average temperature for each of the 12 year periods would not require that you use the date information, but there are other comparisons you can come up with that would make use of date information)
  This program is due by 5:00 on Wednesday, November 14, 2012. Hard copy should be delivered to my office (201 Steinmetz Hall), electronic version uploaded through nexus.

Example Lab

File Reading Exercise

Your task is to compute the average closing price over the entire period that the stock has been traded.

DO NOT USE readlines() or read(). DO USE readline().

Remember that the easy way to handle file access is to work in your own directory, and run python from the OS prompt in a Terminal window.

How to do this? If you want to plunge on ahead with no helpful hints, go for it! If you want some hints and tips, click here.

Assignments & Grades

There will be a regular programming assignments. As the term progresses these will utilize an increasing number of features of the Python language. We may also use other packages, such as the R statistical analysis package, later in the term.

There will be two in-class exams and a final exam. There may also be pop quizzes on material covered in prior classes and the reading. The intent is not that these be "punitive" in any way but, rather, that they provide motivation for you to keep up. Learning to program is like learning a foreign language. If you don't speak it during some part of every day your progress will be quite slow.

Grading:

The allocation of emphasis among the course components is as follows:

• Quizzes: 5 percent
• Attendance: 5 percent
• Homework and Labs: 30 percent
• Midterm Exams: 15 percent each
• Final Exam: 30 percent
  

Late Work:

No homework will be accepted late unless a prior arrangement is made. Just in case you missed that the first time No homework will be accepted late unless a prior arrangement is made.

All hardcopy of homework is due at the beginning of class on the due date. Electronic submission of program executables must be done before you arrive at class.

Schedule