CS 6301-001: Language-based Security

Course Information

Title: CS 6301-001: Language-based Security
Times: MW 1:00–2:15
Location: CB3 1.310
Instructor: Dr. Kevin Hamlen (hamlen AT utdallas)
Instructor's Office Hours: MW 2:15–3:15, ECSS 3.704
Teaching Assistant: Ali Ghanbari (Ali.Ghanbari AT utdallas)
TA Office Hours: F 1:30–3:00, ECSS 4.620 (please email first)

Course Summary

This course introduces and surveys the field of Language-based Software Security, in which techniques from compilers and programming language theory are leveraged to address issues in computer security. Topics include:

  1. Formal Methods-based Software Development
  2. In-lined Reference Monitors
  3. Software Fault Isolation
  4. Address Space Randomization
  5. Binary Software Attack Surface Reduction
  6. Certifying Compilation
  7. Web Scripting Security
  8. Information Flow Controls
  9. Cyber-deceptive Software Engineering

About half of the course is devoted to formal methods-based development of secure software, with a focus on program-proof co-development and higher-order functional programming. The remainder of the course surveys alternative and complementary cybersecurity solutions founded on programming language semantics-based analysis, transformation, monitoring, and compilation of code. In-course discussion will highlight opportunities for cutting-edge research in each area. If you conduct work involving software security, this course will provide you with an array of powerful tools for addressing software security issues. If your work involves programming language design and implementation, this course will show you how to take techniques that you already know and apply them to a new and important problem domain. If your career involves management or development of high-assurance software systems, this course will provide a comparative analysis of traditional versus language-based techniques.

The course is open to Ph.D. students and Masters students. Interested undergraduates should see the instructor for permission to take the course.

Suggested complementary course: Interested students may wish to consider taking CS 6371 Advanced Programming Languages before/after this course or concurrently, since it presents material that supplements and enhances several of the above topics.

Grading

Homework (30%): For the first 10 weeks of the course, students will complete a series of programming exercises assigned through eLearning. Background material helpful for completing the exercises can be found in the online textbook Software Foundations.

Quizzes (30%): Most classes will begin with a short quiz testing the students comprension of an assigned reading for the day. Questions will typically be multiple choice or short answer. The easier questions will be designed to test whether the student has read the material, and the harder ones will test deeper understanding of more subtle points.

Class Participation (10%): Students are expected to come to class or attend online having read the assigned paper(s), and prepared with questions, critiques, and discussion topics. Regular attendance and class participation will count 10% towards their grades in the course.

Project (30%): Students will work individually or in a small team for the last 6 weeks of the course to complete a small project. A typical project will involve implementing and/or formally verifying a language-based security mechanism covered in the class or related to class material. Students conducting research are encouraged to choose projects that synergize with their research activities. Students will present their projects in class, with the presentation counting toward their project grade.

COVID-19 Policy

(Last updated: 8/20/2021)

In accordance with university guidelines, the course will be taught in-person but initially in a "flexible" mode that allows out-of-country international graduate students who have not yet arrived in the US (e.g., due to pandemic-induced travel restrictions) to receive class lectures remotely. The enrollment for this course is expected to be significantly less than the classroom capacity, which should permit adequate social distancing with full attendance throughout the semester.

Instructor's informal (non-mandated) attendance preference: The instructor's experience is that despite advances in remote learning, online attendance of intellectually challenging classes like this one unfortunately remains significantly less effective for student learning than in-person attendance for a variety of reasons. Nevertheless, the risks associated with COVID-19 demand that we all be vigilent and respectful of each other's health needs. The instructor therefore recommends (but cannot mandate or enforce) a balanced approach: If you are experiencing symptoms, have come in close contact with an infected individual, or have other reason to believe you are at risk of spreading illness to others, please do not come to class in-person. Contact the instructor to conduct class activities on-line or through delayed make-up work. If you are not experiencing symptoms, have no reason to believe you are infected or contagious, are at low risk (e.g., vaccinated), and are present in the country, please do come to class. Wear a mask (ideally one that is effective against COVID) if desired, and socially distance. In-person attendance will make the class more effective for everyone if we can achieve it safely.

Texts

In our study of the Coq theorem proving system, we will be using the following online textbook:

For those who wish to explore Coq in greater depth (e.g., for developing projects), the following book by the Coq developers is recommended:

Additionally, the following text available through the UTD library may be useful for general background on type theory and functional programming:

Tentative Course Schedule

Date Topic Assigned Reading(s) Coq Exercises
Program-Proof Co-development
Lecture 1:
Mon 8/23
Introduction to Language-based Security
Lecture Slides
Coq Transcript
Assignment 1 due 9/1
(Coq Basics)
Lecture 2:
Wed 8/25
Functional Programming with Coq
Coq Transcript
Software Foundations: Basics chapter, up to and including the first two exercises (nandb, andb3).
Lecture 3:
Mon 8/30
Program-Proof Co-development
Coq Transcript
F. Williams. Investigating SANS/CWE Top 25 Programming Errors. Tech. Rep. ICTN 6870, E. Carolina University, 2009.
Lecture 4:
Wed 9/1
Parametric Polymorphism
Coq Transcript
J. Walden, J. Stuckman, and R. Scandariato. Predicting Vulnerable Components: Software Metrics vs Text Mining. In Proc. 25th Int. Sym. Software Reliability Engineering (ISSRE), pp. 23–33, November 2014.
No Class:
Mon 9/6
No class: Labor Day Assignment 2 due 9/15
(Induction)
Lecture 5:
Wed 9/8
Type Universes
Coq Transcript
P. Hudak. Conception, Evolution, and Application of Functional Programming Languages. ACM Computing Surveys, 21(3):359–411, May 1989.
  • Required sections: Abstract, Introduction, 2.1, 2.3, and 2.4
Lecture 6:
Mon 9/13
Inductive Propositions
Coq Transcript
Tactic Quick-reference Sheet
X. Leroy. Formal Verification of a Realistic Compiler. Communications of the ACM, 52(7):107–115, 2009.
Lecture 7:
Wed 9/15
Minimal Trusted Computing Base
Coq Transcript
no assigned reading
Lecture 8:
Mon 9/20
Constructivistic Logic
Coq Transcript
J. Bau and J. C. Mitchell. Security Modeling and Analysis. IEEE Security & Privacy 9(3):18–25, 2011. Assignment 3 due 9/27
(Lists)
Language-based Security Foundations
Lecture 9:
Wed 9/22
The Science of Software Security
Lecture Slides
M. Müller-Olm, D. Schmidt, and B. Steffen. Model-Checking: A Tutorial Introduction. In Proc. 6th Int. Sym. Static Analysis (SAS), pp. 330–354, September 1999.
  • Required sections: 1–4.2 and 5
  • I will not quiz you on the parts about fixed point theory in Section 3.3, but do read the paragraph on Computational Tree Logic.
Lecture 10:
Mon 9/27
Superset Disassembly
Lecture Slides
E. Bauman, Z. Lin, and K.W. Hamlen. Superset Disassembly: Statically Rewriting x86 Binaries Without Heuristics. In Proc. Network and Distributed Systmes Security Sym. (NDSS), 2018.
Code-reuse Attacks and Defenses
Lecture 11:
Wed 9/29
Return-oriented Programming
Lecture Slides
E.J. Schwartz, T. Avgerinos, and D. Brumley. Q: Exploit Hardening Made Easy. In Proc. 20th USENIX Security Symposium, 2011. Assignment 4 due 10/4
(Polymorphism)
Lecture 12:
Mon 10/4
Artificial Diversity
Lecture Slides
H. Shacham, M. Page, B. Pfaff, E.-J. Goh, N. Modadugu, and D. Boneh. On the Effectiveness of Address-Space Randomization. In Proc. ACM Conf. Computer and Communications Security (CCS), pp. 298–307, 2004.
Lecture 13:
Wed 10/6
Control-flow Integrity
Lecture Slides
M. Abadi, M. Budiu, Ú. Erlingsson, and J. Ligatti. Control-Flow Integrity: Principles, Implementations, and Applications. In Proc. ACM Conf. Computer and Communications Security, pp. 340–353, 2005. Assignment 5 due 10/13
(Logical Operators)
Lecture 14:
Mon 10/11
Binary Stirring
Lecture Slides
R. Wartell, V. Mohan, K.W. Hamlen, and Z. Lin. Binary Stirring: Self-randomizing Instruction Addresses of Legacy x86 Binary Code. In Proc. ACM Conf. Computer and Communications Security (CCS), 2012.
Lecture 15:
Wed 10/13
Binary Attack Surface Reduction
Project Slides
M. Ghaffarinia and K.W. Hamlen. Binary Control-flow Trimming. In Proc. ACM Conf. Computer and Communications Security (CCS), 2019.
In-lined Reference Monitors
Lecture 16:
Mon 10/18
Theory of IRMs
Lecture Slides
F.B. Schneider. Enforceable Security Policies. ACM Transactions on Information and System Security, 3(1):30–50, 2000. Assignment 6 due 11/1
(Inductive Props)
Lecture 17:
Wed 10/20
Cryptojacking
Project Slides
W. Wang, B. Ferrell, X. Xu, K.W. Hamlen, and S. Hao. SEISMIC: SEcure In-lined Script Monitors for Interrupting Cryptojacks. In Proc. European Sym. on Research in Computer Security (ESORICS), 2018.
Lecture 18:
Mon 10/25
IRM Validation
Lecture Slides
K.W. Hamlen, M.M. Jones, and M. Sridhar. Aspect-oriented Runtime Monitor Certification. In Proc. 18th Int. Conf. Tools and Algorithms for the Construction and Analysis of Systems (TACAS), pp. 126–140, March–April 2012.
Lecture 19:
Wed 10/27
Web Scripting Security
Lecture Slides
P.H. Phung, M. Monshizadeh, M. Sridhar, K.W. Hamlen, and V.N. Venkatakrishnan. Between Worlds: Securing Mixed JavaScript/ActionScript Multi-party Web Content. IEEE Transactions on Dependable and Secure Computing (TDSC), 12(4):44–457, July–August 2015.
Lecture 20:
Mon 11/1
Machine-checked Native Code Validation D. Brumley, I. Jager, T. Avgerinos, and E.J. Schwartz. BAP: A Binary Analysis Platform. In Proc. Int. Conf. Computer Aided Verification (CAV), 2011.
Confidentiality Enforcement
Lecture 21:
Wed 11/3
Intro to Information Flow
Lecture Slides
A. Sabelfeld and A.C. Myers. Language-Based Information-Flow Security. IEEE J. Selected Areas in Communications, 21(1):5–19, 2003. Project due 12/15
Lecture 22:
Mon 11/8
Type-based Information Flow Controls
(Lecture on Projects/Picinae)
A.C. Myers. JFlow: Practical Mostly-Static Information Flow Control. In Proc. 26th ACM Sym. Principles of Programming Languages (POPL), pp. 228–241, 1999.
Current Research
Lecture 23:
Wed 11/10
Cyber-deceptive Software Engineering F. Araujo, K.W. Hamlen, S. Biedermann, and S. Katzenbeisser. From Patches to Honey-Patches: Lightweight Attacker Misdirection, Deception, and Disinformation. In Proc. ACM Computer and Communications Security (CCS), 2014.
Lecture 24:
Mon 11/15
Binary Rewriting Compatibility Challenges
(No lecture: Project team meetings)
X. Xu, M. Ghaffarinia, W. Wang, K.W. Hamlen, and Z. Lin. ConFIRM: Evaluating Compatibility and Relevance of Control-flow Integrity Protections for Modern Software. In Proc. USENIX Security Symposium, 2019.
Lecture 25:
Wed 11/17
Object Flow Integrity
(Lecture on Projects/Picinae)
W. Wang, X. Xu, and K.W. Hamlen. Object Flow Integrity. In Proc. ACM Computer and Communications Security (CCS), 2017.
No Class:
Mon 11/22
No class: Fall break
No Class:
Wed 11/24
No class: Fall break
Conclusions
Lecture 26:
Mon 11/29
Future Research no assigned reading
Presentations 1:
Wed 12/1
Course Wrap-up & Conclusions
Lecture Slides
Presentations 2:
Mon 12/6
Project Presentations
  1. Team HIP
  2. Temoq Team
  3. Team Angr Lifter