It Will Never Work in Theory

A new update of Derek Jones' Empirical Software Engineering Using R is now available. At 283 pages, it includes an introduction to statistics, an overview of the forces shaping actual software engineering practices, lots of practical case studies that students can work through, and almost 700 references. What's more, the code and data are all on GitHub, along with slides to use in workshops. This is a tremendous resource, and we hope the community rallies behind it.

This site is on hiatus until further notice: in the wake of recent political events, our community's energy and attention should be focused on more important things. If you can, please get involved in projects like this.

Derek Jones has begun assembling a book to be titled Empirical Software Engineering Using R. You can view the work so far or file issues in this GitHub repo. I think this is a really exciting development, and I'm looking forward to seeing it evolve.

Davide Fucci, Giuseppe Scanniello, Simone Romano, Martin Shepperd, Boyce Sigweni, Fernando Uyaguari, Burak Turhan, Natalia Juristo, and Markku Oivo: "An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach". ESEM 2016, http://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf. Context: Test-driven development (TDD) is an agile practice claimed to improve the quality of a software product, as well as the productivity of its developers. A previous study (i.e., baseline experiment) at the University of Oulu (Finland) compared TDD to a test-last development (TLD) approach through a randomized controlled trial. The results failed to support the claims. Goal: We want to validate the...

Chris A. Mattmann, Joshua Garcia, Ivo Krka, Daniel Popescu, and Nenad Medvidovic: "Revisiting the Anatomy and Physiology of the Grid". Journal of Grid Computing, 13(1), March 2015, http://link.springer.com/article/10.1007/s10723-015-9324-0. A domain-specific software architecture (DSSA) represents an effective, generalized, reusable solution to constructing software systems within a given application domain. In this paper, we revisit the widely cited DSSA for the domain of grid computing. We have studied systems in this domain over the last ten years. During this time, we have repeatedly observed that, while individual grid systems are widely used and deemed successful, the grid DSSA is actually underspecified to...

Xinyu Wang, Sumit Galwani, and Rishabh Singh: "FIDEX: Filtering Spreadsheet Data using Examples". OOPSLA 2016, https://www.cs.utexas.edu/~xwang/pubs/oopsla16.pdf. Data filtering in spreadsheets is a common problem faced by millions of end-users. The task of data filtering requires a computational model that can separate intended positive and negative string instances. We present a system, FIDEX, that can efficiently learn desired data filtering expressions from a small set of positive and negative string examples. There are two key ideas of our approach. First, we design an expressive DSL to represent disjunctive filter expressions needed for several real-world data filtering tasks. Second, we develop an...

Dohyeong Kim, Yongwhi Kwon, Peng Liu, I Luk Kim, David Mitchel Perry, Xiangyu Zhang, and Gustavo Rodriguez-Rivera: "Apex: Automatic Programming Assignment Error Explanation". OOPSLA 2016, http://researcher.watson.ibm.com/researcher/files/us-liup/apex.pdf. This paper presents Apex, a system that can automatically generate explanations for programming assignment bugs, regarding where the bugs are and how the root causes led to the runtime failures. It works by comparing the passing execution of a correct implementation (provided by the instructor) and the failing execution of the buggy implementation (submitted by the student). The technique overcomes a number of technical challenges caused by syntactic and semantic differences of the two...

Santiago Perez De Rosso and Daniel Jackson: "Purposes, Concepts, Misfits, and a Redesign of Git", SPLASH 2016. Git is a widely used version control system that is powerful but complicated. Its complexity may not be an inevitable consequence of its power but rather evidence of flaws in its design. To explore this hypothesis, we analyzed the design of Git using a theory that identifies concepts, purposes, and misfits. Some well-known difficulties with Git are described, and explained as misfits in which underlying concepts fail to meet their intended purpose. Based on this analysis, we designed a reworking of Git (called...

Denae Ford, Justin Smith, Philip J. Guo, and Chris Parnin: "Paradise Unplugged: Identifying Barriers for Female Participation on Stack Overflow". FSE'16, https://denaeford.wordpress.com/2016/07/20/paradise-unplugged-barriers-to-stack-overflow-use/. It is no secret that females engage less in programming fields than males. However, in online communities, such as Stack Overflow, this gender gap is even more extreme: only 5.8% of contributors are female. In this paper, we use a mixed-methods approach to identify contribution barriers females face in online communities. Through 22 semi-structured interviews with a spectrum of female users ranging from non-contributors to a top 100 ranked user of all time, we identified 14 barriers preventing...

These papers were all presented at the 12th Annual International Computing Education Research conference in Melbourne earlier this month, and give a good sense of what CS education researchers are looking at and what they're finding. Elizabeth Patitsas, Jesse Berlin, Michelle Craig, and Steve Easterbrook: "Evidence That Computer Science Grades Are Not Bimodal", 10.1145/2960310.2960312. Although it has never been rigourously demonstrated, there is a common belief that CS grades are bimodal. We statistically analyzed 778 distributions of final course grades from a large research university, and found only 5.8% of the distributions passed tests of multimodality. We then devised a...

I noted earlier this week that the ACM was making papers from this year's International Computing Education Research conference freely available—but only for two weeks, which isn't what anyone else means by "open access" (and is frankly ridiculous). I'm therefore grateful to Neil Brown for pointing out bullet #4 on this page, which says that authors and owners permanently hold the right to: Post the Accepted Version of the Work on (1) the Author's home page, (2) the Owner's institutional repository, (3) any repository legally mandated by an agency funding the research on which the Work is based, and (4)...

We decided in 2012 that we would only review material that is openly available. I was therefore pleased to discover earlier this week that I could actually download papers from this year's International Computing Education Research conference from the ACM's site. ...until I read that "#ICER2016 papers are open access for next two weeks" (emphasis added). If that's true, it's a real shame: a lot of very cool things are being presented at ICER that deserve to be more widely known, but there's no point posting links that are going to 403 by the time most people outside the Great...

Jonathan L. Krein, Lutz Prechelt, Natalia Juristo, Aziz Nanthaamornphong, Jeffrey C. Carver, Sira Vegas, Charles D. Knutson, Kevin D. Seppi, and Dennis L. Eggett: "A Multi-Site Joint Replication of a Design Patterns Experiment Using Moderator Variables to Generalize Across Contexts". IEEE Trans. Software Engineering, 42(4), April 2016, 10.1109/TSE.2015.2488625. Context. Several empirical studies have explored the benefits of software design patterns, but their collective results are highly inconsistent. Resolving the inconsistencies requires investigating moderators—i.e., variables that cause an effect to differ across contexts. Objectives. Replicate a design patterns experiment at multiple sites and identify sufficient moderators to generalize the results across...

Helen Sharp, Yvonne Dittrich, and Cleidson R.B. de Souza: "The Role of Ethnographic Studies in Empirical Software Engineering". IEEE Trans. Software Engineering, 42(8), August 2016, 10.1109/TSE.2016.2519887. Ethnography is a qualitative research method used to study people and cultures. It is largely adopted in disciplines outside software engineering, including different areas of computer science. Ethnography can provide an in-depth understanding of the socio-technological realities surrounding everyday software development practice, i.e., it can help to uncover not only what practitioners do, but also why they do it. Despite its potential, ethnography has not been widely adopted by empirical software engineering researchers, and...

Felienne Hermans and Efthimia Aivaloglou: "Do Code Smells Hamper Novice Programming?" TUD-SERG-2016-006, 2016. Recently, block-based programming languages like Alice, Scratch and Blockly have become popular tools for programming education. There is substantial research showing that block-based languages are suitable for early programming education. But can block-based programs be smelly too? And does that matter to learners? In this paper we explore the code smells metaphor in the context of block-based programming language Scratch. We conduct a controlled experiment with 61 novice Scratch programmers, in which we divided the novices into three groups. One third receive a non-smelly program, while the...

Perspectives on Data Science for Software Engineering presents the best practices of seasoned data miners in software engineering. Its goal is to transfer the knowledge of experts from seasoned software engineers and data scientists to newcomers in the field. While there are many books covering data mining and software engineering basics, they present only the fundamentals and lack the perspective that comes from real-world experience. This book offers unique insights into the wisdom of the community’s leaders. Ideas are presented in digestible chapters designed to be applicable across many domains. Topics included cover data collection, data sharing, data mining, and...

You are invited to participate in a survey on software licensing designed to investigate how well software developers understand common open source software licenses. We are looking for software developers that have built or are currently building on open source software in their projects (and I am personally interested in hearing from people building open source software for research). The study is being conducted by Prof. Gail Murphy (murphy@cs.ubc.ca) and graduate student Daniel Almeida (daa@cs.ubc.ca); participating in the anonymous online survey will take approximately 30 minutes. If you are interested in participating, please go to: https://survey.ubc.ca/surveys/danielalmeida/software-licensing-survey/ If you have any...

Have you read any empirical software engineering research papers recently that you think a wider audience would enjoy? If so, please send us pointers: we'd be happy to feature them. (But please note that we only discuss work that is openly available—nothing paywalled, please.)

Functional Geekery's interview with Andreas Stefik is a great summary of what we actually know about the usability of programming languages—it's worth listening to the whole thing, especially the detailed discussion of studies of statically and dynamically typed languages.

Beatrice Åkerblom and Tobias Wrigstad: "Measuring Polymorphism in Python Programs". SPLASH'15, October 2015, https://people.dsv.su.se/~beatrice/python/dls15_large_images.pdf. In a break from our usual practice of quoting abstracts, here are this paper's conclusions: Our results show that while Python’s dynamic typing allows unbounded polymorphism, Python programs are predominantly monomorphic, that is, variables only hold values of a single type. This is true for program start-up and normal runtime, in library code and in program-specific code. Nevertheless, most programs have a few places which are megamorphic, meaning that variables in those places contain values of many different types at different times or in different contexts....