{"title": "Mining Internet-Scale Software Repositories", "book": "Advances in Neural Information Processing Systems", "page_first": 929, "page_last": 936, "abstract": "Large repositories of source code create new challenges and opportunities for statistical machine learning. Here we first develop an infrastructure for the automated crawling, parsing, and database storage of open source software. The infrastructure allows us to gather Internet-scale source code. For instance, in one experiment, we gather 4,632 java projects from SourceForge and Apache totaling over 38 million lines of code from 9,250 developers. Simple statistical analyses of the data first reveal robust power-law behavior for package, SLOC, and method call distributions. We then develop and apply unsupervised author-topic, probabilistic models to automatically discover the topics embedded in the code and extract topic-word and author-topic distributions. In addition to serving as a convenient summary for program function and developer activities, these and other related distributions provide a statistical and information-theoretic basis for quantifying and analyzing developer similarity and competence, topic scattering, and document tangling, with direct applications to software engineering. Finally, by combining software textual content with structural information captured by our CodeRank approach, we are able to significantly improve software retrieval performance, increasing the AUC metric to 0.86-- roughly 10-30% better than previous approaches based on text alone.", "full_text": "Mining Internet-Scale Software Repositories\n\nErik Linstead, Paul Rigor, Sushil Bajracharya, Cristina Lopes and Pierre Baldi\n\nDonald Bren School of Information and Computer Science\n\nUniversity of California, Irvine\n\nIrvine, CA 92697-3435\n\n{elinstea,prigor,sbajrach,lopes,pfbaldi}@ics.uci.edu\n\nAbstract\n\nLarge repositories of source code create new challenges and opportunities for sta-\ntistical machine learning. Here we \ufb01rst develop Sourcerer, an infrastructure for\nthe automated crawling, parsing, and database storage of open source software.\nSourcerer allows us to gather Internet-scale source code. For instance, in one ex-\nperiment, we gather 4,632 java projects from SourceForge and Apache totaling\nover 38 million lines of code from 9,250 developers. Simple statistical analyses\nof the data \ufb01rst reveal robust power-law behavior for package, SLOC, and lexical\ncontainment distributions. We then develop and apply unsupervised author-topic,\nprobabilistic models to automatically discover the topics embedded in the code\nand extract topic-word and author-topic distributions. In addition to serving as\na convenient summary for program function and developer activities, these and\nother related distributions provide a statistical and information-theoretic basis for\nquantifying and analyzing developer similarity and competence, topic scattering,\nand document tangling, with direct applications to software engineering. Finally,\nby combining software textual content with structural information captured by our\nCodeRank approach, we are able to signi\ufb01cantly improve software retrieval per-\nformance, increasing the AUC metric to 0.84\u2013 roughly 10-30% better than pre-\nvious approaches based on text alone. Supplementary material may be found at:\nhttp://sourcerer.ics.uci.edu/nips2007/nips07.html.\n\n1 Introduction\n\nLarge repositories of private or public software source code, such as the open source projects avail-\nable on the Internet, create considerable new opportunities and challenges for statistical machine\nlearning, information retrieval, and software engineering. Mining such repositories is important, for\ninstance, to understand software structure, function, complexity, and evolution, as well as to improve\nsoftware information retrieval systems and identify relationships between humans and the software\nthey produce. Tools to mine source code for functionality, structural organization, team structure,\nand developer contributions are also of interest to private industry, where these tools can be applied\nto such problems as in-house code reuse and project staf\ufb01ng. While some progress has been made\nin the application of statistics and machine learning techniques to mine software corpora, empirical\nstudies have typically been limited to small collections of projects, often on the order of one hundred\nprojects or less, several orders of magnitude smaller than publicly available repositories(eg. [1]).\nMining large software repositories requires leveraging both the textual and structural aspects of soft-\nware data, as well as any relevant meta data. Here we develop Sourcerer, a large-scale infrastructure\nto explore such aspects. We \ufb01rst identify a number of robust power-law behaviors by simple statisti-\ncal analyses. We then develop and apply unsupervised author-topic probabilistic models to discover\nthe topics embedded in the code and extract topic-word and author-topic distributions. Finally, we\nleverage the dual textual and graphical nature of software to improve code search and retrieval.\n\n\f2\n\nInfrastructure and Data\n\nTo allow for the Internet-scale analysis of source code we have built Sourcerer, an extensive infras-\ntructure designed for the automated crawling, downloading, parsing, organization, and storage of\nlarge software repositories in a relational database. A highly con\ufb01gurable crawler allows us to spec-\nify the number and types of projects desired, as well as the host databases that should be targeted,\nand to proceed with incremental updates in an automated fashion. Once target projects are down-\nloaded, a depackaging module uncompresses archive \ufb01les while saving useful metadata (project\nname, version, etc). While the infrastructure is general, we apply it here to a sample of projects\nin Java. Speci\ufb01cally, for the results reported, we download 12,151 projects from Sourceforge and\nApache and \ufb01lter out distributions packaged without source code (binaries only). The end result is\na repository consisting of 4,632 projects, containing 244,342 source \ufb01les, with 38.7 million lines\nof code, written by 9,250 developers. For the software author-topic modeling approach we also\nemploy the Eclipse 3.0 source code as a baseline. Though only a single project, Eclipse is a large,\nactive open source effort that has been widely studied. In this case, we consider 2,119 source \ufb01les,\nassociated with about 700,000 lines of code, a vocabulary of 15,391 words, and 59 programmers.\nMethods for extracting and assigning words and programmers to documents are described in the\nnext sections. A complete list of all the projects contained in our repository is available from the\nsupplementary materials web pages.\n\n3 Statistical Analysis\n\nDuring the parsing process our system performs a static analysis on project source code \ufb01les to\nextract code entities and their relationships, storing them in a relational database. For java these en-\ntities consist of packages, classes, interfaces, methods, and \ufb01elds, as well as more speci\ufb01c constructs\nsuch as constructors and static initializers. Relations capture method calls, inheritance, and encap-\nsulation, to name a few. The populated database represents a substantial foundation on which to\nbase statistical analysis of source code. Parsing the multi-project repository described above yields\na repository of over 5 million entities organized into 48 thousand packages, 560 thousand classes,\nand 3.2 million methods, participating in over 23.4 million relations. By leveraging the query capa-\nbilities of the underlying database we can investigate other interesting statistics. For example, table\n1 contains the frequencies of Java keywords across all 4,632 projects. Upon examining this data we\ncan see that the \u2019default\u2019 keyword occurs about 6 percent less frequently than the \u2019switch\u2019 keyword,\ndespite the fact that best practice typically mandates all switch statements contain a default block.\nMoreover, the \u2019for\u2019 loop is about twice as pervasive as the \u2019while\u2019 loop, suggesting that the bound on\nthe number of iterations is more likely to be known or based on the size of a known data structure.\n\nTable 1: Frequency of java keyword occurrence\n\nPercentage Keyword\nboolean\n\nKeyword\n\npublic\n\nif\nnew\nreturn\nimport\n\nint\nnull\nvoid\nprivate\nstatic\n\ufb01nal\nelse\nthrows\n\n12.53\n8.44\n8.39\n7.69\n6.89\n6.54\n5.52\n4.94\n3.66\n3.16\n3.01\n2.33\n2.16\n\nPercentage\n\n2.12\n1.69\n1.60\n1.60\n1.36\n1.33\n1.33\n1.22\n1.22\n1.16\n0.96\n0.93\n0.89\n\nKeyword\n\nthis\nbreak\nwhile\nsuper\n\ninstanceof\n\ndouble\nlong\n\nimplements\n\nchar\n\ufb02oat\n\nabstract\n\nsynchronized\n\nshort\n\nfalse\ncase\ntrue\nclass\n\nprotected\n\ncatch\nfor\ntry\nthrow\npackage\n\nbyte\n\nextends\n\nPercentage Keyword\nswitch\ninterface\ncontinue\n\ufb01nally\ndefault\nnative\ntransient\n\ndo\n\nassert\nenum\nvolatile\nstrictfp\n\n0.89\n0.85\n0.63\n0.57\n0.56\n0.55\n0.54\n0.43\n0.30\n0.28\n0.25\n0.25\n0.20\n\nPercentage\n\n0.19\n0.17\n0.15\n0.14\n0.13\n0.08\n0.06\n0.05\n0.03\n0.02\n0.04\n\n2.49E-06\n\nFinally, statistical analyses of distributions also identify several power-law distributions. We have\nobserved power-law distributions governing package, SLOC, and inside relation (lexical contain-\n\n\fment) counts. For instance, Figure 1 shows the log-log plots for the number of packages across\nprojects. Similar graphs for other distributions are available from the supplemental materials page.\n\nFigure 1: Approximate power-law distribution for packages over projects\n\n4 Topic and Author-Topic Probabilistic Modeling of Source Code\n\nAutomated topic and author-topic modeling have been successfully used in text mining and infor-\nmation retrieval where they have been applied, for instance, to the problem of summarizing large\ntext corpora. Recent techniques include Latent Dirichlet Allocation (LDA), which probabilistically\nmodels text documents as mixtures of latent topics, where topics correspond to key concepts pre-\nsented in the corpus [2] (see also [3]). Author-Topic (AT) modeling is an extension of topic modeling\nthat captures the relationship of authors to topics in addition to extracting the topics themselves. An\nextension of LDA to probabilistic AT modeling has been developed in [4]. In the literature [5],\nthese more recent approaches have been found to produce better results than more traditional meth-\nods such as latent semantic analysis (LSA) [6]. Despite previous work in classifying code based\non concepts [1], applications of LDA and AT models have been limited to traditional text corpora\nsuch as academic publications, news reports, corporate emails, and historical documents [7, 8]. At\nthe most basic level, however, a code repository can be viewed as a text corpus, where source \ufb01les\nare analogous to documents and developers to authors. Though vocabulary, syntax, and conventions\ndifferentiate a programming language from a natural language, the tokens present in a source \ufb01le\nare still indicative of its function (ie. its topics). Thus here we develop and apply probabilistic AT\nmodels to software data.\nIn AT models for text, the data consists of a set of documents. The authors of each documents are\nknown and each document is treated as a bag of words. We let A be the total number of authors, W\nthe total number of distinct words (vocabulary size), and T the total number of topics present in the\ndocuments. While non-parametric Bayesian [9] and other [10] methods exist to try to infer T from\nthe data, here we assume that T is \ufb01xed (e.g. T = 100), though we explore different values.\nAs in [7], our model assumes that each topic t is associated with a multinomial distribution \u03c6\u2022t over\nwords w, and each author a is associated with a multinomial distribution \u03b8\u2022a over topics. More\nprecisely, the parameters are given by two matrices: a T \u00d7 A matrix \u0398 = (\u03b8ta) of author-topic\ndistributions, and a W \u00d7 T matrix \u03a6 = (\u03c6wt) of topic-word distributions. Given a document d\ncontaining Nd words with known authors, in generative mode each word is assigned to one of the\nauthors a of the document uniformly, then the corresponding \u03b8\u2022a is sampled to derive a topic t, and\n\ufb01nally the corresponding \u03c6\u2022t is sampled to derive a word w. A fully Bayesian model is derived by\nputting symmetric Dirichlet priors with hyperparameters \u03b1 and \u03b2 over the distributions \u03b8\u2022a and \u03c6\u2022t.\nSo for instance the prior on \u03b8\u2022a is given by\n\nand similarly for \u03c6\u2022t. If A is the set of authors of the corpus and document d has Ad authors, it is\neasy to see that under these assumptions the likelihood of a document is given by:\n\nD\u03b1(\u03b8\u2022a) =\n\n\u0393(T \u03b1)\n(\u0393(\u03b1))T\n\n\u03b8\u03b1\u22121\n\nta\n\nT(cid:89)\n\nt=1\n\nP (d|\u0398, \u03a6,A) =\n\n\u03c6wit\u03b8ta\n\nNd(cid:89)\n\n(cid:88)\n\nT(cid:88)\n\n1\nAd\n\ni=1\n\na\n\nt=1\n\n100101102103104100101102103Number of PackagesRankDistribution of Packages over Projects\fwhich can be integrated over \u03c6 and \u03b8 and their Dirichlet distributions to get P (d|\u03b1, \u03b2,A). The\nposterior can be sampled ef\ufb01ciently using Markov Chain Monte Carlo Methods (Gibbs sampling)\nand, for instance, the \u0398 and \u03a6 parameter matrices can be estimated by MAP or MPE methods.\nOnce the data is obtained, applying this basic AT model to software requires the development of\nseveral tools to facilitate the processing and modeling of source code. In addition to the crawling\ninfrastructure described above, the primary functions of the remaining tools are to extract and resolve\nauthor names from source code, as well as convert the source code to the bag-of-words format.\n\n4.1\n\nInformation Extraction from Source Code\n\nAuthor-Document: The author-document matrix is produced from the output of our author extrac-\ntion tool. It is a binary matrix where entry [i,j]=1 if author i contributed to document j, and 0\notherwise. Extracting author information is ultimately a matter of tokenizing the code and associat-\ning developer names with \ufb01le (document) names when this information is available. This process is\nfurther simpli\ufb01ed for java software due to the prevalence of javadoc tags which present this metadata\nin the form of attribute-value pairs.\nExploratory analysis of the Eclipse 3.0 code base, however, shows that most source \ufb01les are credited\nto \u201cThe IBM Corporation\u201d rather than speci\ufb01c developers. Thus, to generate a list of authors for\nspeci\ufb01c source \ufb01les, we parsed the Eclipse bug data available in [11]. After pruning \ufb01les not\nassociated with any author, this input dataset consists of 2,119 Java source \ufb01les, comprising 700,000\nlines of code, from a total of 59 developers.\nWhile leveraging bug data is convenient (and necessary) to generate the developer list for Eclipse\n3.0, it is also desirable to develop a more \ufb02exible approach that uses only the source code itself,\nand not other data sources. Thus to extract author names from source code we also develop a\nlightweight parser that examines the code for javadoc \u2019@author\u2019 tags, as well as free form labels such\nas \u2019author\u2019 and \u2019developer.\u2019 Occurrences of these labels are used to isolate and identify developer\nnames. Ultimately author identi\ufb01ers may come in the form of full names, email addresses, url\u2019s,\nor CVS account names. This multitude of formats, combined with the fact that author names are\ntypically labeled in the code header, is key to our decision to extract developer names using our own\nparsing utilities, rather than part-of-speech taggers [12] leveraged in other text mining projects.\nA further complication for author name extraction is the fact that the same developer may write\nhis name in several different ways. For example, \u201cJohn Q. Developer\u201d alternates between \u201cJohn\nDeveloper,\u201d \u201cJ. Q. Developer,\u201d or simply \u201cDeveloper.\u201d To account for this effect, we implement\nalso a two-tiered approach to name resolution using the q-gram algorithm [13]. When an individual\nproject is parsed, a list of contributing developers (and the \ufb01les they modi\ufb01ed) is created. A pairwise\ncomparison of author-names is then performed using q-gram similarity, and pairs of names whose\nsimilarity is greater than a threshold t1 are merged. This process continues until all pairwise simi-\nlarities are below the threshold, and the project list is then added to a global list of authors. When\nparsing is complete for all projects, the global author list is resolved using the same process, but\nwith a new threshold, t2, such that t2 > t1. This approach effectively implements more conser-\nvative name resolution across projects in light of the observation that the scope of most developer\nactivities is limited to a relatively small number (1 in many cases) of open source efforts. In prac-\ntice, we set t1 = .65 and t2 = .75. Running our parser on the multi-project repository yields 9,250\ndistinct authors respectively.\nWord-Document: To produce the word-document matrix for our input data we have developed a\ncomprehensive tokenization tool tuned to the Java programming language. This tokenizer includes\nlanguage-speci\ufb01c heuristics that follow the commonly practiced naming conventions. For example,\nthe Java class name \u201cQuickSort\u201d will generate the words \u201cquick\u201d and \u201csort\u201d. All punctuation is\nignored. As an important step in processing source \ufb01les our tool removes commonly occurring stop\nwords. We augment a standard list of stop words used for the English language (e.g. and, the, but,\netc) to include the names of all classes from the Java SDK (eg. ArrayList, HashMap, etc). This is\ndone to speci\ufb01cally avoid extracting common topics relating to the Java collections framework.We\nrun the LDA-based AT algorithm on the input matrices and set the total number of topics (100)\nand the number of iterations by experimentation. For instance, the number of iterations, i, to run the\nalgorithm is determined empirically by analyzing results for i ranging from 500 to several thousands.\nThe results presented in the next section are derived using 3,000 iterations, which were found to\n\n\fproduce interpretable topics in a reasonable amount of time (a week or so). Because the algorithm\ncontains a stochastic component we also veri\ufb01ed the stability of the results across multiple runs.\n\n4.2 Topic and Author-Topic Modeling Results\n\nA representative subset of 6 topics extracted via Author-Topic modeling on the selected 2,119 source\n\ufb01les from Eclipse 3.0 is given in Table 2. Each topic is described by several words associated with\nthe topic concept. To the right of each topic is a list of the most likely authors for each topic with\ntheir probabilities. Examining the topic column of the table it is clear that various functions of the\nEclipse framework are represented. For example, topic 1 clearly corresponds to unit testing, topic\n2 to debugging, topic 4 to building projects, and topic 6 to automated code completion. Remaining\ntopics range from package browsing to compiler options.\n\nTable 2: Representative topics and authors from Eclipse 3.0\n\n#\n\n1\n\n2\n\n3\n\nTopic\njunit\nrun\n\nlistener\nitem\nsuite\ntarget\nsource\ndebug\n\nbreakpoint\nlocation\n\nast\n\nbutton\ncplist\nentries\nastnode\n\nAuthor Probabilities\n\negamma 0.97065\nwmelhem 0.01057\n\ndarin 0.00373\n\nkrbarnes 0.00144\nkkolosow 0.00129\njaburns 0.96894\ndarin 0.02101\n\nlbourlier 0.00168\ndarins 0.00113\njburns 0.00106\nmaeschli 0.99161\nmkeller 0.00097\nothomann 0.00055\ntmaeder 0.00055\nteicher 0.00046\n\n#\n\n4\n\n5\n\n6\n\nTopic\nnls-1\nant\n\nmanager\nlistener\nclasspath\n\ntype\nlength\nnames\nmatch\nmethods\ntoken\n\ncompletion\n\ncurrent\nidenti\ufb01er\n\nassist\n\nAuthor Probabilities\n\ndarins 0.99572\ndmegert 0.00044\n\nnick 0.00044\n\nkkolosow 0.00036\nmaeschli 0.00031\nkjohnson 0.59508\njlanneluc 0.32046\n\ndarin 0.02286\njohna 0.00932\npmulet 0.00918\ndaudel 0.99014\nteicher 0.00308\njlanneluc 0.00155\ntwatson 0.00084\ndmegert 0.00046\n\nTable 3 presents 6 representative author-topic assignments from the multi-project repository. This\ndataset yields a substantial increase in topic diversity. Topics representing major sub-domains of\nsoftware development are clearly represented, with the \ufb01rst topic corresponding to web applica-\ntions, the second to databases, the third to network applications, and the fourth to \ufb01le processing.\nTopics 5 and 6 are especially interesting, as they correspond to common examples of crosscutting\nconcerns from aspect-oriented programming [14], namely security and logging. Topic 5 is also\ndemonstrative of the inherent dif\ufb01culty of resolving author names, and the shortcomings of the q-\ngram algorithm, as the developer \u201cgert van ham\u201d and the developer \u201chamgert\u201d are most likely the\nsame person documenting their name in different ways.\nSeveral trends reveal themselves when all results are considered. Though the majority of topics\ncan be intuitively mapped to their corresponding domains, some topics are too noisy to be able to\nassociate any functional description to them. For example, one topic extracted from our repository\nconsists of Spanish words unrelated to software engineering which seem to represent the subset\nof source \ufb01les with comments in Spanish. Other topics appear to be very project speci\ufb01c, and\nwhile they may indeed describe a function of code, they are not easily understood by those who\nare only casually familiar with the software artifacts in the codebase. This is especially true with\nEclipse, which is limited in both the number and diversity of source \ufb01les. In general noise appears to\ndiminish as repository size grows. Noise can be controlled to some degree with tuning the number\nof topics to be extracted, but of course can not be eliminated completely.\nExamining the author assignments (and probabilities) for the various topics provides a simple means\nby which to discover developer contributions and infer their competencies. It should come as no\nsurprise that the most probable developer assigned to the JUnit framework topic is \u201cegamma\u201d, or\nErich Gamma.\nIn this case, there is a 97% chance that any source \ufb01le in our dataset assigned\nto this topic will have him as a contributor. Based on this rather high probability, we can also\ninfer that he is likely to have extensive knowledge of this topic. This is of course a particularly\n\n\fTable 3: Representative topics and authors from the multi-project repository\n\n#\n\n1\n\n2\n\n3\n\nTopic\nservlet\nsession\nresponse\nrequest\n\nhttp\nsql\n\ncolumn\n\njdbc\ntype\nresult\npacket\ntype\nsession\n\nsnmpwalkmv\n\naddress\n\nAuthor Probabilities\n\ncraig r mcclanahan 0.19147\nremy maucherat 0.08301\npeter rossbach 0.04760\ngreg wilkins 0.04251\n\namy roh 0.03100\n\nmark matthews 0.33265\n\names 0.02640\n\nmike bowler 0.02033\n\nmanuel la\ufb02amme 0.02027\n\ngavin king 0.01813\nbrian weaver 0.14015\n\napache directory project 0.10066\n\nopennms 0.08667\n\nmatt whitlock 0.06508\n\ntrustin lee 0.04752\n\n#\n\n4\n\n5\n\n6\n\nTopic\n\ufb01le\npath\ndir\n\ndirectory\nstream\ntoken\nkey\n\nsecurity\nparam\ncert\n\nservice\n\nstr\nlog\n\ncon\ufb01g\nresult\n\nmatthew hawthorne 0.01170\n\nlk 0.01106\n\nwerner dittmann 0.09409\n\napache software foundation 0.06117\n\ngert van ham 0.05153\n\nhamgert 0.05144\n\njcetaglib.sourceforge.net 0.05133\n\nAuthor Probabilities\nadam murdoch 0.02466\npeter donald 0.02056\nludovic claude 0.01496\n\nwayne m osse 0.44638\ndirk mascher 0.07339\ndavid irwin 0.04928\n\nlinke 0.02823\njason 0.01505\n\nattractive example because Erich Gamma is widely known for being a founder of the JUnit project,\na fact which lends credibility to the ability of the topic modeling algorithm to assign developers to\nreasonable topics. One can interpret the remaining author-topic assignments along similar lines. For\nexample, developer \u201cdaudel\u201d is assigned to the topic corresponding to automatic code completion\nwith probability .99. Referring back to the Eclipse bug data it is clear that the overwhelming majority\nof bug \ufb01xes for the codeassist framework were made by this developer. One can infer that this is\nlikely to be an area of expertise of the developer.\nIn addition to determining developer contributions, one may also be curious to know the scope\nof a developer\u2019s involvement. Does a developer work across application areas, or are his contri-\nbutions highly focused? How does the breadth of one developer compare to another? These are\nnatural questions that arise in the software development process. To answer these questions within\nthe framework of author-topic models, we can measure the breadth of an author a by the entropy\nt \u03b8ta log \u03b8ta of the corresponding distribution over topics. Applying the measure to\nour multi-project dataset, we \ufb01nd that the average measure is 2.47 bits. The developer with the low-\nest entropy is \u201cthierry danard,\u201d with .00076 bits. The developer with the highest entropy is \u201cwdi\u201d\nwith 4.68 bits, with 6.64 bits being the maximum possible score for 100 topics. While the entropy\n\nH(a) = \u2212(cid:80)\n\nFigure 2: All 59 Eclipse 3.0 authors clustered by KL divergence\n\nof the distribution of an author over topics measures the author\u2019s breadth, the similarity between two\nauthors can be measured by comparing their respective distributions over topics. Several metrics\nare possible for this purpose, but one of the most natural measures is provided by the symmetrized\nKullback-Leibler (KL) divergence. Multidimensional scaling (MDS) is employed to further visual-\n\negammajeromelkjohnsondmegertkmaetzelcwongptfflbourlierjfogellprapicaudwilsonjburnsmaeschlimannkkolosowbbaumgartakiezundaudelmkellermrenniejaburnsdarinsothomannmfarajsfranklinjohnajeemdejantmaederaweinandmvanmeektodbbokowskitwatsonkhornedpollockoliviertbbiggsdarinjeffdbirsankrbarnesffusierikhelifisxenospmuletjdesriviereswmelhemschanrchavesmaeschlidjcmartisarsenaukentteicherjlanneluctwidmerdbaeumernick\fize author similarities, resulting in Figure 2 for the Eclipse project. The boxes represent individual\ndevelopers, and are arranged such that developers with similar topic distributions are nearest one an-\nother. A similar \ufb01gure, displaying only a subset of the 4,500 SourceForge and Apache authors due to\nspace and legibility constraints, is available in the supplementary materials. This information is es-\npecially useful when considering how to form a development team, choosing suitable programmers\nto perform code updates, or deciding to whom to direct technical questions. Two other important\ndistributions that can be retrieved from the AT modeling approach are the distribution of topics\nacross documents, and the distribution of documents across topics (not shown). The corresponding\nentropies provide an automated and novel way to precisely formalize and measure topic scattering\nand document tangling, two fundamental concepts of software design [14], which are important to\nsoftware architects when performing activities such as code refactoring.\n\n5 Code Search and Retrieval\n\nSourcerer relies on a deep analysis of code to extract pertinent textual and structural features that can\nbe used to improve the quality and performance of source code search, as well as augment the ways\nin which code can be searched. By combining standard text information retrieval techniques with\nsource-speci\ufb01c heuristics and a relational representation of code, we have available a comprehensive\nplatform for searching software components. While there has been progress in developing source-\ncode-speci\ufb01c search engines in recent years (e.g. Koders, Krugle, and Google\u2019s CodeSearch), these\nsystems continue to focus strictly on text information retrieval, and do not appear to leverage the\ncopious relations that can be extracted and analyzed from code.\nPrograms are best modeled as graphs, with code entities comprising the nodes and various relations\nthe edges. As such, it is worth exploring possible ranking methods that leverage the underlying\ngraphs. A natural starting point is Google\u2019s PageRank [15], which considers hyperlinks to formulate\na notion of popularity among web pages. This can be applied to source as well, as it is likely that a\ncode entity referenced by many other entities are more robust than those with few references.\nWe used Google\u2019s PageRank [15] almost verbatim. The Code Rank of a code entity (package, class,\nor method) A is given by: CR(A) = (1 \u2212 d) + d(CR(T1)/C(T1) + ... + CR(Tn)/C(Tn)) where\nT1...Tn are the code entities referring to A, C(A) is the number of outgoing links of A, and d is a\ndamping factor.\nUsing the CodeRank algorithm as a basis it is possible to devise many ranking schemes by building\ngraphs from the many entities and relations stored in our database, or subsets thereof. For example,\none may consider the graph of only method call relationships, package dependencies, or inheritance\nhierarchies. Moreover, graph-based techniques can be combined with a variety of heuristics to\nfurther improve code search. For example, keyword hits to the right of the fully-quali\ufb01ed name can\nbe boosted, hits in comments can be discounted, and terms indicative of test articles can be ignored.\nWe are conducting detailed experiments to assess the effectiveness of graph-based algorithms in con-\njunction with standard IR techniques to search source code. Current evidence strongly indicates that\nbest results are ultimately obtained by combining term-based ranking with source-speci\ufb01c heuris-\ntics and coderank. After de\ufb01ning a set of 25 control queries with known \u201dbest\u201d hits, we compared\nperformances using standard information retrieval metrics, such as area under curve (AUC). Queries\nwere formulated to represent users searching for speci\ufb01c algorithms, such as \u2019depth \ufb01rst search,\u2019 as\nwell as users looking to reuse complete components, such as \u2019database connection manager.\u2019 Best\nhits were determined manually with a team of 3 software engineers serving as human judges of re-\nsult quality, modularity, and ease of reuse. Results clearly indicate that the general Google search\nengine is ineffective for locating relevant source code, with a mean AUC of .31 across the queries.\nBy restricting its corpus to code alone, Google\u2019s code search engine yields substantial improvement\nwith an AUC of approximately .66. Despite this improvement this system essentially relies only\non regular expression matching of code keywords. Using a Java-speci\ufb01c keyword and comment\nparser our infrastructure yields an immediate improvement with an AUC of .736. By augmenting\nthis further with the heuristics above and CodeRank (consisting of class and method relations), the\nmean AUC climbs to .841. At this time we have conducted extensive experiments for 12 ranking\nschemes corresponding to various combinations of graph-based and term-based heuristics, and have\nobserved similar improvements. While space does not allow their inclusion, additional results are\navailable from our supplementary materials page.\n\n\f6 Conclusion\n\nHere we have leveraged a comprehensive code processing infrastructure to facilitate the mining of\nlarge-scale software repositories. We conduct a statistical analysis of source code on a previously un-\nreported scale, identifying robust power-law behavior among several code entities. The development\nand application of author-topic probabilistic modeling to source code allows for the unsupervised ex-\ntraction of program organization, functionality, developer contributions, and developer similarities,\nthus providing a new direction for research in this area of software engineering. The methods de-\nveloped are applicable at multiple scales, from single projects to Internet-scale repositories. Results\nindicate that the algorithm produces reasonable and interpretable automated topics and author-topic\nassignments. The probabilistic relationships between author, topics, and documents that emerge\nfrom the models naturally provide an information-theoretic basis to de\ufb01ne and compare developer\nand program similarity, topic scattering, and document tangling with potential applications in soft-\nware engineering ranging from bug \ufb01x assignments and staf\ufb01ng to software refactoring. Finally,\nby combining term-based information retrieval techniques with graphical information derived from\nprogram structure, we are able to signi\ufb01cantly improve software search and retrieval performance.\nAcknowledgments: Work in part supported by NSF MRI grant EIA-0321390 and a Microsoft\nFaculty Research Award to PB, as well as NSF grant CCF-0725370 to CL and PB.\n\nReferences\n[1] S. Ugurel, R. Krovetz, and C. L. Giles. What\u2019s the code?: automatic classi\ufb01cation of source code archives.\nIn KDD \u201902: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery\nand data mining, pages 632\u2013638, New York, NY, USA, 2002. ACM Press.\n\n[2] D.M. Blei, A.Y. Ng, and M.I. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research,\n\n3:993\u20131022, January 2003.\n\n[3] W. Buntine. Open source search: a data mining platform. SIGIR Forum, 39(1):4\u201310, 2005.\n[4] M. Steyvers, P. Smyth, M. Rosen-Zvi, and T. Grif\ufb01ths. Probabilistic author-topic models for information\ndiscovery. In KDD \u201904: Proceedings of the tenth ACM SIGKDD international conference on Knowledge\ndiscovery and data mining, pages 306\u2013315, New York, NY, USA, 2004. ACM Press.\n\n[5] D. Newman, C. Chemudugunta, P. Smyth, and M. Steyvers. Analyzing entities and topics in news articles\n\nusing statistical topic models. In ISI, pages 93\u2013104, 2006.\n\n[6] S. Deerwester, S. Dumais, T. Landauer, G. Furnas, and R. Harshman. Indexing by latent semantic analysis.\n\nJournal of the American Society of Information Science, 41(6):391\u2013407, 1990.\n\n[7] Michal Rosen-Zvi, Thomas Grif\ufb01ths, Mark Steyvers, and Padhraic Smyth. The author-topic model for\nIn UAI \u201904: Proceedings of the 20th conference on Uncertainty in arti\ufb01cial\n\nauthors and documents.\nintelligence, pages 487\u2013494, Arlington, Virginia, United States, 2004. AUAI Press.\n\n[8] D. Newman and S. Block. Probabilistic topic decomposition of an eighteenth-century american newspa-\n\nper. J. Am. Soc. Inf. Sci. Technol., 57(6):753\u2013767, 2006.\n\n[9] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet processes. Journal of the\n\nAmerican Statistical Association, 101(476):1566\u20131581, 2006.\n\n[10] T. L. Grif\ufb01ths and M. Steyvers. Finding scienti\ufb01c topics. Proc Natl Acad Sci U S A, 101 Suppl 1:5228\u2013\n\n5235, April 2004.\n\n[11] A. Schr\u00a8oter, T. Zimmermann, R. Premraj, and A. Zeller. If your bug database could talk. . . . In Proceedings\nof the 5th International Symposium on Empirical Software Engineering, Volume II: Short Papers and\nPosters, pages 18\u201320, September 2006.\n\n[12] E. Brill. Some advances in transformation-based part of speech tagging.\n\nArti\ufb01cial Intelligence, pages 722\u2013727, 1994.\n\nIn National Conference on\n\n[13] E. Ukkonen. Approximate string-matching with q-grams and maximal matches. Theor. Comput. Sci.,\n\n92(1):191\u2013211, 1992.\n\n[14] G. Kiczales, J. Lamping, A. Menhdhekar, C. Maeda, C. Lopes, J. Loingtier, and J. Irwin. Aspect-oriented\nprogramming. In Mehmet Aks\u00b8it and Satoshi Matsuoka, editors, Proceedings European Conference on\nObject-Oriented Programming, volume 1241, pages 220\u2013242. Springer-Verlag, Berlin, Heidelberg, and\nNew York, 1997.\n\n[15] R. Motwani L. Page, S. Brin and T. Winograd.\n\nThe pagerank citation ranking: Bringing or-\nStanford Digital Library working paper SIDL-WP-1999-0120 of 11/11/1999 (see:\n\nder to the web.\nhttp://dbpubs.stanford.edu/pub/1999-66).\n\n\f", "award": [], "sourceid": 637, "authors": [{"given_name": "Erik", "family_name": "Linstead", "institution": null}, {"given_name": "Paul", "family_name": "Rigor", "institution": null}, {"given_name": "Sushil", "family_name": "Bajracharya", "institution": null}, {"given_name": "Cristina", "family_name": "Lopes", "institution": null}, {"given_name": "Pierre", "family_name": "Baldi", "institution": null}]}