{"title": "Neurally-Guided Procedural Models: Amortized Inference for Procedural Graphics Programs using Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 622, "page_last": 630, "abstract": "Probabilistic inference algorithms such as Sequential Monte Carlo (SMC) provide powerful tools for constraining procedural models in computer graphics, but they require many samples to produce desirable results. In this paper, we show how to create procedural models which learn how to satisfy constraints. We augment procedural models with neural networks which control how the model makes random choices based on the output it has generated thus far. We call such models neurally-guided procedural models. As a pre-computation, we train these models to maximize the likelihood of example outputs generated via SMC. They are then used as efficient SMC importance samplers, generating high-quality results with very few samples. We evaluate our method on L-system-like models with image-based constraints. Given a desired quality threshold, neurally-guided models can generate satisfactory results up to 10x faster than unguided models.", "full_text": "Neurally-Guided Procedural Models:\n\nAmortized Inference for Procedural Graphics\n\nPrograms using Neural Networks\n\nDaniel Ritchie\n\nStanford University\n\nAnna Thomas\n\nStanford University\n\nPat Hanrahan\n\nStanford University\n\nNoah D. Goodman\nStanford University\n\nAbstract\n\nProbabilistic inference algorithms such as Sequential Monte Carlo (SMC) provide\npowerful tools for constraining procedural models in computer graphics, but they\nrequire many samples to produce desirable results. In this paper, we show how\nto create procedural models which learn how to satisfy constraints. We augment\nprocedural models with neural networks which control how the model makes\nrandom choices based on the output it has generated thus far. We call such models\nneurally-guided procedural models. As a pre-computation, we train these models\nto maximize the likelihood of example outputs generated via SMC. They are then\nused as ef\ufb01cient SMC importance samplers, generating high-quality results with\nvery few samples. We evaluate our method on L-system-like models with image-\nbased constraints. Given a desired quality threshold, neurally-guided models can\ngenerate satisfactory results up to 10x faster than unguided models.\n\n1\n\nIntroduction\n\nProcedural modeling, or the use of randomized procedures to generate computer graphics, is a\npowerful technique for creating visual content. It facilitates ef\ufb01cient content creation at massive scale,\nsuch as procedural cities [13]. It can generate \ufb01ne detail that would require painstaking effort to\ncreate by hand, such as decorative \ufb02oral patterns [24]. It can even generate surprising or unexpected\nresults, helping users to explore large or unintuitive design spaces [19].\nMany applications demand control over procedural models: making their outputs resemble ex-\namples [22, 2], \ufb01t a target shape [17, 21, 20], or respect functional constraints such as physical\nstability [19]. Bayesian inference provides a general-purpose control framework: the procedural\nmodel speci\ufb01es a generative prior, and the constraints are encoded as a likelihood function. Posterior\nsamples can then be drawn via Markov Chain Monte Carlo (MCMC) or Sequential Monte Carlo\n(SMC). Unfortunately, these algorithms often require many samples to converge to high-quality\nresults, limiting their usability for interactive applications. Sampling is challenging because the\nconstraint likelihood implicitly de\ufb01nes complex (often non-local) dependencies not present in the\nprior. Can we instead make these dependencies explicit by encoding them in a model\u2019s generative\nlogic? Such an explicit model could simply be run forward to generate high-scoring results.\nIn this paper, we propose an amortized inference method for learning an approximation to this perfect\nexplicit model. Taking inspiration from recent work in amortized variational inference, we augment\nthe procedural model with neural networks that control how the model makes random choices based\non the partial output it has generated. We call such a model a neurally-guided procedural model.\nWe train these models by maximizing the likelihood of example outputs generated via SMC using a\nlarge number of samples, as an of\ufb02ine pre-process. Once trained, they can be used as ef\ufb01cient SMC\nimportance samplers. By investing time up-front generating and training on many examples, our\nsystem effectively \u2018pre-compiles\u2019 an ef\ufb01cient sampler that can generate further results much faster.\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\fFor a given likelihood threshold, neurally-guided models can generate results which reliably achieve\nthat threshold using 10-20x fewer particles and up to 10x less compute time than an unguided model.\nIn this paper, we focus on accumulative procedural models that repeatedly add new geometry to a\nstructure. For our purposes, a procedural model is accumulative if, while executing, it provides a\n\u2018current position\u2019 p from which geometry generation will continue. Many popular growth models,\nsuch as L-systems, are accumulative [16]. We focus on 2D models (p \u2208 R2) which generate images,\nthough the techniques we present extend naturally to 3D.\n\n2 Related Work\n\nGuided Procedural Modeling Procedural models can be guided using non-probabilistic methods.\nOpen L-systems can query their spatial position and orientation, allowing them to prune their growth\nto an implicit surface [17, 14]. Recent follow-up work supports larger models by decomposing them\ninto separate regions with limited interaction [1]. These methods were speci\ufb01cally designed to \ufb01t\nprocedural models to shapes. In contrast, our method learns how to guide procedural models and is\ngenerally applicable to constraints expressable as likelihood functions.\n\nGeneratively Capturing Dependencies in Procedural Models A recent system by Dang et al.\nmodi\ufb01es a procedural grammar so that its output distribution re\ufb02ects user preference scores given\nto example outputs [2]. Like us, they use generative logic to capture dependencies induced by a\nlikelihood function (in their case, a Gaussian process regression over user-provided examples). Their\nmethod splits non-terminal symbols in the original grammar, giving it more combinatorial degrees\nof freedom. This works well for discrete dependencies, whereas our method is better suited for\ncontinuous constraint functions, such as shape-\ufb01tting.\n\nNeural Amortized Inference Our method is also inspired by recent work in amortized variational\ninference using neural variational families [11, 18, 8], but it uses a different learning objective.\nPrior work has also aimed to train ef\ufb01cient neural SMC importance samplers [5, 15]. These efforts\nfocused on time series models and Bayesian networks, respectively; we focus on a class of structured\nprocedural models, the characteristics of which permit different design decisions:\n\n\u2022 The likelihood of a partially-generated output can be evaluated at any time and is a good\nheuristic for the likelihood of the completed output. This is different from e.g. time series\nmodels, where the likelihood at each step considers a previously-unseen data point.\n\n\u2022 They make many local random choices but have no global/top-level parameters.\n\u2022 They generate images, which naturally support coarse-to-\ufb01ne feature extraction.\n\nThese properties informed the design of our neurally-guided model architecture.\n\n3 Approach\n\nConsider a simple procedural modeling program chain that recursively generates a random sequence\nof linear segments, constrained to match a target image. Figure 1a shows the text of this program,\nalong with samples generated from it (drawn in black) against several target images (drawn in gray).\nChains generated by running the program forward do not match the targets, since forward sampling\nis oblivious to the constraint. Instead, we can generate constrained samples using Sequential Monte\nCarlo (SMC) [20]. This results in \ufb01nal chains that more closely match the target images. However,\nthe algorithm requires many particles\u2014and therefore signi\ufb01cant computation\u2014to produce acceptable\nresults. Figure 1a shows that N = 10 particles is not suf\ufb01cient.\nIn an ideal world, we would not need costly inference algorithms to generate constraint-satisfying\nresults. Instead, we would have access to an \u2018oracle\u2019 program, chain_perfect, that perfectly \ufb01lls in the\ntarget image when run forward. While such an oracle can be dif\ufb01cult or impossible to write by hand,\nit is possible to learn a program chain_neural that comes close. Figure 1b shows our approach. For\neach random choice in the program text (e.g. gaussian, flip), we replace the parameters of that choice\nwith the output of a neural network. This neural network\u2019s inputs (abstracted as \u201c...\u201d) include the\ntarget image as well the partial output image the program has generated thus far. The network thus\n\n2\n\n\ffunction chain(pos, ang) {\n\nfunction chain_neural(pos, ang) {\n\nvar newang = ang + gaussian(0, PI/8);\nvar newpos = pos + polarToRect(LENGTH, newang);\ngenSegment(pos, newpos);\nif (flip(0.5)) chain(newpos, newang);\n\nvar newang = ang + gaussMixture(nn1(...));\nvar newpos = pos + polarToRect(LENGTH, newang);\ngenSegment(pos, newpos);\nif (flip(nn2(...))) chain_neural(newpos, newang);\n\n}\n\nForward\nSamples\n\nSMC\nSamples\n(N = 10)\n\n}\n\nForward\nSamples\n\nSMC\nSamples\n(N = 10)\n\n(a)\n\n(b)\n\nFigure 1: Turning a linear chain model into a neurally-guided model. (a) The original program. When\noutputs (shown in black) are constrained to match a target image (shown in gray), SMC requires many\nparticles to achieve good results. (b) The neurally-guided model, where random choice parameters\nare computed via neural networks. Once trained, forward sampling from this model adheres closely\nto the target image, and SMC with only 10 particles consistently produces good results.\n\nshapes the distribution over possible choices, guiding the programs\u2019s future output based on the target\nimage and its past output. These neural nets affect both continuous choices (e.g. angles) as well as\ncontrol \ufb02ow decisions (e.g. recursion): they dictate where the chain goes next, as well as whether it\nkeeps going at all. For continuous choices such as gaussian, we also modify the program to sample\nfrom a mixture distribution. This helps the program handle situations where the constraints permit\nmultiple distinct choices (e.g. in which direction to start the chain for the circle-shaped target image\nin Figure 1).\nOnce trained, chain_neural generates constraint-satisfying results more ef\ufb01ciently than its un-guided\ncounterpart. Figure 1b shows example outputs: forward samples adhere closely to the target images,\nand SMC with 10 particles is suf\ufb01cient to produce chains that fully \ufb01ll the target shape. The next\nsection describes the process of building and training such neurally-guided procedural models.\n\n4 Method\n\nFor our purposes, a procedural model is a generative probabilistic model of the following form:\n\nPM(x) =\n\npi(xi; \u03a6i(x1, . . . , xi\u22121))\n\ni=1\n\nHere, x is the vector of random choices the procedural modeling program makes as it executes. The\npi\u2019s are local probability distributions from which each successive random choice is drawn. Each pi\nis parameterized by a set of parameters (e.g. mean and variance, for a Gaussian distribution), which\nare determined by some function \u03a6i of the previous random choices x1, . . . , xi\u22121.\nA constrained procedural model also includes an unnormalized likelihood function (cid:96)(x, c) that\nmeasures how well an output of the model satis\ufb01es some constraint c:\n\n|x|(cid:89)\n\n|x|(cid:89)\n\nPCM(x|c) =\n\n\u00b7 PM(x) \u00b7 (cid:96)(x, c)\n\n1\nZ\n\nIn the chain example, c is the target image, with (cid:96)(\u00b7, c) measuring similarity to that image.\nA neurally-guided procedural model modi\ufb01es a procedural model by replacing each parameter\nfunction \u03a6i with a neural network:\n\nPGM(x|c; \u03b8) =\n\n\u02dcpi(xi; NNi(I(x1, . . . , xi\u22121), c; \u03b8))\n\nwhere I(x1, . . . , xi\u22121) renders the model output after the \ufb01rst i \u2212 1 random choices, and \u03b8 are the\nnetwork parameters. \u02dcpi is a mixture distribution if random choice i is continuous; otherwise, \u02dcpi = pi.\n\ni=1\n\n3\n\n\fFigure 2: Network architecture for neurally-guided procedural models. The outputs are the parameters\nfor a random choice probability distribution. The inputs come from three sources: Local State\nFeatures are the arguments to the function in which the random choice occurs; Partial Output\nFeatures come from 3x3 pixel windows of the partial image the model has generated, extracted at\nmultiple resolutions, around the procedural model\u2019s current position; Target Image Features are\nanalogous windows extracted from the target image, if the constraint requires one.\n\nTo train a neurally-guided procedural model, we seek parameters \u03b8 such that PGM is as close\nas possible to PCM. This goal can be formalized as minimizing the conditional KL divergence\nDKL(PCM||PGM) (see the supplemental materials for derivation):\n\nN(cid:88)\n\ns=1\n\n|x|(cid:88)\n\nDKL(PCM||PGM) \u2248 max\n\n\u03b8\n\nmin\n\n\u03b8\n\n1\nN\n\nlog PGM(xs|cs; \u03b8)\n\nxs \u223c PCM(x) , cs \u223c P (c)\n\n(1)\n\nwhere the xs are example outputs generated using SMC, given a cs drawn from some distribution\nP (c) over constraints, e.g. uniform over a set of training images. This is simply maximizing the\nlikelihood of the xs under the neurally-guided model. Training then proceeds via stochastic gradient\nascent using the gradient\n\n\u2207 log PGM(x|c; \u03b8) =\n\n\u2207 log \u02dcpi(xi; NNi(I(x1, . . . , xi\u22121), c; \u03b8))\n\n(2)\n\ni=1\n\nThe trained PGM(x|c; \u03b8) can then be used as an importance distribution for SMC.\nIt is worth noting that using the other direction of KL divergence, DKL(PGM||PCM), leads to\nthe marginal likelihood lower bound objective used in many black-box variational inference al-\ngorithms [23, 6, 11]. This objective requires training samples from PGM, which are much less\nexpensive to generate than samples from PCM. When used for procedural modeling, however, it\nleads to models whose outputs lack diversity, making them unsuitable for generating visually-varied\ncontent. This behavior is due to a well-known property of the objective: minimizing it produces\napproximating distributions that are overly-compact, i.e. concentrating their probability mass in a\nsmaller volume of the state space than the true distribution being approximated [10]. Our objective\nis better suited for training proposal distributions for importance sampling methods (such as SMC),\nwhere the target density must be absolutely continuous with respect to the proposal density [3].\n\n4.1 Neural Network Architecture\n\nEach network NNi should predict a distribution over choice i that is as close as possible to its true\nposterior distribution. More complex networks capture more dependencies and increase accuracy but\nrequire more computation time to execute. We can also increase accuracy at the cost of computation\ntime by running SMC with more particles. If our networks are too complex (i.e. accuracy provided\nper unit computation time is too low), then the neurally-guided model PGM will be outperformed by\nsimply using more particles with the original model PM. For neural guidance to provide speedups,\nwe require networks that pack as much predictive power into as simple an architecture as possible.\n\n4\n\nFullyConnected(FC)tanhFCBoundsOutputParamsnfnanpnf236c36cnf2npfunction\t\r \u00a0branch(\t\r \u00a0pos,\t\r \u00a0ang,\t\r \u00a0width\t\r \u00a0)\t\r \u00a0{...}\t\r \u00a0(Optional)Target Image (50x50)DownsampleDownsampleCurrentPartialOutput(50x50)Target Image FeaturesPartial Output FeaturesLocal State Features\fScribbles\n\nGlyphs\n\nPhyloPic\n\nFigure 3: Example images from our datasets.\n\nFigure 2 shows our network architecture: a multilayer perceptron with nf inputs, one hidden layer of\nsize nf/2 with a tanh nonlinearity, and np outputs, where np is the number of parameters the random\nchoice expects. We found that a simpler linear model did not perform as well per unit time. Since\nsome parameters are bounded (e.g. Gaussian variance must be positive), each output is remapped\nvia an appropriate bounding transform (e.g. ex for non-negative parameters). The inputs come from\nseveral sources, each providing the network with decision-critical information:\n\nLocal State Features The model\u2019s current position p, the current orientation of any local reference\nframe, etc. We access this data via the arguments of the function call in which the random choice\noccurs, extracting all na scalar arguments and normalizing them to [\u22121, 1].\nPartial Output Features Next, the network needs information about the output the model has\nalready generated. The raw pixels of the current partial output image I(\u00b7) provide too much data; we\nneed to summarize the relevant image contents. We extract 3x3 windows of pixels around the model\u2019s\ncurrent position p at four different resolution levels, with each level computed by downsampling the\nprevious level via a 2x2 box \ufb01lter. This results in 36c features for a c-channel image. This architecture\nis similar to the foveated \u2018glimpses\u2019 used in visual attention models [12]. Convolutional networks\nmight also be used here, but this approach provided better performance per unit of computation time.\n\nTarget Image Features Finally, if the constraint being enforced involves a target image, as in\nthe chain example of Section 3, we also extract multi-resolution windows from this image. These\nadditional 36c features allow the network to make appropriate decisions for matching the image.\n\n4.2 Training\n\nWe train with stochastic gradient ascent (see Equation 2). We use the Adam algorithm [7] with\n\u03b1 = \u03b2 = 0.75, step size 0.01, and minibatch size one. We terminate training after 20000 iterations.\n\n5 Experiments\n\nIn this section, we evaluate how well neurally-guided procedural models capture image-based\nconstraints. We implemented our prototype system in the WebPPL probabilistic programming\nlanguage [4] using the adnn neural network library.1 All timing data was collected on an Intel Core\ni7-3840QM machine with 16GB RAM running OSX 10.10.5.\n\n5.1\n\nImage Datasets\n\nIn experiments which require target images, we use the following image collections:\n\n\u2022 Scribbles: 49 binary mask images drawn by hand with the brush tool in Photoshop. Includes\n\nshapes with thick and thin regions, high and low curvature, and self-intersections.\n\n\u2022 Glyphs: 197 glyphs from the FF Tartine Script Bold typeface (all glyphs with only one\n\nforeground component and at least 500 foreground pixels when rendered at 129x97).\n\n\u2022 PhyloPic: 35 images from the PhyloPic silhouette database.2\n\nWe augment the dataset with a horizontally-mirrored copy of each image, and we annotate each\nimage with a starting point and direction from which to initialize the execution of a procedural model.\nFigure 3 shows some representative images from each collection.\n\n1https://github.com/dritchie/adnn\n2http://phylopic.org\n\n5\n\n\fTarget\n\nReference\n\nGuided\n\nUnguided\n(Equal N)\n\nUnguided\n(Equal Time)\n\nN = 600 , 30.26 s\n\nN = 10 , 1.5 s\n\nN = 10 , 0.1 s\n\nN = 45 , 1.58 s\n\nFigure 4: Constraining a vine-growth procedural model to match a target image. N is the number of\nSMC particles used. Reference shows an example result after running SMC on the unguided model\nwith a large number of particles. Neurally-guided models generate results of this quality in a couple\nseconds; the unguided model struggles given the same amount of particles or computation time.\n\n5.2 Shape Matching\n\nWe \ufb01rst train neurally-guided procedural models to match 2D shapes speci\ufb01ed as binary mask images.\nIf D is the spatial domain of the image, then the likelihood function for this constraint is\n\nsim(I(x), c) \u2212 sim(0, c)\n\n1 \u2212 sim(0, c)\n\n(cid:96)shape(x, c) = N (\n\n(cid:80)\np\u2208D w(p) \u00b7 1{I1(p) = I2(p)}\n\n(cid:80)\n\np\u2208D w(p)\n\nw(p) =\n\n, 1, \u03c3shape)\n\n\uf8f1\uf8f2\uf8f31\n\n1\nw\ufb01lled\n\nif I2(p) = 0\nif ||\u2207I2(p)||= 1\nif ||\u2207I2(p)||= 0\n\n(3)\n\nsim(I1, I2) =\n\nwhere \u2207I(p) is a binary edge mask computed using the Sobel operator. This function encourages\nthe output image I(x) to be similar to the target mask c, where similarity is normalized against\nc\u2019s similarity to an empty image 0. Each pixel p\u2019s contribution is weighted by w(p), determined\nby whether the target mask is empty, \ufb01lled, or has an edge at that pixel. We use w\ufb01lled = 0.\u00af6, so\nempty and edge pixels are worth 1.5 times more than \ufb01lled pixels. This encourages matching of\nperceptually-important contours and negative space. \u03c3shape = 0.02 in all experiments.\nWe wrote a WebPPL program which recursively generates vines with leaves and \ufb02owers and then\ntrained a neurally-guided version of this program to capture the above likelihood. The model was\ntrained on 10000 example outputs, each generated using SMC with 600 particles. Target images were\ndrawn uniformly at random from the Scribbles dataset. Each example took on average 17 seconds to\ngenerate; parallelized across four CPU cores, the entire set of examples took approximately 12 hours\nto generate. Training took 55 minutes in our single-threaded implementation.\nFigure 4 shows some outputs from this program. 10-particle SMC produces recognizable results with\nthe neurally-guided model (Guided) but not with the unguided model (Unguided (Equal N)). A more\nequitable comparison is to give the unguided model the same amount of wall-clock time as the guided\nmodel. While the resulting outputs fare better, the target shape is still obscured (Unguided (Equal\nTime)). We \ufb01nd that the unguided model needs \u223c200 particles to reliably match the performance of\nthe guided model. Additional results are shown in the supplemental materials.\nFigure 5 shows a quantitative comparison between \ufb01ve different models on the shape matching task:\n\n\u2022 Unguided: The original, unguided procedural model.\n\u2022 Constant Params: The neural network for each random choice is a vector of constant\n\nparameters (i.e. a partial mean \ufb01eld approximation [23]).\n\n\u2022 + Local State Features: Adding the local state features described in Section 4.1.\n\u2022 + Target Image Features: Adding the target image features described in Section 4.1.\n\u2022 All Features: Adding the partial output features described in Section 4.1.\n\nWe test each model on the Glyph dataset and report the median normalized similarity-to-target\nachieved (i.e. argument one to the Gaussian in Equation 3), plotted in Figure 5a. The performance\n\n6\n\n\f(a)\n\n(b)\n\nFigure 5: Shape matching performance comparison. \u201cSimilarity\u201d is median normalized similarity to\ntarget, averaged over all targets in the test dataset. Bracketing lines show 95% con\ufb01dence bounds.\n(a) Performance as number of SMC particles increases. The neurally-guided model achieves higher\naverage similarity as more features are added. (b) Computation time required as desired similarity\nincreases. The vertical gap between the two curves indicates speedup (which can be up to 10x).\n\n(a)\n\n(b)\n\nFigure 6: (a) Using four-component mixtures for continuous random choices boosts performance. (b)\nThe effect of training set size on performance (at 10 SMC particles), plotted on a logarithmic scale.\nAverage similarity-to-target levels off at \u223c1000 examples.\n\nof the guided model improves with the addition of more features; at 10 particles, the full model\nis already approaching an asymptote. Figure 5b shows the wall-clock time required to achieve\nincreasing similarity thresholds. The vertical gap between the two curves shows the speedup given\nby neural guidance, which can be as high as 10x. For example, the + Local State Features model\nreaches similarity 0.35 about 5.5 times faster than the Unguided model, the + Target Image Features\nmodel is about 1.5 times faster still, and the All Features Model is about 1.25 times faster than that.\nNote that we trained on the Scribbles dataset but tested on the Glyphs dataset; these results suggest\nthat our models can generalize to qualitatively-different previously-unseen images.\nFigure 6a shows the bene\ufb01t of using mixture guides for continuous random choices. The experimental\nsetup is the same as in Figure 5. We compare a model which uses four-component mixtures with a\nno-mixture model. Using mixtures boosts performance, which we alluded to in Section 3: at shape\nintersections, such as the crossing of the letter \u2018t,\u2019 the model bene\ufb01ts from multimodal uncertainty.\nUsing more than four mixture components did not improve performance on this test dataset.\nWe also investigate how the number of training examples affects performance. Figure 6b plots the\nmedian similarity at 10 particles as training set size increases. Performance increases rapidly for the\n\ufb01rst few hundred examples before leveling off, suggesting that \u223c1000 sample traces is suf\ufb01cient\n(for our particular choice of training set, at least). This may seem surprising, as many published\nneurally-based learning systems require many thousands to millions of training examples. In our case,\n\n7\n\n01234567891011Number of Particles0.00.10.20.30.40.50.60.7Similarity0.000.050.100.150.200.250.300.350.400.450.500.550.600.650.700.75Similarity0.00.20.40.60.81.0Computation Time (seconds)ConditionAll Features+ Target Image Features+ Local State FeaturesConstant ParamsUnguided01234567891011Number of Particles0.00.10.20.30.40.50.60.7SimilarityConditionWith Mixture DistributionsWithout Mixture Distributions10205010020050010002000Number of Training Traces0.00.10.20.30.40.50.60.7Similarity at 10 Particles\fReference N = 600\n\nGuided N = 15\n\nUnguided\n(Equal N)\n\nUnguided\n(Equal Time)\n\nFigure 7: Constraining the vine-growth program to generate circuit-like patterns. Reference outputs\ntook around \u223c 70 seconds to generate; outputs from the guided model took \u223c 3.5 seconds.\n\neach training example contains hundreds to thousands of random choices, each of which provides\na learning signal\u2014in this way, the training data is \u201cbigger\u201d than it appears. Our implementation\ngenerates 1000 samples in just over an hour using four CPU cores.\n\n5.3 Stylized \u201cCircuit\u201d Design\n\nWe next train neurally-guided procedural models to capture a likelihood that does not use a target\nimage: constraining the vines program to resemble a stylized circuit design. To achieve the dense\npacking of long wire traces that is one of the most striking visual characteristics of circuit boards,\nwe encourage a percentage \u03c4 of the image to be \ufb01lled (\u03c4 = 0.5 in our results) and to have a dense,\nhigh-magnitude gradient \ufb01eld, as this tends to create many long rectilinear or diagonal edges:\n\n(cid:96)circ(x) = N (edge(I(x)) \u00b7 (1 \u2212 \u03b7(\ufb01ll(I(x)), \u03c4 )), 1, \u03c3circ)\n\n(4)\n\nedge(I) =\n\n1\n|D|\n\n||\u2207I(p)||\n\n\ufb01ll(I) =\n\n1\n|D|\n\n(cid:88)\n\np\u2208D\n\n(cid:88)\n\np\u2208D\n\nI(p)\n\nwhere \u03b7(x, \u00afx) is the relative error of x from \u00afx and \u03c3circ = 0.01. We also penalize geometry outside\nthe bounds of the image, encouraging the program to \ufb01ll in a rectangular \u201cdie\u201d-like region. We train\non 2000 examples generated using SMC with 600 particles. Example generation took 10 hours and\ntraining took under two hours. Figure 7 shows outputs from this program. As with shape matching,\nthe neurally-guided model generates high-scoring results signi\ufb01cantly faster than the unguided model.\n\n6 Conclusion and Future Work\n\nThis paper introduced neurally-guided procedural models: constrained procedural models that use\nneural networks to capture constraint-induced dependencies. We showed how to train guides for\naccumulative models with image-based constraints using a simple-yet-powerful network architecture.\nExperiments demonstrated that neurally-guided models can generate high-quality results signi\ufb01cantly\nfaster than unguided models.\nAccumulative procedural models provide a current position p, which is not true of other generative\nparadigms (e.g. texture generation, which generates content across its entire spatial domain). In such\nsettings, the guide might instead learn what parts of the current partial output are relevant to each\nrandom choice using an attention process [12].\nUsing neural networks to predict random choice parameters is just one possible program transforma-\ntion for generatively capturing constraints. Other transformations, such as control \ufb02ow changes, may\nbe necessary to capture more types of constraints. A \ufb01rst step in this direction would be to combine\nour approach with the grammar-splitting technique of Dang et al. [2].\nMethods like ours could also accelerate inference for other applications of procedural models, e.g. as\npriors in analysis-by-synthesis vision systems [9]. A robot perceiving a room through an onboard\ncamera, detecting chairs, then \ufb01tting a procedural model to the detected chairs could learn importance\ndistributions for each step of the chair-generating process (e.g.\nthe number of parts, their size,\narrangement, etc.) Future work is needed to determine appropriate neural guides for such domains.\n\n8\n\n\fReferences\n[1] Bed\u02c7rich Bene\u0161, Ond\u02c7rej \u0160t\u2019ava, Radomir M\u02c7ech, and Gavin Miller. Guided Procedural Modeling.\n\nEurographics 2011.\n\nIn\n\n[2] Minh Dang, Stefan Lienhard, Duygu Ceylan, Boris Neubert, Peter Wonka, and Mark Pauly. Interactive\n\nDesign of Probability Density Functions for Shape Grammars. In SIGGRAPH Asia 2015.\n\n[3] Charles Geyer. Importance Sampling, Simulated Tempering, and Umbrella Sampling. In S. Brooks,\nA. Gelman, G. Jones, and X.L. Meng, editors, Handbook of Markov Chain Monte Carlo. CRC Press, 2011.\n\n[4] Noah D Goodman and Andreas Stuhlm\u00fcller. The Design and Implementation of Probabilistic Programming\n\nLanguages. http://dippl.org, 2014. Accessed: 2015-12-23.\n\n[5] Shixiang Gu, Zoubin Ghahramani, and Richard E. Turner. Neural Adaptive Sequential Monte Carlo. In\n\nNIPS 2015.\n\n[6] K. Norman J. Manning, R. Ranganath and D. Blei. Black Box Variational Inference. In AISTATS 2014.\n\n[7] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR 2015.\n\n[8] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In ICLR 2014.\n\n[9] T. Kulkarni, P. Kohli, J. B. Tenenbaum, and V. Mansinghka. Picture: An Imperative Probabilistic\n\nProgramming Language for Scene Perception. In CVPR 2015.\n\n[10] David J. C. MacKay. Information Theory, Inference & Learning Algorithms. Cambridge University Press,\n\n2002.\n\n[11] Andriy Mnih and Karol Gregor. Neural Variational Inference and Learning in Belief Networks. In ICML\n\n2014.\n\n[12] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent Models of Visual\n\nAttention. In NIPS 2014.\n\n[13] Pascal M\u00fcller, Peter Wonka, Simon Haegler, Andreas Ulmer, and Luc Van Gool. Procedural Modeling of\n\nBuildings. In SIGGRAPH 2006.\n\n[14] Radom\u00edr M\u02c7ech and Przemyslaw Prusinkiewicz. Visual Models of Plants Interacting with Their Environment.\n\nIn SIGGRAPH 1996.\n\n[15] B. Paige and F. Wood. Inference Networks for Sequential Monte Carlo in Graphical Models. In ICML\n\n2016.\n\n[16] P. Prusinkiewicz and Aristid Lindenmayer. The Algorithmic Beauty of Plants. Springer-Verlag New York,\n\nInc., 1990.\n\n[17] Przemyslaw Prusinkiewicz, Mark James, and Radom\u00edr M\u02c7ech. Synthetic Topiary. In SIGGRAPH 1994.\n\n[18] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approxi-\n\nmate Inference in Deep Generative Models. In ICML 2014.\n\n[19] Daniel Ritchie, Sharon Lin, Noah D. Goodman, and Pat Hanrahan. Generating Design Suggestions under\n\nTight Constraints with Gradient-based Probabilistic Programming. In Eurographics 2015.\n\n[20] Daniel Ritchie, Ben Mildenhall, Noah D. Goodman, and Pat Hanrahan. Controlling Procedural Modeling\n\nPrograms with Stochastically-Ordered Sequential Monte Carlo. In SIGGRAPH 2015.\n\n[21] Jerry O. Talton, Yu Lou, Steve Lesser, Jared Duke, Radom\u00edr M\u02c7ech, and Vladlen Koltun. Metropolis\n\nProcedural Modeling. ACM Trans. Graph., 30(2), 2011.\n\n[22] O. \u0160t\u2019ava, S. Pirk, J. Kratt, B. Chen, R. M\u02c7ech, O. Deussen, and B. Bene\u0161. Inverse Procedural Modelling of\n\nTrees. Computer Graphics Forum, 33(6), 2014.\n\n[23] David Wingate and Theophane Weber. Automated Variational Inference in Probabilistic Programming. In\n\nNIPS 2012 Workshop on Probabilistic Programming.\n\n[24] Michael T. Wong, Douglas E. Zongker, and David H. Salesin. Computer-generated Floral Ornament. In\n\nSIGGRAPH 1998.\n\n9\n\n\f", "award": [], "sourceid": 340, "authors": [{"given_name": "Daniel", "family_name": "Ritchie", "institution": "Stanford University"}, {"given_name": "Anna", "family_name": "Thomas", "institution": "Stanford University"}, {"given_name": "Pat", "family_name": "Hanrahan", "institution": "Stanford University"}, {"given_name": "Noah", "family_name": "Goodman", "institution": "Stanford University"}]}