%PDF-1.3 1 0 obj << /Kids [ 4 0 R 5 0 R 6 0 R 7 0 R 8 0 R 9 0 R 10 0 R 11 0 R 12 0 R ] /Type /Pages /Count 9 >> endobj 2 0 obj << /Subject (Neural Information Processing Systems http\072\057\057nips\056cc\057) /Publisher (Curran Associates\054 Inc\056) /Language (en\055US) /Created (2013) /EventType (Spotlight) /Description-Abstract (We study the problem of estimating continuous quantities\054 such as prices\054 probabilities\054 and point spreads\054 using a crowdsourcing approach\056 A challenging aspect of combining the crowd\047s answers is that workers\047 reliabilities and biases are usually unknown and highly diverse\056 Control items with known answers can be used to evaluate workers\047 performance\054 and hence improve the combined results on the target items with unknown answers\056 This raises the problem of how many control items to use when the total number of items each workers can answer is limited\072 more control items evaluates the workers better\054 but leaves fewer resources for the target items that are of direct interest\054 and vice versa\056 We give theoretical results for this problem under different scenarios\054 and provide a simple rule of thumb for crowdsourcing practitioners\056 As a byproduct\054 we also provide theoretical analysis of the accuracy of different consensus methods\056) /Producer (PyPDF2) /Title (Scoring Workers in Crowdsourcing\072 How Many Control Questions are Enough\077) /Date (2013) /ModDate (D\07220151202104902\05508\04700\047) /Published (2013) /Type (Conference Proceedings) /firstpage (1914) /Book (Advances in Neural Information Processing Systems 26) /Description (Paper accepted and presented at the Neural Information Processing Systems Conference \050http\072\057\057nips\056cc\057\051) /Editors (C\056J\056C\056 Burges and L\056 Bottou and M\056 Welling and Z\056 Ghahramani and K\056Q\056 Weinberger) /Author (Qiang Liu\054 Alexander T\056 Ihler\054 Mark Steyvers) /lastpage (1922) >> endobj 3 0 obj << /Type /Catalog /Pages 1 0 R >> endobj 4 0 obj << /Contents 13 0 R /Parent 1 0 R /Resources 14 0 R /MediaBox [ 0 0 612 792 ] /Annots [ 32 0 R 33 0 R 34 0 R 35 0 R ] /Type /Page >> endobj 5 0 obj << /Contents 36 0 R /Parent 1 0 R /Resources 37 0 R /MediaBox [ 0 0 612 792 ] /Annots [ 78 0 R 79 0 R 80 0 R 81 0 R 82 0 R 83 0 R 84 0 R 85 0 R 86 0 R 87 0 R 88 0 R 89 0 R 90 0 R 91 0 R 92 0 R 93 0 R 94 0 R ] /Type /Page >> endobj 6 0 obj << /Contents 95 0 R /Parent 1 0 R /Resources 96 0 R /MediaBox [ 0 0 612 792 ] /Annots [ 112 0 R 113 0 R 114 0 R 115 0 R ] /Type /Page >> endobj 7 0 obj << /Contents 116 0 R /Parent 1 0 R /Resources 117 0 R /MediaBox [ 0 0 612 792 ] /Annots [ 118 0 R 119 0 R 120 0 R 121 0 R 122 0 R 123 0 R 124 0 R 125 0 R 126 0 R 127 0 R 128 0 R 129 0 R 130 0 R 131 0 R ] /Type /Page >> endobj 8 0 obj << /Contents 132 0 R /Parent 1 0 R /Resources 133 0 R /MediaBox [ 0 0 612 792 ] /Annots [ 139 0 R 140 0 R 141 0 R 142 0 R 143 0 R 144 0 R 145 0 R 146 0 R 147 0 R 148 0 R 149 0 R 150 0 R 151 0 R ] /Type /Page >> endobj 9 0 obj << /Contents 152 0 R /Parent 1 0 R /Resources 153 0 R /MediaBox [ 0 0 612 792 ] /Annots [ 162 0 R 163 0 R 164 0 R 165 0 R 166 0 R 167 0 R 168 0 R 169 0 R 170 0 R 171 0 R 172 0 R 173 0 R 174 0 R 175 0 R 176 0 R ] /Type /Page >> endobj 10 0 obj << /Contents 177 0 R /Parent 1 0 R /Resources 178 0 R /MediaBox [ 0 0 612 792 ] /Annots [ 257 0 R 258 0 R 259 0 R 260 0 R 261 0 R 262 0 R 263 0 R 264 0 R 265 0 R ] /Type /Page >> endobj 11 0 obj << /Contents 266 0 R /Parent 1 0 R /Type /Page /Resources 267 0 R /MediaBox [ 0 0 612 792 ] >> endobj 12 0 obj << /Contents 477 0 R /Parent 1 0 R /Type /Page /Resources 478 0 R /MediaBox [ 0 0 612 792 ] >> endobj 13 0 obj << /Length 3123 /Filter /FlateDecode >> stream xڵZKܶ͜]/KQ$.+CbgqAj=F7qR>h 4Fя1 z܄W7|C/JxIyIPt_o|^q$Rɋ2qeYe=C=b?rw]7D-}_;{M7nk%@|KC4l/_0HȻʠkbGeݵ:n<n^?^k! 8ZЫa;/JƳv {X(d@QyLIt^oDXuAp*Ko<U֝z=TZj8],L|duXhy$H5JìUA'^Ç5NЃ|t8 ?C>`$J,g< Ǵ'9dpUI}~mc-݄A$ij<2.f5EK0co^Vuu()r]Zc}fGnߨQq!$ ^Я vFC>VQߌՑq|uÙDYku3ISt;PӜ{%k7c4$HT}FؗgUVGɰyQ6j3sVkѬ2?uKKC 1g 0CDzFwDUHki&bhьڄ-?ͪQ7 UC(g= owhT{ŢUo ;tx d31TUnxhIGv>㡦x|o)ߧinv`){@t 4ЕSvxV{faΰ|+gIx$D.4Tum XmmPCGf;6lB 7%#=l켜}NW[:2h:ym-Lۍ6^ пeš?4(]2^d!١Vey-^J̚K], "yswZ, MaG]*+9a` tENv!=.$A@9?COJMZvBa}RGxkhN?6