"crowdsourcing" entries

Exploring methods in active learning

Tips on how to build effective human-machine hybrids, from crowdsourcing expert Adam Marcus.

15146_ORM_Webcast_ad(archived)In a recent O’Reilly webcast, “Crowdsourcing at GoDaddy: How I Learned to Stop Worrying and Love the Crowd,” Adam Marcus explains how to mitigate common challenges of managing crowd workers, how to make the most of human-in-the-loop machine learning, and how to establish effective and mutually rewarding relationships with workers. Marcus is the director of data on the Locu team at GoDaddy, where the “Get Found” service provides businesses with a central platform for managing their online presence and content.

In the webcast, Marcus uses practical examples from his experience at GoDaddy to reveal helpful methods for how to:

  • Offset the inevitability of wrong answers from the crowd
  • Develop and train workers through a peer-review system
  • Build a hierarchy of trusted workers
  • Make crowd work inspiring and enable upward mobility

What to do when humans get it wrong

It turns out there is a simple way to offset human error: redundantly ask people the same questions. Marcus explains that when you ask five different people the same question, there are some creative ways to combine their responses, and use a majority vote. Read more…

Comment
Four short links: 11 February 2015

Four short links: 11 February 2015

Crowdsourcing Working, etcd DKVS, Psychology Progress, and Inferring Logfile Rules

  1. Crowdsourcing Isn’t Broken — great rundown of ways to keep crowdsourcing on track. As with open sourcing something, just throwing open the doors and hoping for the best has a low probability of success.
  2. etcd Hits 2.0 — first major stable release of an open source, distributed, consistent key-value store for shared configuration, service discovery, and scheduler coordination.
  3. You Can’t Play 20 Questions With Nature and Win (PDF) — There is, I submit, a view of the scientific endeavor that is implicit (and sometimes explicit) in the picture I have presented above. Science advances by playing 20 questions with nature. The proper tactic is to frame a general question, hopefully binary, that can be attacked experimentally. Having settled that bits-worth, one can proceed to the next. The policy appears optimal – one never risks much, there is feedback from nature at every step, and progress is inevitable. Unfortunately, the questions never seem to be really answered, the strategy does not seem to work. An old paper, but still resonant today. (via Mind Hacks)
  4. Sequence: Automated Analyzer for Reducing 100k Messages to 10s of Patterns — induces patterns from the examples in log files.
Comment
Four short links: 11 December 2014

Four short links: 11 December 2014

Crowdsourcing Framework, Data Team Culture, Everybody Scrolls, and Honeypot Data

  1. Hive — open source crowdsourcing framework from NYT Labs.
  2. Prezi Data Team Culture — good docs on logging, metrics, etc. The vision is a great place to start.
  3. Scroll Behaviour Across the Web (Chartbeat) — nobody reads above the fold, they immediately scroll.
  4. threat_research (github) — shared raw data and stats from honeypots.
Comment
Four short links: 21 August 2014

Four short links: 21 August 2014

Open Data Glue, Smithsonian Crowdsourcing, MIT Family Creativity, and Hardware Owie

  1. Datan open source project that provides a streaming interface between every file format and data storage backend. See the Wired piece on it.
  2. Smithsonian Crowdsourcing Transcription (Smithsonian) — 49 volunteers transcribed 200 pages of correspondence between the Monuments Men in a week. Soon it’ll be mathematics test questions: “if 49 people transcribe 200 pages in 7 days, how many weeks will it take …”
  3. MIT Guide to Family CompSci SessionsThis guide is for educators, community center staff, and volunteers interested in engaging their young people and their families to become designers and inventors in their community.
  4. What to Do When You Screw up 2,000 Orders (SparkFun) — even hardware companies need to do retrospectives.
Comment
Four short links: 1 August 2014

Four short links: 1 August 2014

Data Storytelling Tools, Massive Dataset Mining, Failed Crowdsourcing, and IoT Networking

  1. MisoDataset, a JavaScript client-side data management and transformation library, Storyboard, a state and flow-control management library & d3.chart, a framework for creating reusable charts with d3.js. Open source designed to expedite the creation of high-quality interactive storytelling and data visualisation content.
  2. Mining of Massive Datasets (PDF) — book by Stanford profs, focuses on data mining of very large amounts of data, that is, data so large it does not fit in main memory. Because of the emphasis on size, many of our examples are about the Web or data derived from the Web. Further, the book takes an algorithmic point of view: data mining is about applying algorithms to data, rather than using data to “train” a machine-learning engine of some sort.
  3. Lessons from Iceland’s Failed Crowdsourced Constitution (Slate) — Though the crowdsourcing moment could have led to a virtuous deliberative feedback loop between the crowd and the Constitutional Council, the latter did not seem to have the time, tools, or training necessary to process carefully the crowd’s input, explain its use of it, let alone return consistent feedback on it to the public.
  4. Thread a ZigBee Killer?Thread is Nest’s home automation networking stack, which can use the same hardware components as ZigBee, but which is not compatible, also not open source. The Novell NetWare of Things. Nick Hunn makes argument that Google (via Nest) are taking aim at ZigBee: it’s Google and Nest saying “ZigBee doesn’t work”.
Comment
Four short links: 8 July 2014

Four short links: 8 July 2014

Virtual Economies, Resource UAVs, Smarter Smaller Crowds, and Scaling Business

  1. Virtual Economies — new book from MIT Press on economics in games. The book will enable developers and designers to create and maintain successful virtual economies, introduce social scientists and policy makers to the power of virtual economies, and provide a useful guide to economic fundamentals for students in other disciplines.
  2. Resource Industry UAV Conference Presentations — collection of presentations from a recent resources industry conference. Includes UaaS: UAVs as a Service. (via DIY Drones)
  3. The Wisdom of Smaller, Smarter Crowdsin domains in which some crowd members have demonstrably more skill than others, smart sub-crowds could possibly outperform the whole. The central question this work addresses is whether such smart subsets of a crowd can be identified a priori in a large-scale prediction contest that has substantial skill and luck components. (via David Pennock)
  4. Larry and Sergey with Vinod (YouTube) — see transcription. I really liked Page’s point about scaling the number of things that companies do, and the constraints on such scaling.
Comment
Four short links: 3 April 2014

Four short links: 3 April 2014

Github for Data, Open Laptop, Crowdsourced Analysis, and Open Source Scraping

  1. dat — github-like tool for data, still v. early. It’s overdue. (via Nelson Minar)
  2. Novena Open Laptop — Bunnie Huang’s laptop goes on sale.
  3. Crowd Forecasting (NPR) — How is it possible that a group of average citizens doing Google searches in their suburban town homes can outpredict members of the United States intelligence community with access to classified information?
  4. Portia — open source visual web scraping tool.
Comment: 1

Crowdsourcing Feature discovery

More than algorithms, companies gain access to models that incorporate ideas generated by teams of data scientists

Data scientists were among the earliest and most enthusiastic users of crowdsourcing services. Lukas Biewald noted in a recent talk that one of the reasons he started CrowdFlower was that as a data scientist he got frustrated with having to create training sets for many of the problems he faced. More recently, companies have been experimenting with active learning (humans1 take care of uncertain cases, models handle the routine ones). Along those lines, Adam Marcus described in detail how Locu uses Crowdsourcing services to perform structured extraction (converting semi/unstructured data into structured data).

Another area where crowdsourcing is popping up is feature engineering and feature discovery. Experienced data scientists will attest that generating features is as (if not more) important than choice of algorithm. Startup CrowdAnalytix uses public/open data sets to help companies enhance their analytic models. The company has access to several thousand data scientists spread across 50 countries and counts a major social network among its customers. Its current focus is on providing “enterprise risk quantification services to Fortune 1000 companies”.

CrowdAnalytix breaks up projects in two phases: feature engineering and modeling. During the feature engineering phase, data scientists are presented with a problem (independent variable(s)) and are asked to propose features (predictors) and brief explanations for why they might prove useful. A panel of judges evaluate2 features based on the accompanying evidence and explanations. Typically 100+ teams enter this phase of the project, and 30+ teams propose reasonable features.

Read more…

Comments: 4

9 branding trends for startups in 2014

Laura Busche looks at trends through a Lean Startup lens

320_lean_brandingYour code can be clean as a whistle and your software deployment on-time, but if you’re a startup, branding is as vital to your success as any non-crashing app. The following trends are discussed through a frame of Lean Startup practices, relevant to any startup.

I sat down with the one-in-a-million Laura Busche, author of the upcoming book Lean Branding, at the recent Lean Startup Conference in San Francisco. We talked about what she sees as important branding trends for 2014. Those trends follow below, in both text and video formats.

You’ll find one additional surprise trend covered below, the video portion of which you can see in the Top Lean Branding Trends for 2014 compilation video below.

Are there any patterns within the nine trends discussed? Considered as a whole, a feeling of intimacy and intrigue certainly stand out. The intimacy element comes in the form of big brands trying to let customers feel closer to them by acting like smallish local brands, as well as the intimacy of working directly with customers to create the message of a brand (crowdsourcing) rather than lecturing potential customers about the merits of a brand. The intrigue element comes in the form of the use of images rather than words to communicate brands, playing on immediate emotional response, and also from brands surprising customers with their unusual personalities and intentional quirkiness, as well as by turning up in customer conversations to solve problems.

Read more…

Comments: 6
Four short links: 16 December 2013

Four short links: 16 December 2013

Data Pipeline, Data Driven Education, Crowdsourced Proofreading, and 3D Printed Shoes

  1. Suro (Github) — Netflix data pipeline service for large volumes of event data. (via Ben Lorica)
  2. NIPS Workshop on Data Driven Education — lots of research papers around machine learning, MOOC data, etc.
  3. Proofist — crowdsourced proofreading game.
  4. 3D-Printed Shoes (YouTube) — LeWeb talk from founder of the company, Continuum Fashion). (via Brady Forrest)
Comment