We examined the trade-off between the cost of response redundancy and the gain in output quality on the popular crowdsourcing platform Mechanical Turk, as a partial replication of Kosinski et al. (2012) who demonstrated a significant improvement in performance by aggregating multiple responses through majority vote. We submitted single items from a validated intelligence test as Human Intelligence Tasks (HITs) and aggregated the responses from “virtual groups” consisting of 1 to 24 workers. While the original study relied on resampling from a relatively small number of responses across a range of experimental conditions, we randomly and independently sampled from a large number of HITs, focusing only on the main effect of group size. We found that – on average – a group of six MTurkers has a collective IQ one standard deviation above the mean for the general population, thus demonstrating a “wisdom of the crowd” effect. The relationship between group size and collective IQ was characterised by diminishing returns, suggesting moderately sized groups provide the best return on investment. We also analysed performance of a smaller subset of workers who had each completed all 60 test items, allowing for a direct comparison between a group’s collective IQ and the individual IQ of its members. This demonstrated that randomly selected groups collectively equalled the performance of the best-performing individual within the group. Our findings support the idea that substantial intellectual capacity can be gained through crowdsourcing, contingent on moderate redundancy built into the task request.

As COVID-19 was declared a health emergency in March 2020, there was immense demand for information about the novel pathogen. This paper examines the clinician-reported impact of Project ECHO COVID-19 Clinical Rounds on clinician learning. Primary sources of study data were Continuing Medical Education (CME) Surveys for each session from the dates of March 24, 2020 to July 30, 2020 and impact surveys conducted in November 2020, which sought to understand participants’ overall assessment of sessions. Quantitative analyses included descriptive statistics and Mann-Whitney testing. Qualitative data were analyzed through inductive thematic analysis. Clinicians rated their knowledge after each session as significantly higher than before that session. 75.8% of clinicians reported they would ‘definitely’ or ‘probably’ use content gleaned from each attended session and clinicians reported specific clinical and operational changes made as a direct result of sessions. 94.6% of respondents reported that COVID-19 Clinical Rounds helped them provide better care to patients. 89% of respondents indicated they ‘strongly agree’ that they would join ECHO calls again.COVID-19 Clinical Rounds offers a promising model for the establishment of dynamic peer-to-peer tele-mentoring communities for low or no-notice response where scientifically tested or clinically verified practice evidence is limited.

The Portable Antiquities of the Netherlands (PAN) is an online system aimed at recording and documenting archaeological finds by the public. Since PAN launched in 2016, it has become an important data contributor to Dutch archaeology, amassing over 100,000 recorded finds. These data, mostly the result of metal detection, enable scholars to gain new insights and policy makers to make more informed decisions. This review describes the context in which PAN was established, along with its current structure and scope, before looking at its different components, including the underlying database and linked data reference collection. In a final section, the article briefly addresses some common issues inherent to public reporting programs and how PAN approaches these issues.

This article proposes the solver-aware system architecting framework for leveraging the combined strengths of experts, crowds and specialists to design innovative complex systems. Although system architecting theory has extensively explored the relationship between alternative architecture forms and performance under operational uncertainty, limited attention has been paid to differences due to who generates the solutions. The recent rise in alternative solving methods, from gig workers to crowdsourcing to novel contracting structures emphasises the need for deeper consideration of the link between architecting and solver-capability in the context of complex system innovation. We investigate these interactions through an abstract problem-solving simulation, representing alternative decompositions and solver archetypes of varying expertise, engaged through contractual structures that match their solving type. We find that the preferred architecture changes depending on which combinations of solvers are assigned. In addition, the best hybrid decomposition-solver combinations simultaneously improve performance and cost, while reducing expert reliance. To operationalise this new solver-aware framework, we induce two heuristics for decomposition-assignment pairs and demonstrate the scale of their value in the simulation. We also apply these two heuristics to reason about an example of a robotic manipulator design problem to demonstrate their relevance in realistic complex system settings.

Learning idiomatic expressions is seen as one of the most challenging stages in second-language learning because of their unpredictable meaning. A similar situation holds for their identification within natural language processing applications such as machine translation and parsing. The lack of high-quality usage samples exacerbates this challenge not only for humans but also for artificial intelligence systems. This article introduces a gamified crowdsourcing approach for collecting language learning materials for idiomatic expressions; a messaging bot is designed as an asynchronous multiplayer game for native speakers who compete with each other while providing idiomatic and nonidiomatic usage examples and rating other players’ entries. As opposed to classical crowd-processing annotation efforts in the field, for the first time in the literature, a crowd-creating & crowd-rating approach is implemented and tested for idiom corpora construction. The approach is language-independent and evaluated on two languages in comparison to traditional data preparation techniques in the field. The reaction of the crowd is monitored under different motivational means (namely, gamification affordances and monetary rewards). The results reveal that the proposed approach is powerful in collecting the targeted materials, and although being an explicit crowdsourcing approach, it is found entertaining and useful by the crowd. The approach has been shown to have the potential to speed up the construction of idiom corpora for different natural languages to be used as second-language learning material, training data for supervised idiom identification systems, or samples for lexicographic studies.

‘Openness’ is one of the key concepts brought forward by postindustrial narratives questioning the modern repartition of roles between industries and customers. In these narratives, citizen participation in design and intellectual property management based on open source principles are the promise of more sustainable production models. In this context, openness in product design and development has been the object of growing interest and experimentation from academia, businesses and grassroots communities. As a result, numerous concepts emerged that attempt to grasp the essence of this phenomenon, unfortunately leading to overlapping, conflicting or speculative depictions. In this article, we share the understanding we gained throughout 6 years of research on Open Design and Open Source Hardware and attempt to make the difference between myths and facts. We depict an enthusiastic but realistic picture of Open Design and Open Source Hardware practices as we could observe them and deliver a structured framework to situate concepts and their differences. From this, we share seven observations leading to seven corresponding research questions and establish a research agenda to stimulate further investigations into this socially relevant and potentially ground-breaking phenomenon.

The internal crowdsourcing-based ideation within a company can be defined as an involvement of its staff, specialists, managers, and other employees, to propose solution ideas for a pre-defined problem. This paper addresses a question, how many participants of the company-internal ideation process are required to nearly reach the ideation limit for the problems with a finite number of workable solutions. To answer the research question, the author proposes a set of metrics and a non-linear ideation performance function with a positive decreasing slope and ideation limit for the closed-ended problems. Three series of experiments helped to explore relationships between the metric attributes and resulted in a mathematical model which allows companies to predict the productivity metrics of their crowdsourcing ideation activities such as quantity of different ideas and ideation limit as a function of the number of contributors, their average personal creativity and ideation efficiency of a contributors’ group.

Every internet search query made out of curiosity by anyone who observed something in nature, as well as every photo uploaded to the internet, constitutes a data point of potential use to scientists. Researchers have now begun to exploit the vast online data accumulated through passive crowdsourcing for studies in ecology and epidemiology. Here, we demonstrate the usefulness of iParasitology, i.e. the use of internet data for tests of parasitological hypotheses, using hairworms (phylum Nematomorpha) as examples. These large worms are easily noticeable by people in general, and thus likely to generate interest on the internet. First, we show that internet search queries (collated with Google Trends) and photos uploaded to the internet (specifically, to the iNaturalist platform) point to parts of North America with many sightings of hairworms by the public, but few to no records in the scientific literature. Second, we demonstrate that internet searches predict seasonal peaks in hairworm abundance that accurately match scientific data. Finally, photos uploaded to the internet by non-scientists can provide reliable data on the host taxa that hairworms most frequently parasitize, and also identify hosts that appear to have been neglected by scientific studies. Our findings suggest that for any parasite group likely to be noticeable by non-scientists, information accumulating through internet search activity, photo uploads, social media or any other format available online, represents a valuable source of data that can complement traditional scientific data sources in parasitology.

Knowledge about political representatives' behavior is crucial for a deeper understanding of politics and policy-making processes. Yet resources on legislative elites are scattered, often specialized, limited in scope or not always accessible. This article introduces the Comparative Legislators Database (CLD), which joins micro-data collection efforts on open-collaboration platforms and other sources, and integrates with renowned political science datasets. The CLD includes political, sociodemographic, career, online presence, public attention, and visual information for over 45,000 contemporary and historical politicians from ten countries. The authors provide a straightforward and open-source interface to the database through an R package, offering targeted, fast and analysis-ready access in formats familiar to social scientists and standardized across time and space. The data is verified against human-coded datasets, and its use for investigating legislator prominence and turnover is illustrated. The CLD contributes to a central hub for versatile information about legislators and their behavior, supporting individual-level comparative research over long periods.

This article draws on a collection of crowdsourced biographical sketches of Black women suffragists to explore the contributions of these activists to the expansion of voting rights that accompanied the passage of the Nineteenth Amendment. It explores the motivations and strategies adopted by Black women suffragists and interracial alliances that emerged in the course of the suffrage struggle, comparing and contrasting the experiences of suffragists across racial lines.

Crowdfunding is the process of taking a project in need of investment and asking a large group of people to supply the investment. It allows organisations to sell their product before production, reducing the risk of new product development. Organisations such as Tesla and General Electric have used crowdfunding successfully but crowdfunding is yet to be explored as part of a formalised product development framework. This paper includes the business case for commercialising new products with crowdfunding and presents crowdfunding as part of a product development and commercialisation framework.