Climate change poses new, intricate challenges for public authorities. They have to develop and adjust policies in order to adapt to these changing conditions. To do so, novel ideas and innovative approaches are developed and tested in pilot projects. Such projects are often acknowledged as a relatively easy manner to explore new pathways to deal with climate change consequences. They are usually organized at the boundaries or even outside the dominant policy regime. We explore, with help of a comparative case study of six climate adaptation pilots in the Netherlands the existence of a ‘pilot paradox’. This paradox states that the conditions that are necessary to give a pilot room to experiment and to learn also seem to constitute the main barriers to the translation of its results into changes in policy. Pilot results are often misaligned with prevailing policy paradigms and, because of the distance between the pilot and the mother organizations, the latter are ill-prepared to receive pilot results. We can conclude that designing impactful experiments is only possible when actors recognize the pilot paradox. Within the pilot design, attention must be paid to issues of external representativeness and boundary spanning between the niche and the regime.