The Health Technology Assessment (HTA) researchers at the Institute of Health Economics, in collaboration with researchers from two HTA agencies in Australia and Spain, developed, piloted, and validated a 20-criteria quality appraisal checklist specific for case series studies (before-after single arm studies with no control group) (1,2). Since its publication in 2012, the use of the checklist has spread globally through the HTA community and to researchers in other areas. This presentation will briefly introduce the tool, summarize user experiences, outline potential challenges, and provide practical solutions for using or adapting the checklist to various HTA topics.

Feedback from fifteen researchers was collected informally by email and/or formally by questionnaire. The questions included focused on the relevance, clarity, and usefulness of the checklist and its instructions, as well as potential revisions and/or addition of other criteria.

While some of the checklist's criteria apply to all studies of a particular type, others are specific to the research question and/or the technology under investigation; discussion on the modification and/or adaptation of the checklist and its instructions is therefore required before commencing appraisal. Some criteria are difficult to score owing to study reporting limitations. Quality assessment can be challenging when multiple types of studies are included; however, currently there is no single universally validated tool available for diverse study designs. There are frequent demands for a cut-off point in order to separate high- from low-quality studies. However, no scale or numeric scoring was developed for the checklist, due to the well-recognized risks associated with such a scoring system.

The increased use of the checklist and general positive feedback indicates the need for such a tool. User feedback helped improve our understanding of the checklist's applicability with various topics, as well as the potential refinements needed to increase its utility and robustness.