Hostname: page-component-848d4c4894-cjp7w Total loading time: 0 Render date: 2024-07-01T12:40:33.292Z Has data issue: false hasContentIssue false

Development and validation of an instrument for measuring competencies on public health informatics of primary health care worker (PHIC4PHC) in Indonesia

Published online by Cambridge University Press:  06 July 2020

Enny Rachmani
Affiliation:
College of Medical Science and Technology, Graduate Institute of Biomedical Informatics, Taipei Medical University, Taipei City, Taiwan Department of Health Information Management, Faculty of Health Science, Universitas Dian Nuswantoro, Semarang, Jawa Tengah, Indonesia
Chien-Yeh Hsu*
Affiliation:
Department of Information Management, National Taipei University of Nursing and Health Science, Taipei City, Taiwan Master Program in Global Health and Development, Taipei Medical University, Taipei City, Taiwan
Peter WuShou Chang
Affiliation:
TUFTS University Medical School, Chung Shan Medical University, Taichung, Taiwan
Anis Fuad
Affiliation:
College of Medical Science and Technology, Graduate Institute of Biomedical Informatics, Taipei Medical University, Taipei City, Taiwan Department of Biostatistics, Epidemiology and Population Health, Public Health, Faculty of Medicine, Gajah Mada University, Yogjakarta, Indonesia
Nurjanah Nurjanah
Affiliation:
Department of Public Health, Faculty of Health Science, Universitas Dian Nuswantoro, Kota Semarang, Jawa Tengah, Indonesia
Guruh Fajar Shidik
Affiliation:
Department of Health Information Management, Faculty of Health Science, Universitas Dian Nuswantoro, Semarang, Jawa Tengah, Indonesia Department of Informatics, Faculty of Computer Science, Universitas Dian Nuswantoro, Semarang, Jawa Tengah, Indonesia
Dina Nur Anggraini Ningrum
Affiliation:
Department of Public Health, Semarang State University, Kota Semarang, Jawa Tengah, Indonesia
Ming-Chin Lin*
Affiliation:
College of Medical Science and Technology, Graduate Institute of Biomedical Informatics, Taipei Medical University, Taipei City, Taiwan Department of Surgery, Division of Neurosurgery, Taipei Medical University-Shuang Ho Hospital, New Taipei City, Taiwan International Center for Health Information Technology (ICHIT), Taipei Medical University, Taipei, Taiwan
*
Author for correspondence: Ming-Chin Lin, College of Medical Science and Technology, Graduate Institute of Biomedical Informatics, Taipei Medical University, 15F, No. 172-1, Sec. 2 Keelung Road, Da’an District, Taipei City 106, Taiwan. Email: arbiter@tmu.edu.tw; Chien-Yeh Hsu, Email: cyhsu@ntunhs.edu.tw
Author for correspondence: Ming-Chin Lin, College of Medical Science and Technology, Graduate Institute of Biomedical Informatics, Taipei Medical University, 15F, No. 172-1, Sec. 2 Keelung Road, Da’an District, Taipei City 106, Taiwan. Email: arbiter@tmu.edu.tw; Chien-Yeh Hsu, Email: cyhsu@ntunhs.edu.tw
Rights & Permissions [Opens in a new window]

Abstract

Because of the increasing adoption and use of technology in primary health care (PHC), public health informatics competencies (PHIC) are becoming essential for public health workers. Unfortunately, no studies have measured PHIC in resource-limited setting. This paper describes the process of developing and validating Public Health Informatics Competencies for Primary Health Care (PHIC4PHC), an instrument for measuring PHC workers’ competencies in public health informatics. Method: This study developed a questionnaire that had three stages: the Delphi technique, a pretest, and field test. Eleven academicians from a university and 13 PHC workers joined 2 rounds of group discussion in the first stage. The second stage comprised two pilot studies with 75 PHC workers in Semarang Municipality. The third stage involved validating the questionnaire with 462 PHC workers in Kendal District. This study used Pearson’s product-moment correlation for the validity check and Cronbach’s alpha coefficient for determining the internal consistency. This study used the K-means algorithm for clustering the results of the PHIC4PHC questionnaire. Results and Conclusion: PHIC4PHC is the first comprehensive PHIC questionnaire administered in a resource-limited setting, consisting of 11 indicators and 42 measurement items concerning knowledge of health information systems, skills required for health data management, ethical aspects of data sharing and health information literacy. The final results of PHIC4PHC were clustered into three classes based on the K-means algorithm. Overall, 45.7% PHC workers achieved medium competency, whereas 25.6% and 27.7% achieved low and high competency, respectively. Men had higher competency than women. The higher the worker’s level of education, the higher the PHIC level; the longer the worker’s work experience, the lower the PHIC score; and the greater the worker’s age, the lower the PHIC score. Measuring and monitoring PHIC is vital to support successful health IT adoption in PHC.

Type
Development
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
© The Author(s) 2020

Introduction

The resolution of universal health coverage (UHC) by the UN General Assembly in December 2012 identified UHC as a central global health objective (Vega, Reference Vega2013). This challenge should place health workers at the center of each country’s response, including its stock, skill mix, distribution, productivity, and quality. Particularly, in the context of low- and middle-income countries, gaining competent health workers is a foundation for accelerating the attainment of UHC (Campbell et al., Reference Campbell, Buchan, Cometto, David, Dussault, Fogstad, Fronteira, Lozano, Nyonator and Pablos-Méndez2013).

In many low- and middle-income countries, primary health care (PHC) has been chosen as the primary strategy to achieve equitable, patient- and community-oriented and comprehensive approaches to achieving UHC (Sachs, Reference Sachs2012). Developing the function of PHC is crucial to serving people in remote areas where PHC requires improvements in infrastructure, skilled human resources in health care, appropriate health technologies, and financial support, as well as comprehensive health program management (Hall and Taylor, Reference Hall and Taylor2003; De Maeseneer et al., Reference De Maeseneer, Willems, De Sutter, Geuchte Van de Geuchte and Billings2007; WHO, 2014).

Studies have found that improving communication networks and internet availability have improved access to health information and furthermore could elevate digital health literacy in the community, including among PHC workers (Edejer, Reference Edejer2000; Berland et al., Reference Berland, Elliott, Morales, Algazy, Kravitz, Broder, Kanouse, Muñoz, Puyol and Lara2001; Cline and Haynes, Reference Cline and Haynes2001; Norman and Skinner, Reference Norman and Skinner2006b; Gilmour, Reference Gilmour2007; Bujnowska-Fedak, Reference Bujnowska-Fedak2015; Vâjâean and Bãban, Reference Vâjâean and Bãban2015). Several studies have reported the increasing use of health information technologies in various programs in PHC, such as inpatient electronic registry, processing, and evaluation programs and management, clinical decision support systems, surveillance, and patients monitoring (Pambudi et al., Reference Pambudi, Hayasaka, Tsubota, Wada and Yamaguchi2004; Tomasi et al., Reference Tomasi, Facchini and Maia2004; Ludwick and Doucette, Reference Ludwick and Doucette2009; Tomasi et al., Reference Tomasi, Facchini, Thumé, Maia, Osorio, Richard Wootton, Nivritti and Richard2009; Denomme et al., Reference Denomme, Terry, Brown, Thind and Stewart2011; Rachmani et al., Reference Rachmani, Hsu and Kurniadi2013; Yazdi-Feyzabadi et al., Reference Yazdi-Feyzabadi, Emami and Mehrolhassani2015). Some countries still report poor performance despite having dedicated information systems for PHC (Belanger et al., Reference Belanger, Bartlett, Dawes, Rodriguez and Hasson-Gidoni2012; Farahat et al., Reference Farahat, Hegazy and Mowafy2018). Different settings pose specific challenges during implementation, and human resources have been considered as the most critical among the factors that contribute to the success of health IT in PHC (Ludwick and Doucette, Reference Ludwick and Doucette2009).

In the 21st century, public health professionals face major challenges, particularly in terms of technological advances, and demographic changes (Hernandez et al., Reference Hernandez, Rosenstock and Gebbie2003). Public health informatics competencies (PHIC) have become critical to PHC workers because of the current trend of health IT adoption and its necessity for jobs in PHC to be performed efficiently (Alpay et al., Reference Alpay, Needham and Murray2000; Montague, Reference Montague2014). Public health informatics (PHI) is the application of computer science and information technology systems to public health practice, research, and learning (Friede et al., Reference Friede, Blum and McDonald1995). It integrates public health and IT and consists of four knowledge domains; organization and management systems, public health, information system and IT (Magnuson and Fu, Reference Magnuson and Fu2014). PHI could improve public health surveillance capacity and response, but confidentiality and security of the information systems is a concern (Hernandez et al., Reference Hernandez, Rosenstock and Gebbie2003).

Public health workers should be able to support public health decisions by facilitating the availability of timely, relevant, and high-quality information. In other words, they should always be able to provide advice on methods for achieving a public health goal faster, better, or at a lower cost by leveraging computer science, information science, or technology (Savel and Foldy, Reference Savel and Foldy2012). Furthermore, public health professionals in PHC need to understand many facets of health care, including public health, health promotion, health services research, and information and communication technology (Joshi and Perin, Reference Joshi and Perin2012).

Indonesia is a developing country committed to achieving UHC by 2019 (Simmonds and Hort, Reference Simmonds and Hort2013). Indonesia has 9859 PHC facilities distributed across the archipelago serving an estimated 250 million people. In 2011, among all PHC facilities, 78.4% had computers and 46% had adopted PHC information systems. (Indonesia, 2011; Indonesia, 2016). The number of PHC facilities with a health information system (HIS) has increased dramatically because of the enactment of a national social security program in 2014. In this regard, measuring the PHIC of PHC workers is crucial to ensure optimal functioning of PHC activities.

A recent study described PHIC in developed countries and for the mid-tier level of health professionals (Hsu et al., Reference Hsu, Dunn, Juo, Danko, Johnson, Mas and Sheu2012). As of yet, no studies have measured the PHIC of PHC workers in low- and middle-income countries despite their adoption of technology into PHC. The objective of this study was to develop an assessment instrument to measure PHIC in the domain of information systems and IT for PHC workers with limited education and resources in developing countries. The instrument was developed in three stages: first, constructing categories, indicators, and items for the questionnaire; second, conducting a pretest via two pilot tests; and third, conducting a field test with PHC workers. This study used the Delphi technique to construct the questionnaire items with the judgment of experts to generate the final set of questionnaire items.

Methods

Research setting and design

This study had three stages (Figure 1): the first stage was development of the instrument, the second stage was pretest studies, and the third stage was field testing of the questionnaire. The first and second stages were conducted in Semarang Municipality, Central Java Province of Indonesia. The third stage was conducted in Kendal District, Central Java Province of Indonesia in 25 PHC facilities. Semarang Municipality was used to obtain experts opinion and conduct the pretest because it is an urban city in which people are more exposed to technology, whereas Kendal District is a rural-urban city and can therefore represent the characteristic of Indonesia’s PHC facilities, which are located in both rural and urban area.

Figure 1. Steps in developing and validating a PHIC4PHC instrument

This study used the Delphi technique to challenge and construct the Public Health Informatics Competencies for Primary Health Care (PHIC4PHC) questionnaire. The Delphi technique is a method for structuring group communication using a series of questionnaires. The technique can ensure that the communication process is effective and that a consensus can be reached between the researcher and a group of experts on a specified topic. The technique is used when the opinions and judgment of experts are needed but precise information is unavailable (Hasson et al., Reference Hasson, Keeney and McKenna2000; Hsu and Sandford, Reference Hsu and Sandford2007; Keller and Heiko, Reference Keller and Heiko2014). This study employed a modified Delphi through two steps with face-to-face meetings of experts on education and PHC. The purpose of the meetings was to achieve a consensus regarding the construction of the questionnaire. Furthermore, the expert opinions were also used in the second stage to reduce the numbers of questionnaire items. The purposive snowball sampling technique was used to identify experts who know other experts with similar characteristics such as knowledge, skills, and experiences (Biernacki and Waldorf, Reference Biernacki and Waldorf1981; Palinkas et al., Reference Palinkas, Horwitz, Green, Wisdom, Duan and Hoagwood2015). Experts including academicians and PHC workers evaluated a list of potential competencies derived from a literature review. Eleven academicians participated in the first round and 13 PHC practitioners joined the second round.

An instrument agreed upon in the first stage was pretested in two pilot studies in the second stage. In the first pretest, 40 public health workers filled in the questionnaire at a Gunung Pati PHC facility, and the following pretest involved 35 public health workers at Semarang Municipality Health Office.

Figure 1 shows the flow of this study and the three-stage construction of the PHI4PHC questionnaire. The participants in the first and the second stages were 24 experts and 75 public health workers, respectively. The third stage involved 462 PHC workers filling out the PHIC4PHC questionnaire in the field test.

Stage 1: instrument development

Literature review

A literature review was performed to assess recent instruments acknowledged as information and communication technology (ICT) measurement tools to obtain a comprehensive viewpoint for developing PHIC assessment tools. Previous studies have developed instruments for measuring computer literacy, computer competency, computer knowledge, computer usage, attitudes toward computers, ICT literacy, and so forth but few have specialized in health. Table 1 provides a brief description of 10 computer literacy and competency measurement tools. The table shows that the computer-email-web fluency and eHealth Literacy Scale (eHeals) tools included the internet as a component of digital technology in their constructs; however, only one instrument is concerned with health.

Table 1. Recent instruments for measuring ICT literacy

The initial structure of PHIC assessment tool in this study was based on a framework of ICT literacy, Digital Competence Assessment, and the eHeals because this study focused on information system and IT competencies specific to the health area (Panel, Reference Panel2002; Norman and Skinner, Reference Norman and Skinner2006a; Cartelli et al., Reference Cartelli, Dagiene and Futschek2012). The concept of the PHIC assessment tool in this study consists of technical, ethical, and cognitive competencies and health information literacy.

Designing the main categories, indicators, and items

The process for developing the questionnaire items in this study was based on previous studies on ICT literacy in the health area (Jiang et al., Reference Jiang, Chen and Chen2004; Yang et al., Reference Yang, Yu, Lin and Hsu2004; Norman and Skinner, Reference Norman and Skinner2006a; Neter and Brainin, Reference Neter and Brainin2012; Gürdaş Topkaya and Kaya, Reference Gürdaş Topkaya and Kaya2015). As shown in Table 2, the PHIC4PHC has 4 main categories or domains of cognitive proficiency, technical proficiency, and ethical proficiency, and health information literacy, in addition to 12 indicators. This step created the initial version of the questionnaire (PHI4PHC v.0) consisting of 85 questions rated on a 5-point Likert scale (where 1 is strongly unimportant and 5 is strongly important) as shown in Table 3.

Table 2. Ranking of indicators in the research process

Table 3. Reduction of questionnaire items via the experts panel

Round 1: Delphi panel

In the first round, PHIC4PHC ver.0 consisting of 85 items (Table 3) was distributed to 11 academicians. They were asked to judge the importance of each question using a Likert scale ranging from 1 (strongly unimportant) to 5 (strongly important). The expert judgments of academicians were crucial to improving the content validity of the questionnaire. Items were accepted in the questionnaires construction using a mean cutoff value of 4 with standard deviation (SD) ≤ 0.75 to gain a consensus (Hsu and Sandford, Reference Hsu and Sandford2007). In this round, 42 items with a mean score of less than 4 were removed from the questionnaire construction. Furthermore, this process also eliminated the security knowledge indicator from the ethical proficiency domain. In this round, academicians reached a consensus. Table 2 shows the mean scores and ranking from the panel round. This round resulted in PHIC4PHC ver.1 (Table 3) which was later distributed to PHC experts.

Round 2: Delphi panel

In the second round, PHIC4PHC ver.1 was distributed to 13 PHC experts. The second Delphi panel added two items and reinstated one item that was judged to be unimportant by the academicians and withdrawn in the first round. These three items related to the importance of handling data and troubleshooting errors. This study labeled all items with a mean score ≥ 4 and SD ≥ 0.75 as important. The second round resulted in the PHI4PHC v.2 questionnaire (Table 3).

Table 2 shows that the academicians and PHC experts had different views about the importance of the main categories. Academicians judged the health information literacy category as the most important, whereas for the PHC expert, the technical proficiency, ethical proficiency, and health information literacy categories were the highest ranking.

Stage 2: pretesting of the questionnaire

Pilot Study 1

The PHI4PHC v.2 questionnaire was distributed to 40 staff at the PHC in Gunung Pati, Semarang. Table 5 shows the characteristics of the respondents in the pilot study.

The PHI4PHC v.2 questionnaire comprised 46 items rated using a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The construction of the PHI4PHC v.2 questionnaire included both positive and negative statements to prevent any tendency in the respondents to give the same answers. Based on PHC experts’ advice, the questionnaire included three additional items asking about computer troubleshooting skills.

The results of validity and reliability testing indicated that two questions were not valid, with item-total correlation ≤ 0.263 and a Cronbach’s alpha coefficient of 0.956. The construction of the questionnaire resulting from the pilot study was discussed with the PHC experts, who still judged that two of the items were important for measuring PHIC. Based on the discussion, the sentence ‘I know the regulations concerning the protection of personal information on computer’ was revised to ‘I know the regulations concerning the protection of patient identity on computers.’ The sentence ‘I can determine the quality of health information found via the internet’ was revised to ‘I can differentiate between correct and incorrect information found via the internet.’ This step resulted in the PHIC4PHC v.3 questionnaire (Table 4).

Table 4. The Reducing Items Questionnaire on pretest

Table 5. Characteristics of the participants

Pilot Study 2

This study distributed the PHIC4PHC v.3 questionnaire to 35 PHC workers at a monthly meeting at Semarang Municipality Health Office. In this pilot study, three questions had a total item correlation ≤ 0.283 (Items 1,30,31) and a Cronbach’s alpha coefficient of 0.923. The result of the discussion with the experts revealed items that could be withdrawn from the construction because the content could be represented by other items. In this pilot study, the experts reached a consensus on the construction of the questionnaire, thus signaling it the end of the process. This second pilot study generated the Draft PHIC4PHC. Table 4 shows the the questionnaire items.

Stage 3: field testing the PHIC4PHC questionnaire

In this stage, the Draft PHIC4PHC questionnaire from Pilot Study 2 was validated to 462 PHC workers in Kendal District. The results of validity testing indicated that Item 13 (‘I do not know network and computer application that have been used in the health information system in the workplace’) was not valid with item-total correlation of 0.087, and therefore question was removed from the construct and the following process. The next reliability test for 42 items resulted in a Cronbach’s alpha coefficient of 0.946 for the entire scale ranging from 0.944 to 0.946. All of the items were valid with an item-total correlation ranging from 0.37 to 0.80 (Table 6).

Table 6. Items analysis of the final PHIC4PHC questionnaire

Data analysis

This study applied two-stage data processing. In the first stage, a statistical method was used to validate the PHIC4PHC questionnaire. Pearson’s product-moment correlation was used to determine the internal consistency; the Cronbach’s alpha coefficient was measured for each item as well as for the entire scale. The instrument has acceptable reliability if the Cronbach’s alpha coefficient is 0.70 or above. The statistical analyses were performed using the Statistical Package for Social Science 19 (Nie et al., Reference Nie, Bent and Hull1975; Bryman and Cramer, Reference Bryman and Cramer2012).

In the second stage, the results of the field test with 42 attributes were clustered into 3 categories using a data mining technique with K-means using tool Rapid Miner 8.1 (Kotu and Deshpande, Reference Kotu and Deshpande2014). Because the PHIC4PHC questionnaire is the first tool developed to measure PHIC in developing countries, no standard categories are available for judging PHIC in such cases. Therefore, this study applied K-means to categorize the PHIC standard into three unsupervised clusters. This study classified three clusters because previous studies have widely reported the results of analyzing data sets commonly containing three clusters of observations (Hill et al., Reference Hill, Lewicki and Lewicki2006).

K-means is an unsupervised machine learning algorithm that clusters data, that are similar to one another into one cluster, which is then applied to unlabeled attribute. The K-means algorithm determines a set of K clusters and assigns each datum to exactly one cluster consisting of similar data. The similarity between data is based on a distance measure between them (Gan et al., Reference Gan, Ma and Wu2007). The parameters set in the K-means algorithm were K (3), max run (10), measure type (Bregman divergence), max optimization steps (100), and divergence (square Euclidean distance).

Figure 2 shows K-means algorithm categorization process:

  1. 1. Determine the K value (3) as the number of categories and the metric dissimilarity (distance).

  2. 2. Randomize the initial centroid of each category that will be used to the cluster data.

  3. 3. Allocate all data to the nearest centroid by calculating the distance from the data to the centroid. This study used Euclidean distance to find the distance, as follows:

    $$d\left( {{x_j},{c_j}} \right) = \sqrt {\sum\limits_{j = 1}^n {{{\left( {{x_j} - {c_j}} \right)}^2}} }$$
  4. 4. Recalculate centroid C based on data that follows each cluster as follows:

    $${c_j} = {{1}\over{{Nk}}}\mathop {\sum} \limits_{i = 1}^{Nk} {x_{jl}}$$
    where Nk is the amount of data that incorporated in a cluster.
  5. 5. Repeat Steps 3 and 4 until convergence is reached when no data switch clusters (Shidik et al., Reference Shidik, Sulistyowati and Tirta2016).

Figure 2. The K-means algorithm

Result

Participants of the study

Delphi panel

This study recruited two groups of experts to test the initial questionnaire developed based on the literature review. The criterion for expert selection was experiences and knowledge of PHC. This stage involved 11 academicians from the Faculty of Health Science, Dian Nuswantoro University, Semarang and 13 public health practitioners from Semarang Municipality Health Office. Their average age was 40.5 years and the average work experience was 10.3 years. Table 5 shows the experts profiles.

Pretest and field test

Two pilot studies were conducted for pretest study followed by a field test. In the two pilot studies, the questionnaire was distributed to 40 and 35 PHC workers on Semarang District; in the field test, questionnaire was distributed to 462 PHC workers in Kendal District, Indonesia.

Table 5 shows the characteristics of all of the participants in this study. The proportion of women was higher than that of men among the PHC workers. The most common educational background was vocational school, particulary in PHC.

Validating PHIC4PHC

PHIC4PHC was developed through the standard process of questionnaire development, consisting of the construction of items based on the literature and expert judgment, a pilot test and field test (Boynton and Greenhalgh, Reference Boynton and Greenhalgh2004; Rattray and Jones, Reference Rattray and Jones2007). The validation process is shown in Tables 3 and 4. The final PHIC4PHC questionnaire had 42 questions, as shown in Table 6.

The final results of PHIC4PHC were then clustered into three categories, based on the K-means algorithm. The results had normal distribution with 45.7% achieveing medium competency, 25.6% achieveing low competency, and 27.7% achieving high competency.

Table 7 shows that the highest proportion of PHC workers was in the medium category of PHIC (45.7%). Table 8 shows the category distribution for the level of competencies among the PHC workers.

Table 7. Centroid attributes of PHIC4PHC cluster using K-means

Table 8. Category distribution of PHIC4PHC among PHC workers

Table 8 shows that men had higher PHIC than women, and the higher the level of education among the PHC workers, the higher their PHIC; the longer the work experience among the PHC workers, the lower their PHIC; and the older the PHC workers, the lower their PHIC4PHC score.

Discussion

This PHIC4PHC is the first comprehensive questionnaire to assess the competencies required in the digital health era for PHC workers such as computer skills, ethical skills, and health literacy skills. Health literacy has become vital in the digital era because health professionals need to harness the myriad information sources as a consequence of the implementation of ICT in health care facilities (Jackson, Reference Jackson2014).

The implementation of ICT in health care institutions, particularly in low- to middle-income countries such as Indonesia, still raises the concerns about confidentiality and privacy (Koo et al., Reference Koo, O’Carroll and LaVenture2001; Luna et al., Reference Luna, Almerares, Mayan, González Bernaldo de Quirós and Otero2014). Accordingly, this study included confidentiality and privacy as one a measurement indicator to comprehensively capture PHIC.

In the first stage, this study used a modified Delphi technique, a popular strategy that combines quantitative and qualitative method (Murphy et al., Reference Murphy, Coover and Owen1989; De Villiers et al., Reference De Villiers, De Villiers and Kent2005; Fong et al., Reference Fong, Ch’ng and Por2013; Keller and Heiko, Reference Keller and Heiko2014). This study used the Delphi technique to gather the opinions and perspectives of experts, educator, and practitioners about the construction of the questionnaire because no measurement tools are available to measure PHIC in PHC, particularly in developing countries.

The round of the Delphi process that focused on the education experts resulted in the removal of the security indicator from the ethical domain in the initial construction. This indicator was related to knowledge about computer viruses and how to handle them. The ranking of the importance of indicators differed between the two groups of experts. The education experts ranked the information domain as the most important, whereas, for PHC experts, the importance was equal among domains. This study identified 42 items regarding PHIC that are crucial for PHC workers.

Unlike in previous studies, the results of PHIC4PHC were processed using a data mining technique because it provides the ability to detect the optimal combination of precise parameter that should be assigned to each of the variables for classification according to the purpose of this study (Tufféry, Reference Tufféry2011). This study applied cluster analysis because this method is mostly used when no a priori hypotheses are available and research remains in the exploratory phase. Cluster analysis is an exploratory data analysis tool that aims at sorting different objects into groups such that the degree of association between two objects is maximal if they belong to the same group otherwise minimal. Furthermore, this study did not assess statistical significance among clusters because, unlike many other statistical procedures, cluster analysis is a ‘collection’ of different algorithms that ‘place objects into clusters’ according to similarity rules. Hence, statistical significance testing is not appropriate in cluster analysis (Hill et al., Reference Hill, Lewicki and Lewicki2006).

PHIC are crucial for Indonesia’s public health workers because the specific geography of the thousands of islands of Indonesia pose a challenge for PHC service, particularly in rural areas of the country. ICT is a solution for improving effectiveness and efficiency in PHC service with the implication that public health professionals should be the earliest adopters of computers and other information technologies. PHIC will generate innovative ways to promote public health using information science and technology (Yasnoff et al., Reference Yasnoff, O’Carroll, Koo, Linkins and Kilbourne2000).

PHIC4PHC revealed that women likely have lower PHIC than men in PHC, which is consistent with the results of previous studies that a gender issue remains in ICT implementation, particularly in developing countries, despite continual claims that IT is gender-neutral (Hafkin and Taggart, Reference Hafkin and Taggart2001; Hafkin and Huyer, Reference Hafkin and Huyer2007; Flynn-Dapaah and Tareq Rashid, Reference Flynn-Dapaah and Tareq Rashid2010). The implementation of a HIS in PHC should contemplate gender issues at the early stages of ICT adoption to allow women to participate fully in using the HIS, particularly in the rural areas.

This study showed that the longer the work experience of public health practitioners, the lower their PHIC. This finding differs from that of previous studies because ICT literacy requires a certain amount of experience (Usluel, Reference Usluel2007). However, the result is unsurprising because although the duration of using ICT was related to ICT literacy, work experience was related to the age of public health practitioners and older public health practitioners have longer work experience. Older people generally have lower ICT competencies than younger people (Tijdens and Steijn, Reference Tijdens and Steijn2005), which is consistent with the finding of this study that the older public health practitioners had lower competency.

PHIC4PHC could fill the gap in the tools available for measuring the readiness of human resources in PHC institutions to adopt HISs because it can evaluate the PHIC of public health workers. The evaluation results can determine the work necessary to promote the competency of PHC workers in ICT, such as training for existing health workers and developing a curriculum, gender-specific training, education and work experience, and so forth (Hagdrup et al., Reference Hagdrup, Edwards, Carter, Falshaw, Gray and Sheldon1999).

Conclusion

This paper describes the research method for measuring PHIC in the form of a questionnaire comprising 7 indicators and 42 items. The primary indicators were cognitive proficiency, technical proficiency, ethical proficiency, and health information literacy.

Previous studies have measured PHIC in developed countries and typically for PHC workers in higher education. This PHIC4PHC is valid and reliable in measuring PHIC in urban and rural PHC facilities. The final version of the assessment tool developed in this study is expected to be used in the future study of PHI in PHC, particularly in developing countries and resource-limited settings to elevate the success of implementing ICT in health care service.

Acknowledgments

We thank Hanif Pandu Suhito for networking in PHC in Semarang and the departmental cooperation of Semarang City Health Office; all of the PHC workers in Kendal District, Central Java Province for supporting this research; Sri Wahyuni, S.Pd, M.Pd, and Dr. Drs. Slamet Isworo, M.Kes for helping to check the manuscript; Sylvia Anjani, M.Kes for helping the reaserch administration; the education staff of the Faculty of Health Science, Dian Nuswantoro University Semarang for cooperation; Indonesia Ministry of Research Technology and Higher Education for providing a scholarship.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. The author get a scholarship from Indonesia Ministry of Research Technology and Higher Education

Declaration of Conflicting Interests

The authors declare that there is no conflict of interest.

References

Alpay, L, Needham, G and Murray, P (2000) The potential of information technology for nurses in primary care: a review of issues and trends. Primary Health Care Research & Development 1, 513.CrossRefGoogle Scholar
Belanger, E, Bartlett, G, Dawes, M, Rodriguez, C and Hasson-Gidoni, I (2012) Examining the evidence of the impact of health information technology in primary care: an argument for participatory research with health professionals and patients. International Journal of Medical Informatics 81, 654661.CrossRefGoogle ScholarPubMed
Berland, GK, Elliott, MN, Morales, LS, Algazy, JI, Kravitz, RL, Broder, MS, Kanouse, DE, Muñoz, JA, Puyol, J-A and Lara, M (2001) Health information on the internet: accessibility, quality, and readability in English and Spanish. Jama 285, 26122621.CrossRefGoogle ScholarPubMed
Biernacki, P and Waldorf, D (1981) Snowball sampling: problems and techniques of chain referral sampling. Sociological Methods & Research 10, 141163.CrossRefGoogle Scholar
Boynton, PM and Greenhalgh, T (2004) Hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. BMJ: British Medical Journal 328, 1312.CrossRefGoogle Scholar
Bryman, A and Cramer, D (2012) Quantitative data analysis with IBM SPSS 17, 18 & 19: A guide for social scientists. New York: Routledge.CrossRefGoogle Scholar
Bujnowska-Fedak, MM (2015) Trends in the use of the Internet for health purposes in Poland. BMC Public Health 15, 194.CrossRefGoogle ScholarPubMed
Bunz, U (2004) The computer-email-web (CEW) fluency scale-development and validation. International Journal of Human-Computer Interaction 17, 479506.CrossRefGoogle Scholar
Calvani, A, Cartelli, A, Fini, A and Ranieri, M (2009) Models and instruments for assessing digital competence at school. Journal of E-learning and Knowledge Society 4, 183193.Google Scholar
Campbell, J, Buchan, J, Cometto, G, David, B, Dussault, G, Fogstad, H, Fronteira, I, Lozano, R, Nyonator, F and Pablos-Méndez, A (2013) Human resources for health and universal health coverage: fostering equity and effective coverage. Bulletin of the World Health Organization 91, 853863.CrossRefGoogle ScholarPubMed
Cartelli, A, Dagiene, V and Futschek, G (2012) Bebras contest and digital competence assessment: analysis of frameworks. Current Trends and Future Practices for Digital Literacy and Competence, 1, 2439.CrossRefGoogle Scholar
Cline, RJ and Haynes, KM (2001) Consumer health information seeking on the Internet: the state of the art. Health Education Research 16, 671692.CrossRefGoogle ScholarPubMed
De Maeseneer, J, Willems, S, De Sutter, A, Geuchte Van de Geuchte, I and Billings, M (2007) Primary health care as a strategy for achieving equitable carte: a literature review commissioned by the Health Systems Knowledge Network of CSDH.Google Scholar
Denomme, LB, Terry, AL, Brown, JB, Thind, A and Stewart, M (2011) Primary health care teams’ experience of electronic medical record use after adoption. Family Medicine 43, 632642.Google ScholarPubMed
De Villiers, MR, De Villiers, PJ and Kent, AP (2005) The Delphi technique in health sciences education research. Medical Teacher 27, 639643.CrossRefGoogle ScholarPubMed
Edejer, TT-T (2000) Disseminating health information in developing countries: the role of the internet. BMJ 321, 797800.CrossRefGoogle ScholarPubMed
Farahat, TM, Hegazy, NN and Mowafy, M (2018) Information and communication technologies in primary healthcare facilities in Egypt. Primary Health Care Research & Development 19, 8895.CrossRefGoogle ScholarPubMed
Flynn-Dapaah, K and Tareq Rashid, A (2010) Gender digital equality in ICT interventions in health: evidence from IDRC supported projects in developing countries.Google Scholar
Fong, SF, Ch’ng, PE and Por, FP (2013) Development of ICT competency standard using the Delphi technique. Procedia-Social and Behavioral Sciences 103, 299314.CrossRefGoogle Scholar
Friede, A, Blum, HL and McDonald, M (1995) Public health informatics: how information-age technology can strengthen public health. Annual Review of Public Health 16, 239252.CrossRefGoogle ScholarPubMed
Gan, G, Ma, C and Wu, J (2007) Data clustering: theory, algorithms, and applications. Philadelphia, PA: Society for Industrial and Applied Mathematics.CrossRefGoogle Scholar
Gilmour, JA (2007) Reducing disparities in the access and use of internet health information. A discussion paper. International Journal of Nursing Studies 44, 12701278.CrossRefGoogle ScholarPubMed
Gürdaş Topkaya, S and Kaya, N (2015) Nurses’ computer literacy and attitudes towards the use of computers in health care. International Journal of Nursing Practice 21, 141149.CrossRefGoogle ScholarPubMed
Hafkin, NJ and Huyer, S (2007) Women and gender in ICT statistics and indicators for development. Information Technologies & International Development 4, 2541.CrossRefGoogle Scholar
Hafkin, NJ and Taggart, N (2001) Gender, information technology, and developing countries: An analytic study. Washington, DC: Office of Women in Development, Bureau for Global Programs, Field Support.Google Scholar
Hagdrup, NA, Edwards, M, Carter, YH, Falshaw, M, Gray, RW and Sheldon, MG (1999) Why? What? and How? IT provision for medical students in general practice. Medical Education 33, 537541.CrossRefGoogle ScholarPubMed
Hall, JJT and Taylor, R (2003) Health for all beyond 2000: the demise of the Alma-Ata Declaration and primary health care in developing countries. The Medical Journal of Australia 178, 1720.CrossRefGoogle ScholarPubMed
Hasson, F, Keeney, S and McKenna, H (2000) Research guidelines for the Delphi survey technique. Journal of Advanced Nursing 32, 10081015.Google ScholarPubMed
Hernandez, LM, Rosenstock, L and Gebbie, . (2003) Who will keep the public healthy?: educating public health professionals for the 21st century. Washington, DC: National Academies Press.Google Scholar
Hill, T, Lewicki, P and Lewicki, P (2006) Statistics: methods and applications: a comprehensive reference for science, industry, and data mining. Tulsa, OK: StatSoft, Inc.Google Scholar
Hsu, C-C and Sandford, BA (2007) The Delphi technique: making sense of consensus. Practical Assessment, Research & Evaluation 12, 18.Google Scholar
Hsu, CE, Dunn, K, Juo, H-H, Danko, R, Johnson, D, Mas, FS and Sheu, J-J (2012) Understanding public health informatics competencies for mid-tier public health practitioners—a web-based survey. Health Informatics Journal 18, 6676.CrossRefGoogle ScholarPubMed
Indonesia MoHo (2011) Health facility survey. Jakarta, Indonesia: Research and Development of Ministry of Health.Google Scholar
Indonesia MoHo (2016) Health statistics of Indonesia 2015. Jakarta, Indonesia: MOH Indonesia.Google Scholar
Jackson, T (2014) Building the ‘continuous learning’healthcare system. Health Information Management Journal 43, 45.Google ScholarPubMed
Jiang, W-W, Chen, W and Chen, Y-C (2004) Important computer competencies for the nursing profession. Journal of Nursing Research 12, 213226.CrossRefGoogle ScholarPubMed
Joshi, A and Perin, DMP (2012) Gaps in the existing public health informatics training programs: a challenge to the development of a skilled global workforce. Perspectives in Health Information Management/AHIMA, American Health Information Management Association 9, 113.Google ScholarPubMed
Katz, IR (2007) Testing information literacy in digital environments: ETS’s iSkills assessment. Information Technology and Libraries 26, 3.CrossRefGoogle Scholar
Kay, RH (1993) A practical research tool for assessing ability to use computers: the computer ability survey (CAS). Journal of Research on Computing in Education 26, 1627.CrossRefGoogle Scholar
Keller, J and Heiko, A (2014) The influence of information and communication technology (ICT) on future foresight processes—results from a Delphi survey. Technological Forecasting and Social Change 85, 8192.CrossRefGoogle Scholar
Koo, D, O’Carroll, P and LaVenture, M (2001) Public health 101 for informaticians. Journal of the American Medical Informatics Association 8, 585597.CrossRefGoogle ScholarPubMed
Kotu, V and Deshpande, B (2014) Predictive analytics and data mining: concepts and practice with rapidminer. Wayman Street, MA: Morgan Kaufmann.Google Scholar
Ludwick, DA and Doucette, J (2009) Adopting electronic medical records in primary care: Lessons learned from health information systems implementation experience in seven countries. International Journal of Medical Informatics 78, 2231.CrossRefGoogle ScholarPubMed
Luna, D, Almerares, A, Mayan, JC, González Bernaldo de Quirós, F and Otero, C (2014) Health informatics in developing countries: going beyond pilot practices to sustainable implementations: a review of the current challenges. Healthcare Informatics Research 20, 310.CrossRefGoogle ScholarPubMed
Magnuson, J and Fu, PC (2014) Public health informatics and information systems. London: Springer.CrossRefGoogle Scholar
Miller, LA, Stanney, KM and Wooten, W (1997) Development and evaluation of the Windows Computer Experience Questionnaire (WCEQ). International Journal of Human-Computer Interaction 9, 201212.CrossRefGoogle Scholar
Montague, E (2014) The promises and challenges of health information technology in primary health care. Primary Health Care Research & Development 15, 227.CrossRefGoogle ScholarPubMed
Murphy, CA, Coover, D and Owen, SV (1989) Development and validation of the computer self-efficacy scale. Educational and Psychological Measurement 49, 893899.CrossRefGoogle Scholar
Neter, E and Brainin, E (2012) eHealth literacy: extending the digital divide to the realm of health information. Journal of Medical Internet Research 14, e19.CrossRefGoogle ScholarPubMed
Nie, NH, Bent, DH and Hull, CH (1975) SPSS: Statistical package for the social sciences: McGraw-Hill New York.Google Scholar
Norman, CD and Skinner, HA (2006a) eHEALS: the eHealth literacy scale. Journal of Medical Internet Research 8, e27.CrossRefGoogle ScholarPubMed
Norman, CD and Skinner, HA (2006b) eHealth literacy: essential skills for consumer health in a networked world. Journal of Medical Internet Research 8, e9.CrossRefGoogle Scholar
Palinkas, LA, Horwitz, SM, Green, CA, Wisdom, JP, Duan, N and Hoagwood, K (2015) Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research 42, 533544.CrossRefGoogle ScholarPubMed
Pambudi, IT, Hayasaka, T, Tsubota, K-i, Wada, S and Yamaguchi, T (2004) Patient Record Information System (PaRIS) for primary health care centers in Indonesia. Technology and Health Care 12, 347357.CrossRefGoogle ScholarPubMed
Panel, IL (2002) Digital transformation: A framework for ICT literacy. Educational Testing Service.Google Scholar
Panero, JC, Lane, DM and Napier, HA (1997) The computer use scale: four dimensions of how people use computer. Proceedings of the Seventh International Conference on Human-Computer Interaction-Volume 1-Volume I. Elsevier Science Inc., 821824.CrossRefGoogle Scholar
Potosky, D and Bobko, P (1998) The computer understanding and experience scale: a self-report measure of computer experience. Computers in Human Behavior 14, 337348.CrossRefGoogle Scholar
Rachmani, E, Hsu, C and Kurniadi, A (2013) How health information system could help the leprosy control program in Indonesia? International Conference On Instrumentation, Communication, Information Technology and Biomedical Engineering. Bandung, Indonesia, 1620.Google Scholar
Radcliff, CJ, Salem, J, O’Connor, L and Gedeon, JA (2007) Project SAILS Skill Sets for the 2016-2017 Academic Year. Project SAILS, available online at https://www.projectsails.org/abouttest/skillsets.php.Google Scholar
Rattray, J and Jones, MC (2007) Essential elements of questionnaire design and development. Journal of Clinical Nursing 16, 234243.CrossRefGoogle ScholarPubMed
Sachs, JD (2012) Achieving universal health coverage in low-income settings. The Lancet 380, 944947.CrossRefGoogle ScholarPubMed
Savel, TG and Foldy, S (2012) The role of public health informatics in enhancing public health surveillance. MMWR: Surveillance Summaries 61, 2024.Google ScholarPubMed
Shidik, GF, Sulistyowati, NS and Tirta, MB (2016) Evaluation of cluster K-Means as VM selection in dynamic VM consolidation. 2016 22nd Asia-Pacific Conference on Communications (APCC). IEEE, 124128.CrossRefGoogle Scholar
Simmonds, A and Hort, K (2013) Institutional analysis of Indonesia’s proposed road map to universal health coverage. Melbourne, Australia: Noosal Institute for Global Health, University of Melbourne.Google Scholar
Smith, B, Caputi, P and Rawstorne, P (2007) The development of a measure of subjective computer experience. Computers in Human Behavior 23, 127145.CrossRefGoogle Scholar
Tijdens, K and Steijn, B (2005) The determinants of ICT competencies among employees. New Technology, Work and Employment 20, 6073.CrossRefGoogle Scholar
Tomasi, E, Facchini, LA and Maia, MdFS (2004) Health information technology in primary health care in developing countries: a literature review. Bulletin of the World Health Organization 9, 867874.Google Scholar
Tomasi, E, Facchini, LA, Thumé, E, Maia, FS and Osorio, A (2009) Information technology for primary health care in Brazil. In Richard Wootton, KH, Nivritti, G Patil and Richard, E Scott, editors, Telehealth in developing countries. Canada: Royal Society of Medicine Press Ltd.Google Scholar
Tufféry, S (2011) Data mining and statistics for decision making. West Sussex: John Wiley & Sons.CrossRefGoogle Scholar
Usluel, YK (2007) Can ICT usage make a difference on student teachers’ information literacy self-efficacy. Library & information science research 29, 92102.CrossRefGoogle Scholar
Vâjâean, CC and Bãban, A (2015) Emotional and behavioral consequences of online health information-seeking: the role of ehealth literacy. Cognitie, Creier, Comportament/Cognition, Brain, Behavior 19, 327.Google Scholar
Vega, J (2013) Universal health coverage: the post-2015 development agenda. The Lancet 381, 179.CrossRefGoogle ScholarPubMed
WHO (2014) The world health report 2008: rimary health care (now more than ever). Geneva: World Health Organization.Google Scholar
Yang, K-F, Yu, S, Lin, M-S and Hsu, C-L (2004) Study of basic computer competence among public health nurses in Taiwan. Journal of Nursing Research 12, 15.CrossRefGoogle ScholarPubMed
Yasnoff, WA, O’Carroll, PW, Koo, D, Linkins, RW and Kilbourne, EM (2000) Public health informatics: improving and transforming public health in the information age. Journal of Public Health Management and Practice 6, 6775.CrossRefGoogle ScholarPubMed
Yazdi-Feyzabadi, V, Emami, M and Mehrolhassani, MH (2015) Health information system in primary health care: the challenges and barriers from local providers’ perspective of an area in Iran. International Journal of Preventive Medicine 6, 57.Google ScholarPubMed
Figure 0

Figure 1. Steps in developing and validating a PHIC4PHC instrument

Figure 1

Table 1. Recent instruments for measuring ICT literacy

Figure 2

Table 2. Ranking of indicators in the research process

Figure 3

Table 3. Reduction of questionnaire items via the experts panel

Figure 4

Table 4. The Reducing Items Questionnaire on pretest

Figure 5

Table 5. Characteristics of the participants

Figure 6

Table 6. Items analysis of the final PHIC4PHC questionnaire

Figure 7

Figure 2. The K-means algorithm

Figure 8

Table 7. Centroid attributes of PHIC4PHC cluster using K-means

Figure 9

Table 8. Category distribution of PHIC4PHC among PHC workers