Quality of online health information about oral contraceptives from Hebrew-language websites

The Israeli context

A recent Israeli survey of a nationally representative sample of just over 7,000 7th–12th graders in 158 Ministry of Education schools found that among the 98.7% of adolescents who have Internet access, 52% had searched for health-related information online in the 12 months prior to the survey. While many information-seeking adolescents access OHI by going directly to specific health websites and portals, a much greater proportion (70%) seek information using search engines, primarily Google.co.il, Finder, and Kafe [41]. About one third (35%) of the adolescent Internet users reported being aware of the problematic nature and biases of OHI, but few view online information critically. Only 43% of OHI-seekers assess the accuracy of the information they find on the Internet. In a survey of Israeli primary care physicians, one-third (32.3%) expressed concerns about the quality of the information retrieved by their patients [42].

While the majority of adolescent Internet users in Israel seek OHI, there has been little, if any, assessment of the quality of Hebrew-language OHI, particularly on topics relevant to this population group [43]. The provision of high quality online information about oral contraceptive (OC) use is a priority for a number of reasons. Availability of quality information on OC has been shown to improve compliance with medical instructions, and avoid complications and misuse [44], and the Internet is an efficient and effective channel for delivery of sexual and reproductive health promotion messages [45]. In Israel about one-third of 12th grade girls have experienced coitarche [46]; over 40% of sexually active adolescents use OC [47], as much as 60% of adolescents harbor misconceptions about OC use [47], and about 15% of women under the age of 20 have sought termination of pregnancy [48]. Moreover, one-quarter of Israeli adolescents who searched online for health information looked for information about OC [41]. This study aims to 1) assess the accuracy/completeness, credibility, and usability of Hebrew online (non-user-generated) content on OC; 2) assess the correlation between these three parameters; and 3) examine the extent to which the quality of information and the correlation across the three parameters varies according to the type of information provider (e.g., HMO, promotional websites).

Search strategy

The selected websites were chosen based on the results of a national survey that identified web-sources commonly used by Israeli school adolescents when seeking OHI [41]. According to the survey, over 70% of students indicated having sought health information through Google.co.il, Finder, and Kafe. Five other search engines (Tapuz, Nana10, Walla, Bing, and Yahoo) were also mentioned in relation to searching for OHI, but less frequently. Therefore, several searches were conducted using identical search terms in Hebrew using Bing, Finder, Google, Kafe, Nana10, Walla, and Yahoo. Tapuz was excluded as it searches only within the Tapuz website itself.

We gathered the first 100 search results for each search term [various Hebrew permutations of birth control pill, birth control pills, pregnancy prevention pills, and contraceptives] on 6 search engines. The seventh search engine–Nana10, produced fewer than 30 results per search term and therefore all of Nana10’s results were added to the list. To increase the number of possible websites to be assessed and in order to avoid duplicates, the results were organized into four groups based on the type of search engine: 1) Google and Google-powered search engines–Finder and Kafe, 2) Yahoo-powered search engines–Yahoo and Walla, 3) Bing, and 4) Nana10. For each search term, we compared the results from the four search engine groups and excluded overlapping items. A list of 158 websites was compiled, after combining the results of the search of the six search-terms and excluding non-Hebrew websites. Of these, 27 were unique (i.e., not duplicates), accessible, active, and provided (non-user-generated) OC information on at least one webpage. This list contained the websites of two of the four major Israeli HMOs (Health Management Organizations – Kupat Holim), so the websites of the other two HMOs were added in order to assess the quality of all four HMOs. Thus, 29 websites were selected for evaluation. Within each website, text searches were conducted to identify all the pages included in the same domain name with at least one title and one paragraph with information on OC use. When available, non-OC related pages of the website (such as About pages, Contact pages, and pages with information on Privacy policy) were also evaluated to assess parameters of credibility (e.g., website’s funding, authors, and age policies) and usability (e.g., consistency in design and function).

Website quality evaluation

The quality of each website was assessed on three key parameters: accuracy/completeness, credibility, and usability.

Accuracy and completeness

Accuracy/completeness of the OC content was assessed based on 13 established content items selected by a family planning expert from contraception guidelines [49]. Items that were misperceived by Israeli physicians in a study on OC misconceptions [47] (perhaps due to inconclusive scientific evidence) were excluded, as we were unable to definitively classify the information as accurate or inaccurate. The final list included items such as efficacy (e.g., of 1000 women using the pill as prescribed, 3 will become pregnant within a year), effectiveness in the first year of use (e.g., about 8% of typical users initiating OC will become pregnant in the first year of use), safety, reversibility, suggested frequency of use (e.g., pills should be taken daily), STI protection (e.g., pills do not protect against STIs), side effects (e.g., headaches, nausea, vomiting, breast pain/tenderness, weight gain, fatigue, mood changes), dose (e.g., take one pill a day until the pack is finished), back-up contraceptive method (e.g., use a backup contraceptive until taking 7 consecutive pills, if missing a pill, if late starting a new pack, or in case of severe vomiting or diarrhea), or instructions in case of missing a pill (e.g., taking another pill within 12 hours will still prevent pregnancy). The 13 items were assessed on a 4-point scale [inaccurate information (−1), information not presented (0), information partially presented (1), and information complete and accurate (2)], and computed on a composite percentage scale ranging from 0 to 100%, with a score of 26 (the maximum possible for this 13-item scale) representing 100%.

Credibility

The HONcode principles were developed to assess the credibility of OHI [35]. The eight HONcode principles address the web-author’s authoritativeness, the extent to which the information supports, rather than usurps, the doctor-patient relationship, stated privacy policy, attribution of the information to its sources, the extent to which the site backs up claims relating to benefits and performance, accessibility of the web-editor, financial disclosure, and advertising policy. Based on these original eight principles, a set of 14 questions was developed, each on a 3-point scale [parameter absent (1), parameter partially addressed (2), and parameter fully addressed (3)], and computed on a composite percentage scale ranging from 0 to 100%, with a score of 42 (maximum possible for the 14-item scale) representing 100%.

Usability

Basic aspects of the websites’ usability were also assessed by applying an 11-item instrument selected from the Minervation Validation instrument (questions 2.1.2-2.1.5, 2.2.1-2.2.3, and 2.3.4) [50] and/or the University of Michigan “website evaluation checklist” (questions 12, 28, 30, 34, and 35) [51]. The items included aspects such as a “return to home page” option on all pages, consistency of functions and design, and appropriateness of content for the target adolescent audience (e.g., readability as assessed by amount and size of text, use of age-appropriate medical and scientific terminology). Each item was assessed on a 3-point scale [parameter absent (1), parameter partially addressed (2), and parameter fully addressed (3)], and computed on a composite percentage scale ranging from 0 to 100% with a score of 33 (maximum possible for the 11-item scale) representing 100%.

Classification of websites

The websites were classified as belonging to one of the following 5 categories: a) HMOs (i.e., websites of the four leading Kupat Holim), b) health portals (i.e., websites that provide information on a wide scope of health and medical subjects), c) contraception-specific (i.e., websites dedicated primarily to informing the public on the subject of contraception), d) promotional and commercial (i.e., websites whose goal is to promote a certain product, service, or brand), e) life style and women’s websites (i.e., websites that provide information on contraception in the context of women’s wellbeing and child-rearing advice).

Reliability analyses

All websites were evaluated by one of us (LF). Following analyses of the data and in response to a reviewer’s suggestion on a previous version of this report, six randomly selected websites (a total of 38 web pages) were independently assessed by YN and RS, and reassessed by LF. Some inter-rater variation was observed for accuracy/completeness and usability (Cronbach’s alpha coefficients–0.84 and 0.73, respectively; Additional file 1). A greater degree of variation was noted for the 15-item credibility scale, which revealed a Cronbach‘s alpha coefficient of 0.66. This value increased to 0.75 upon deletion of one item (“Is there a clear statement regarding the minimum age for the use or participation on the website?”). The credibility parameter for all 29 websites was therefore reanalyzed using the 14-item scale.

Statistical analysis

Mean scores, standard deviations (SD), ranges, and percentages were calculated for all websites and for category-specific websites. Pearson correlation coefficients of the three parameters assessed were estimated for all, and for category-specific websites.

Statistical analyses were performed using SPSS 17.0.

Accuracy/Completeness

The mean score for accuracy/completeness was 50.9% (SD=30.1%) for all websites evaluated. Contraception-specific websites contained the most accurate and complete information (mean=66.7%) compared with a mean of 62.2% for health portals, 50.0% for HMOs and life style and women’s health websites, and 40.4% for promotional websites.

As seen in Figure 1, variance of accuracy/completeness scores was highest for the HMO websites (SD=42.7%, range=8–92%), followed by promotional websites (SD=27.8%, range=12–89%), life style and women’s health websites (SD=24.9%, range=12–81%), health portals (SD=29.4%, range=31–100%), and contraception-specific websites (SD=37.8%, range=23–89%).

Credibility

Two-thirds (n=18–19, 62.1–65.5%) of the websites stated that the health information they contained should not replace the advice of a health professional, and provided information about the organization that owns the website. Just over half (n=15–17, 51.7–58.6%) of the websites mentioned their content author’s qualifications and expertise, stated that information presented reflects only the opinion of the author, and presented their privacy policy. About three-quarters (n=21–22, 72.4–75.9%) of the websites contained information as to their intended audience and purpose, and provided information about different contraception methods. Nearly all websites had a contact e-mail address (n=28, 96.6%). Credibility parameters that were commonly missed included sources of funding, webmaster contact information, date of content creation and last modification, explicit reference to scientific evidence-based information, and references and citations (Table 1).

The mean score for credibility was 70.6% (SD=15.1%) for all websites evaluated. The HMOs’ websites contained the most credible information (80.0%), while contraception-specific websites had the lowest credibility score (65.0%). Health portals’ credibility (76.2%) was similar to that of the HMOs’ websites and higher than that of the promotional websites (67.1%) and life style and women’s health websites (67.3%).

For all websites, the lowest mean credibility score was 38% and the highest was 98%. Promotional websites (SD=16.8%, range=38–88%) and life style and women’s health websites (SD=17.5%, range=48–86%) exhibited the greatest variability followed by HMO websites (SD=15.5% range=62–98%). Health portals (SD=11.8%; range=60–88%) and contraception-specific websites (SD=7.2%, range=57–71%) showed the least variation in their credibility scores (Figure 1).

Usability

Except for one item (location inside the website), usability items were commonly presented or addressed (n=18–29, 62.1–100%; Table 1). The average usability scores for all websites was 94.5% (SD=6.9%) and were virtually identical across the HMOs’ websites (96.3%), life style and women’s health websites (96.0%), health portals (95.5%), promotional websites (93.7%), and contraception-specific websites (90.0%).

As seen in Figure 1, the greatest variation in usability scores was noted among promotional websites (SD=8.4%, range=79–100%), followed by the HMOs’ websites (SD=7.5%, range=85–100%), contraception-specific websites (SD=7.5%, range=82–97%), health portals (SD=6.2%, range=85–100%), and life style and women’s health websites (SD=5.3%, range=88–100%).

Inter-parameter correlations

HMO websites for youth

The four HMOs’ websites differ in their approach to adolescent health-information seekers. HMO-A’s website targets adolescents directly by dedicating a forum (one of nearly 40 forums) entirely to the adolescent audience. The forum’s specific information compensates to some extent for the insufficient and rather general scope of the editorial content for adolescents. Other than the forum, relevant information for youth, such as OC information, appears in the different website sections. No section, however, is designed specifically for adolescents or brings together information on health issues that are of greatest concern to young adults. At the time this review was conducted, HMO-A’s forums were only compatible with Internet Explorer and not with other popular browsers such as Mozilla, Firefox, and Google Chrome.

HMO-B’s website structure is guided by a concern for providing information about health topics (e.g., nutrition and physical activity) and situations requiring medical assistance (e.g., emergency medicine, pregnancy, and childbirth), rather than for directly addressing specific populations and audiences, such as adolescents. HMO-B’s website has a “Youth” section that features articles on topics such as smoking cessation, drug abuse, and medical screening checkups, although these articles seem to have in mind adults and parents more than adolescents.

HMO-C has a well-developed youth section on its website, featuring articles on a variety of topics, including drugs, mental health, nutrition, and teenage pregnancy. Yet, as with HMO-B, the articles seem to be written primarily for an adult audience, perhaps for concerned parents, and to a much lesser extent for teenagers seeking information on health issues. For example, in their youth section, there is a call to parents to provide adolescents with proper health information on sex, but no such information is actually provided.

HMO-D’s website approaches adolescent health information seekers by offering adolescents the option of asking a physician health questions through its website. Very few answers are published on the website, and interestingly, some of the questions were from parents, indicating that the online Q&A service is not exclusively for adolescents. Since the answers are not published, there is very little information on the website that addresses adolescents directly or even indirectly.

The effect of each particular approach on the process of health information seeking and health seeking behaviors requires further study. Interpretation and application of results of this website-evaluation study warrant a certain degree of caution. Although a comprehensive standard electronic evaluation form was designed and approved by the research team, the relatively low inter-rater reliability score for credibility suggests that credibility parameters might fluctuate from one page to another within a single website, or that the interpretation of these parameters is sensitive to the evaluators’ knowledge and beliefs. Second, the large number of websites and the great amount of online information on a seemingly infinite number of health-related subjects make it nearly impossible to generate a sample of websites that could be considered representative. To compensate, the evaluated websites were selected in such a way so as to resemble the array of OC websites that would be retrieved by an average Israeli adolescent health-information seeker. Finally, this evaluation was restricted to content posted by the website developers and excluded user-generated content (e.g., forums and social networks) that would require a different set of assessment tools [2].