Page 104 - JCTR-11-2

Journal of Clinical and
Translational Research Review of research landscapes and quality

Table 1. Research quality domains and items to be used in the screening tool (total score: 0 – 10)
Relevance (3 items; subtotal Credibility (4 items; subtotal score: 0 – 4) Usefulness (3 items; subtotal score: 0 – 3)
score: 0 – 3)
- Scientific relevance: Indicate this with - Data-collection design: Appropriate for the research question ; - Important outcomes used and reported c
b
an acceptable literature review or experimental vs. non-experimental; time feature of variables - Meaningful estimates: Practical numerical
citing systematic reviews a considered results, taking into consideration the
- Societal relevance: Research area - External validity: Representative of or generalizability to an response rate, missing data, proper
or involvement of end-users (e.g., important and relevant population; comparability between statistical test, and analysis d
patients) groups in randomized control trials - Conclusion: Accounts for study
- Research team/experts: The research - Internal validity: Validated instrument, measurement process, limitations e
is led by experts in the relevant field or and performed by trained or blinded assessors
conducted with relevant experts - Precision: Appropriate sample size estimation and
achievement
Notes: Set the right research priorities, clear research question/hypothesis; Ethical conduct and patient safety/rights/priorities included; Outcomes
a
b
c
that truly matter to patients; The study provides useful data for the intended end-users, unusual or unexpected analysis is explained and justified;
d
e No over-claimed or misleading conclusion.
inter-rater reliability agreement using Cohen’s kappa κ and that the research sets out on a justified scientific foundation
intra-class correlation (ICC). The kappa κ is a measure and is informed by existing evidence. Thus, scientifically
of agreement between different observers beyond chance relevant research is usually globally relevant due to its
agreement. The κ statistic will be computed separately for highly generalizable topics and subjects.
31
each domain’s item (0 or 1). The ICC will be used to assess Societal relevance refers to research that addresses a
the three domains’ subtotal (3, 4, and 3) and the total score true and real societal problem. This relevancy may exist in
(Table 1). smaller (e.g., a particular condition or disease in a unique
The kappa result will be interpreted as follows: Values ≤0 population) or wider (e.g., global) populations. These two
indicate no agreement; 0.01 – 0.20 indicate slight agreement; domains of scientific and societal relevance relate to having
0.21 – 0.40 indicate fair agreement; 0.4 – 0.60 indicate novelty in the research.
moderate agreement; 0.61 – 0.80 indicate substantial The last domain in the relevance category refers to
agreement; and 0.81 – 1.00 indicate almost perfect the research team; that is, investigators and experts of
agreement. 32,33 For the ICC, values <0.40 indicate poor relevant professional qualifications. This may include
correlation; 0.40 – 0.59 indicate fair correlation; 0.60 – 0.74 patients and the public in certain research areas when the
indicate good correlation; and 0.75 – 1.0 indicate excellent opinions of end-users are considered important, such as
correlation. 34,35 We specify that an a priori level of κ > 0.60 in interventions or the experiences of patients or family
and ICC > 0.75 must be achieved before Phase 2 of the members.
study begins. Retraining and reassessment of the reviewers
on different articles will be conducted until the inter-rater 2.5.2. Credibility
agreement reaches the desirable levels. The expected lower This category is assessed after it is judged that the research
bound of a 95% confidence limit for κ is no <0.60, with an is relevant. Four essential features are considered the
assumed marginal prevalence of zero score of 30%. Using minimum requirements for a research study to be credible
alpha and beta error rates of 0.05 and 0.2, respectively, a and for its results to inform or contribute to practice
pair of reviewers will rate 20 papers each, 34,35 with five pairs change; that is, data collection design, precision, external
of reviewers and 100 samples for the subtotal and total ICC validity, and internal validity.
estimation. 33
The design of the data collection process has to align
2.5. Research quality domains for screening with the research objective or question. The approach
used in data collection depends on whether the research
2.5.1. Relevance
is causal or non-causal, as well as whether the research
The relevance of research will be assessed from three is experimental or non-experimental, to provide more
perspectives: scientific relevance, the composition of the accurate data. The time feature or characteristics of the
research team, and societal relevance. Research is considered variables involved in the research should be collected in
scientifically relevant if it addresses a true and real scientific their intended phases or stages, such as a risk factor in the
problem and provides the needed knowledge to understand asymptomatic phase, or symptoms or biomarkers in the
an existing phenomenon. Scientific relevance also denotes latent period.

Volume 11 Issue 2 (2025) 98 doi: 10.36922/jctr.24.00071