A Quarterly Review of the eFiles Customer Experience Index (CEI) - Q3

This review of the CEI data accounts for the data captured between July 1 – September 30, 2024.

271 Completed Surveys July 1 to September 30, 2024

Averages of the CEI question survey results by question:

1. Staff connected with this eFile (e.g., file coordinators, reviewers, senior reviewers, etc.) were helpful and responsive	4.6/5   Indicating an average response between “Agree” and “Strongly Agree”
2. Comments and suggestions in response letters, calls and messenger were clear and actionable.	4.5/5 Indicating an average response between “Agree” and “Strongly Agree”
3. I felt the review was: 1. Highly inconsistent 2. Somewhat inconsistent 3. Somewhat consistent 4. Highly Consistent 5. I don't know	3.8/4*   Indicating an average response between “Somewhat consistent” and “Highly Consistent”
*This average rating is calculated from Responses 1-4 as including ratings of 5 (or “I don’t know”) would have skewed the average upward. There was 4 ratings of “I don’t know” in this data set.
4. Please provide any other feedback specific to this file: [optional open text field]	See feedback themes below.
5. Please rate your overall experience with this particular review  1 – highly negative experience 10 – highly positive experience	8.8/10   Indicating a positive average overall experience.

Key Themes and Actions Taken from Open-text Feedback:

1. Appreciation for quick ARO reviews, PAABs new services such as minor updates, and helpful interactions with reviewers (n=14). A significant number of positive responses and comments persisted. Comment highlighted positive experiences with the review process, reviewer interactions, eFiles upgrades, and policy updates. This is extremely helpful for providing positive feedback to reviewers about what clients are finding particularly helpful.
   Action Taken: Reviewers were provided a summary of key features that resulted in a positive experience for clients with the goal of reinforcing these behaviours.   

2. Clear and Actionable Feedback (n=11). Some clients reported that it would be helpful if comments from reviewers were more specific and actionable (n=5). In one instance a reviewer did not provide specific suggestions for how French translation could be adapted to better match English copy. In the other instances, the comment from the reviewer was felt to be vague resulting in additional rounds of comments to clarify the initial request. Other clients reported receiving clear and actionable feedback (n=6).

   Action Taken: Reviewers have been reminded to provide as much specificity as possible when commenting and to provide examples and suggestions when possible and where appropriate.

3. Changes required to previously approved content (n=3). In one instance, a reference was questioned upon renewal that had been previously accepted. This was due to the claim being modified in the new APS and placed within a different context. This resulted in the claim requiring different substantiation than the prior claim. In another instance, an oversight was made in a previous version of a data presentation that was required to be corrected. In the third instance, the market had changed since the previous copy was accepted and the market entry of a new product changed the acceptability of the prior claim.

   Action Taken: Reviewers have been reminded to provide clear and detailed rationales when previously accepted copy is questioned to avoid extra rounds of review for clarification purposes.

4. Variability in interpretation of patient visuals and visuals that may imply quality of life (QOL) (n=1).

   Action Taken: Visuals and creative are inherently subjective and we understand how it can be perceived as inconsistent when two visuals that are similar on the surface generate different comments during the review. Over the past year, the Creative Committee has been diligently working toward providing a document that outlines a refined approach to reviewing creative and outlining considerations for agencies when undertaking creative development. We are hopeful that this document will provide guardrails when creating and interpreting creatives along with alignment on language when discussing creatives. We look forward to greater alignment with industry as it relates to creative development and review.

 Key Takeaways:

• Survey Completion Rate is 21%, with this data capturing 271 responses out of 1317 surveys sent. Data should be interpreted with this in mind, as this is a relatively low sample size.
• Ratings for all questions, on average, have been positive. This data set is reflective of the complete quarter. Results have remained generally positive and consistent with the data from all quarters of 2023.

A Quarterly Review of the eFiles Customer Experience Index (CEI) - Q2

This review of the CEI data accounts for the data captured between April 1 – June 31, 2024.

322 Completed Surveys April 1 to June 31, 2024

Averages of the CEI question survey results by question:

1. Staff connected with this eFile (e.g., file coordinators, reviewers, senior reviewers, etc.) were helpful and responsive	4.5/5   Indicating an average response between “Agree” and “Strongly Agree”
2. Comments and suggestions in response letters, calls and messenger were clear and actionable.	4.4/5 Indicating an average response between “Agree” and “Strongly Agree”
3. I felt the review was: 1. Highly inconsistent 2. Somewhat inconsistent 3. Somewhat consistent 4. Highly Consistent 5. I don't know	3.7/4*   Indicating an average response between “Somewhat consistent” and “Highly Consistent”
*This average rating is calculated from Responses 1-4 as including ratings of 5 (or “I don’t know”) would have skewed the average upward. There was 4 ratings of “I don’t know” in this data set.
4. Please provide any other feedback specific to this file: [optional open text field]	See feedback themes below.
5. Please rate your overall experience with this particular review  1 – highly negative experience 10 – highly positive experience	8.8/10   Indicating a positive average overall experience.

Key Themes and Actions Taken from Open-text Feedback:

1. Reviews could have been completed in less rounds (n=6) In one instance, the submitters felt the review responses could have been more complete to help streamline revisions in a more efficient manner to reduce the number of rounds. In another instance it took multiple rounds to come to a mutual understanding of the purpose of the copy and nature of the reference.

Action Taken: Reviewers have been reminded to take time to clearly explain the relevant issues and how they may impact other parts of the APS. While the expectation is not to provide an exhaustive list of all possible outcomes, they have been encouraged to look for solutions and be clear about the rationale to assist clients in their revisions. 

2. Late-stage comments delayed timeline (n= 1) In one instance a late-stage comment arose that required additional rounds of review and resulted in delays on a file which was submitted through the accelerated review options (ARO) to address a tight timeline by the client.

Action Taken: Reviewers have been reminded that while we can all appreciate that late-stage comments may arise on occasion, the reviewer should work proactively with the client to ensure that it does not result in delays to approval. This may include proactively reaching out through phone, email or eFiles, and expediting the revision turnaround when the file is returned.

 

3. Positive comments and directions around the reviewer interactions and process (n= 13): There was a significant increase in the number of positive responses and comments provided around the review process and reviewer interactions. This is extremely helpful for providing positive feedback to reviewers about what clients are finding particularly helpful.

Action Taken: Reviewers were provided a summary of key features that resulted in a positive experience for clients with the goal of reinforcing these behaviours.  

 Key Takeaways:

• Survey Completion Rate is 8%, with this data capturing 322 responses out of 1477 surveys sent. Data should be interpreted with this in mind, as this is a relatively low sample size.
• Ratings for all questions, on average, have been positive. This data set is reflective of the complete quarter. Results have remained generally positive and consistent with the data from all quarters of 2023.

A Quarterly Review of the eFiles Customer Experience Index (CEI) - Q1

This review of the CEI data accounts for the data captured between January 1 – March 31, 2024.

231 Completed Surveys January 1 to March 31, 2024

Averages of the CEI question survey results by question:

1. Staff connected with this eFile (e.g., file coordinators, reviewers, senior reviewers, etc.) were helpful and responsive	4.6/5   Indicating an average response between “Agree” and “Strongly Agree”
2. Comments and suggestions in response letters, calls and messenger were clear and actionable.	4.4/5 Indicating an average response between “Agree” and “Strongly Agree”
3. I felt the review was: 1. Highly inconsistent 2. Somewhat inconsistent 3. Somewhat consistent 4. Highly Consistent 5. I don't know	3.7/4*   Indicating an average response between “Somewhat consistent” and “Highly Consistent”
*This average rating is calculated from Responses 1-4 as including ratings of 5 (or “I don’t know”) would have skewed the average upward. There was 4 ratings of “I don’t know” in this data set.
4. Please provide any other feedback specific to this file: [optional open text field]	See feedback themes below.
5. Please rate your overall experience with this particular review  1 – highly negative experience 10 – highly positive experience	8.8/10   Indicating a positive average overall experience.

 

Key Themes and Actions Taken from Open-text Feedback:

1. Perceived inconsistencies with past rulings (n= 6) There have been six comments pertaining to inconsistencies or perceived inconsistencies with previous rulings.

   Action Taken: Reviewers have been reminded to review backfiles when appropriate to ensure consistency where appropriate. In most instances, when the reviewer was made aware of the inconsistency, the copy was ultimately accepted.

2. Reviews could have been completed in less rounds (n=4) In four instances, the submitters felt the review could have been completed in fewer rounds. Reasons for the perceived delays included images being FPO in the layout (n=1), the submitter not having access to the appropriate eFiles associated with linked APS (n=1), submission-related issues (n=1) and a late comment from the reviewer (n=1).

   Action Taken: Reviewers have been reminded to ensure review letters are complete upon   sending to avoid late comments. We encourage submitters to ensure to review the Guidance on Submission Process and Format Requirements and ensure that the submission is complete to avoid delays. If the APS includes links to other PAAB-accepted APS, it is important to ensure those eFile numbers are included within the copy to allow for assessment of the linkages. PAAB has created a manufacturer portal where the sponsor has access to all their ongoing and past files. If a manufacturer is unable to locate and provide a past eFile number to the agency, our file coordinators may be able to help find this information for the new agency of record if provided written permission from the manufacturer. Please note that this option should be utilized only in exceptional circumstances. As a long-term solution, PAAB is diligently working on AI that will help automate these processes in the future.

   For FPO images, we understand that often submitters will not purchase the stock image until PAAB approval. To avoid delays or further questions, please confirm in your cover letters that these are intended to be the final images for purchase and will not be changed post-approval. We have also reminded reviewers that generally images with an FPO watermark are the intended final images and that any issue with the FPO image should be raised on review of the layout to avoid delays.

3. Comment lacked clear substantiation from the PAAB code (n= 1):

   Action Taken: Reviewers have been reminded to ensure all relevant code sections are included to support review comments.

Key Takeaways:

• Survey Completion Rate is 18.5%, with this data capturing 235 responses out of 1254 surveys sent. Data should be interpreted with this in mind, as this is a relatively low sample size.
• Ratings for all questions, on average, have been positive. This data set is reflective of the complete quarter. Results have remained generally positive and consistent with the data from all quarters of 2023.

Have your voice heard! Help us in continually improve by completing your CEI surveys. You can find them in the “My CEI Surveys” Tab in the top navigation bar in eFiles. This helps us identify trends and implement quality improvement initiatives both internally and externally.

We continue to encourage you to be as specific as possible when providing feedback in order to best understand your experience with PAAB and create a meaningful action-plan to improve or disseminate best practices. Thank you for your continued participation in the CEI surveys!

How to make your feedback count! *Management provided variations on comments received to ensure no identifiers Please be specific!  “All staff involved were very responsive. However, the reviewer assigned to the file was not very helpful during the review.” TIP: Add detail! If you were happy with most of the staff, but didn’t find the reviewer particularly helpful, tell us what happened. What made you feel that way? What part of the review was unhelpful?   “N/A” or similar (accompanied by a rating with an asterisk denoting a lower score) TIP: Put yourself in our shoes. If you received a low score on your performance rating, but were not told why, would you know how to improve to better your score in the future? CEI scores are part of our team’s performance evaluations and are used to guide company-wide process improvement and best-practice sharing. The more detail we have, the better we can guide these initiatives! Most of the staff involved in my eFile were very responsive. The file coordinators noticed A PM update was missing and worked with me to quickly get the proper file uploaded and into the queue. However, in the second round of review, I requested a call with my reviewer to discuss Comment 2 of their letter, and the only availability they provided was 4 days after my initial request. It’s my understanding that reviewers are to at least provide options for a return-call within 24 hours, so I was disappointed by this as it impacted my overall timeline. Why this works: It’s detailed! We now understand what the issue is that caused the responder to feel that the reviewer was not helpful and can dig into why this ticket call took so long to book and return. Both positive and constructive feedback was provided. We appreciate that not all cases will have both positive and constructive feedback, but this was particularly helpful to understand the ranking and identify areas for improvement. Where there is positive feedback, it helps us amplify these best practices since we know what is helpful to you. Constructive feedback helps us understand what the challenges were and improve on an individual level and a systems level.

 

Confidence in confidentiality As a reminder, client tags trigger internal audits for validation by PAAB’s Director of Pre-clearance Services, Yin Man. Any tags pertaining to Yin are validated by the Commissioner and removed from the report provided to Yin. No Reviewer, Senior Reviewer, or Director is EVER aware of tags generated by clients. You can be confident in the confidentiality of the tagging system. For additional reassurance, the tagging system, tag assessments, and documented actions taken will periodically be reviewed by an external auditor. What does PAAB use the tags for? Staff and system performance metrics To identify trends and training opportunities To stay inform on what is going well (best practices) and areas for improvement If you’d like to learn more about the client tagging system, check out the Client Tagging System Advisory. You’ll also find links to useful videos on  tagging a review and tagging phone calls.