RTI SmartReview: Using Data Science to Improve Federal Public Comment Review

Rulemaking is used by federal agencies to propose and finalize policies across a wide range of topics. It is also used to communicate with the public, notifying them of proposed policies and allowing them to provide input before policies are finalized. 

Federal rulemaking involves a public comment period, in which individuals, organizations, and stakeholders are invited to submit their feedback on proposed regulations, typically within 60 days. This process is invaluable, but it can create a significant burden on agencies that propose policies through large payment rules that contribute to many federal programs. Agencies may need to review thousands of comments, many of which may not be directly relevant to the proposed policy. 

Timely identification of relevant, substantive comments is critical for agencies to develop informed policies while meeting tight timelines to respond to comments, finalize policies, and publish the final rule. Without advanced data science tools, identifying the relevant comments may take several weeks, as manually reviewing these comments can be labor-intensive and time-consuming.

To address this challenge, RTI International developed RTI SmartReview, a system designed to assist federal agencies in managing public comment review. The RTI SmartReview system—which leverages data science techniques—enables agencies to identify and process relevant public comments rapidly, significantly speeding up the process. Using an automated data pipeline, RTI SmartReview ingests public comments from Regulations.gov and uses natural language processing and machine learning algorithms to identify and prioritize the comments that are most likely to be relevant to the proposed policies.

The platform uses three key processes that make this possible:

1. Keyword Tagging. RTI SmartReview uses a prespecified list of keywords—chosen by subject matter experts for each rule—to flag comments that are likely relevant. The keyword list can be updated dynamically as new information becomes available during the comment period, allowing agencies to fine-tune their searches throughout the submission period.
    
2. Clustering. RTI SmartReview uses a clustering algorithm to group similar comments, making it easier for reviewers to identify form letters or comments with overlapping and/or similar content. This allows for the quick processing of large volumes of similar feedback.
    
3. Relevance Modeling. RTI SmartReview's machine learning model is trained to identify comments that are most relevant based on examples of relevant and non-relevant comments determined on a rolling basis during the comment review period. The model becomes more accurate over time as more comments are processed, ultimately flagging potentially relevant comments as they are received.

After the automated data pipeline has run, it automatically generates a spreadsheet on Sharepoint. Comment reviewers can leverage the spreadsheet to prioritize comments for review and determine their relevance to the policies of interest. Additionally, the machine learning model pulls data from this spreadsheet each night to train the relevance model and updates the spreadsheet automatically (without overriding reviewers’ comments). 

This multipronged approach allows agencies to identify and focus efforts on the most pertinent comments to a policy as the feedback is submitted. For instance, in the 2023 cycle for the Centers for Medicare & Medicaid Services’ (CMS’) CY 2024 Physician Fee Schedule, RTI SmartReview identified 3.5% of the 22,000 comments as likely relevant, accounting for 100% of the 203 relevant submissions. This efficiency provided reviewers more time to focus on the most critical comments that were flagged for prioritized review. 

Since its inception, RTI SmartReview has been used to support comment review for several policies proposed within the Physician Fee Schedule across five rulemaking cycles while also being adapted to support comment review for policies proposed through other rules covering diverse policy areas.

RTI SmartReview aligns with the federal government’s ongoing data modernization efforts and has the potential to change the way federal agencies approach public comment review. The system is designed to be flexible, allowing for adjustments in its implementation, including processing comments from a directory of files in addition to Regulations.gov, tailoring keyword lists, and modifying the relevance and clustering model parameters when needed to suit the needs of each rule. By automating routine tasks to support a prioritized review process, agencies can expedite comment identification and increase availability to develop responses and finalize policies. 

Watch our on-demand webinar for more information on managing large volumes of public comments for prioritized review using RTI SmartReview. Fill out the form below to learn how you can leverage RTI SmartReview to meet your public comment review needs.