Invented by Haiqi WEI, Teodor Pantchev GRANTCHAROV, Babak Taati, Yichen Zhang, Frank RUDZICZ, Kevin Lee YANG, Surgical Safety Technologies Inc
In recent years, there has been a growing emphasis on patient safety and quality improvement in healthcare. Adverse events during surgical procedures can have serious consequences for patients, leading to increased morbidity, mortality, and healthcare costs. As a result, there is a pressing need for effective systems and methods to detect and estimate the severity of adverse events using surgical data.
The market for systems and methods for adverse events detection or severity estimation using surgical data is rapidly expanding. These innovative technologies aim to leverage the vast amount of data generated during surgical procedures to identify potential adverse events and assess their severity in real-time. By doing so, they enable healthcare providers to intervene promptly and mitigate the risks associated with adverse events.
One of the key drivers of this market is the increasing adoption of electronic health records (EHRs) and other digital health technologies. EHRs contain a wealth of information about patients, including their medical history, medications, and surgical procedures. By integrating surgical data from EHRs with advanced analytics and machine learning algorithms, these systems can identify patterns and anomalies that may indicate the occurrence of adverse events.
Another factor contributing to the growth of this market is the rising demand for personalized medicine and precision surgery. As surgical procedures become increasingly complex and tailored to individual patients, the need for accurate adverse event detection and severity estimation becomes even more critical. These systems can provide surgeons with real-time feedback on the potential risks associated with specific procedures, allowing them to make informed decisions and optimize patient outcomes.
Furthermore, regulatory bodies and healthcare organizations are recognizing the importance of adverse event detection and severity estimation in improving patient safety. In many countries, there are stringent regulations and guidelines in place to ensure the reporting and analysis of adverse events. The implementation of systems and methods for adverse event detection can help healthcare providers comply with these requirements and enhance their overall quality of care.
The market for systems and methods for adverse events detection or severity estimation using surgical data is highly competitive, with numerous companies and research institutions actively developing and commercializing these technologies. Some of the key players in this market include healthcare IT companies, medical device manufacturers, and academic institutions specializing in data analytics and machine learning.
However, there are still challenges to overcome in the widespread adoption of these systems. One of the main barriers is the interoperability and integration of different data sources, such as EHRs, surgical instruments, and monitoring devices. Achieving seamless data exchange and integration is essential for the accurate detection and estimation of adverse events.
In conclusion, the market for systems and methods for adverse events detection or severity estimation using surgical data is experiencing significant growth. These technologies have the potential to revolutionize patient safety in surgical settings by enabling real-time identification and assessment of adverse events. As the demand for personalized medicine and precision surgery continues to rise, the need for accurate adverse event detection and severity estimation will only become more crucial. With ongoing advancements in data analytics and machine learning, the future of this market looks promising, promising safer and more efficient surgical procedures.
The Surgical Safety Technologies Inc invention works as follows
Embodiments” described herein can provide devices, methods, computer-readable media, or systems for detecting adverse events and estimating severity in surgical videos. The system is capable of training multiple models to detect adverse events and estimate severity. The system can load models for real-time detection of adverse events and severity estimation.
Background for System and method for adverse events detection or severity estimation using surgical data
As described in certain embodiments, a system is provided for automatically generating data structures that store classifications related to an adverse event. Even though surgeons are highly trained and have the best of intentions, science and technology are still limiting and mistakes are possible.
The specific architecture and machine learning approaches of various embodiments are proposed to help address technical, computer-based problems that arose during system implementation. To address computer-based technical problems during implementation, the specific architecture and machine learning methods of different embodiments are suggested.
The system could be provided as a server computer, which can, for instance, reside in a datacenter that is electronically coupled with a healthcare facility, or within the facility itself. The training can be conducted using electronic data sets that are from a single facility (e.g. local data) or, in some cases across multiple facilities. In some cases the machine learning data architectural model is trained locally, and then federated so that updates to model weights and connections are coordinated. “Using a federated approach, sensitive data can remain on-premises and the model architecture trained across multiple facilities.
The system in some embodiments is directed at healthcare applications that receive disparate data sets. It is not feasible to process and label all the data by humans, especially when time-sensitive analyses are needed (e.g. near-real time intervention). In some embodiments, data outputs can include annotated graphics user interfaces and/or data objects that contain annotations to indicate estimated regions of interests or points of importance associated with a video or audio recording of a medical procedure. These annotations can be used by human reviewers for navigation, so they don’t have to review the entire procedure. Alternate embodiments include data structures that are added to an electronic record of a procedure or a health record in a database or data warehouse storing data related to the procedure.
Applicants have developed a healthcare “black box” “Applicants have developed a healthcare?black box? The sensors are used to send audio and/or videos to the machine learning system. Sensors can include microphones (e.g. ceiling microphones or wearable microphones) and video cameras. Other data feeds may come from devices in the facility such as patient monitors and electronic health records. Medical devices (e.g. a connected stapler) can also provide feeds. These data sets can then be time-synchronized, gathered into a data pipeline and stored on a server for analysis.
Processing data quickly can provide insights that are useful for improving future iterations or, in certain cases, identifying problems with a procedure, and/or implementing an intervention. Automatic adverse event detection is less subjective than the conventional methods of identifying adverse outcomes and reporting them in patient records afterward. Automatic adverse event detection (estimation), for post-operative care reduces the administrative burden of manual review and costs by helping to focus the review, or automatically annotating estimations in an electronic data record.
Applicants note intraoperative adverse event as a risk in any procedure, and a factor of morbidity. It is difficult for healthcare practitioners to account for all uncertainties, especially when it comes to high-risk procedures. Even for minimally-invasive procedures or lower-risk procedures (e.g. a routine hernia surgery), adverse events can still occur. A surgical tool, for example, can accidentally touch tissue, causing unwanted burns (e.g. thermal injury) or cut injuries. Dropped tools are another example of an adverse event.
The approaches described in this document aim to reduce surgical errors by providing a machine-learning based automated mechanism that supports clinical decision-making.
The sensor network receives a large volume of raw data, including audio, video, electronic health records, equipment and lighting. This data can be subject to large variances due to variations in operating theatre configurations, such as differences in lighting, camera movement, or device positioning.
It’s not practical to use manually-crafted detection networks, as they cannot adapt to changes and individualization for each configuration. This is especially true as new features and parameter are added to the computer-based analyses. Hand-crafted detection network approaches include those that are written to fit a particular configuration. For example, static approaches or hard coded values. Hand-crafted approaches have the disadvantage of being domain specific, not being portable between implementations and having a high cost.
Accordingly, applicants have investigated deep-learning approaches for adverse event estimates, and propose an enhanced system that addresses technical issues encountered in relation to practical implementation of a machine learning/deep learning system. As described in this document, for example, the performance and efficiency of various embodiments are compared to alternate machine learning methods, with performance improvements noted when comparing machine learning data architecture models.
The feature extractor can be a convolutional neural network (e.g., a 2D or 3D convolutional neural network, such as an InceptionV1-based 3D convolutional neural net). The feature extractor is a convolutional network (e.g. a 2D, 3D, or InceptionV1 based 3D Convolutional Neural Network). The feature extractor extracts latent vectors from the audio or video data. These vectors are then provided to multiple time-based classifiers, each of which is tasked with a different classification task, e.g. thermal injury detection or bleeding detection.
The time-based classifiers include, among others, recurrent neural network, multi-stage temporal-convolutional networks and self-attention mechanisms, such as transformers. Transformers are able to learn representations using a self-attention method, and they’re more parallelizable.
In some embodiments, the classification tasks may be causally distinct, e.g. events with low or no correlation, and can be adapted to distinguish between related events, such as a first task that classifies bleeding events in contrast to pre-existing blood in a feed of data, and a subsequent task that classifies thermal injury events in contrast to pre-existing marks on the skin captured in the feed.
These distinctions can, for instance, be used to distinguish between injury events that are active and those that have occurred in the past. This is particularly useful in the operating room, where the presence of blood clots/stains, thermal scars, etc. can confuse a machine system in generating estimates. There may be a certain level of tissue injury that is acceptable for a specific procedure (e.g. heart valve replacement surgery) and there can be adverse events when tissue damage exceeds the normal levels.
The feature extractor neural networks of certain embodiments are trained on a data set of training using a sigmoid cross-entropy. Each of the time-based classifications is then trained using a different loss function. The loss function may include, for instance, the sigmoid cross-entropy used to train the feature extraction, and, in some embodiments also a T MSE loss (truncated average square error which minimizes the number transitions between detections). The loss function can also include uncertainty loss, focal loss and other aspects.
As noted in the variant embodiments of the present invention, in certain embodiments the approach can also be adapted to compensate camera motion/jitter. (e.g. a laparoscope traveling through the abdominal wall). In some embodiments, an additional stabilizer is added that receives video frames, and then uses bundled camera path stabilization in order to reduce jitter. In experiments conducted on stabilization, it was found that approaches in which camera paths are estimated and optimized for smoothness, particularly with laparoscopic video, were useful. A discontinuity-preserving term (smoothing term) can be utilized to preserve motion discontinuity, which prevents cropping for a quickly panning scene. “Specific experiments provided evaluation metrics for different technical approaches in frame division, resolution and adaptive parameters.
The Applicant has conducted experiments to identify loss function that improved technical performance. The technical performance is evaluated based on the balance between accuracy, sensitivity, segmental precision and recall.
As described in more detail in different embodiments, various combinations of the multitask loss (e.g. the sigmoid cross-entropy) are modified using focal loss and uncertainty loss. Smoothing loss is also evaluated. Modifying the function of the loss function led to technical distinctions. For example, using focal loss scales binary cross-entropy in each sample, so that the model is focused on learning more difficult examples and addressing imbalance problems that can occur during training. Both the focal loss and uncertainty are used to address issues of class imbalance, among other things. The uncertainty loss is a task-specific measure of uncertainty that guides models to learn more weights.
As noted above, the T-MSE can be included optionally and could be beneficial for some cases. Technical trade-offs between precision and recall might be justified. This loss reduces outliers by forcing the model penalizes more predictions that lead to a transition. “From positive to negative predictions.
Experiments were conducted on data sets obtained from laparoscopic gastric bypass procedures. Trained surgeons created a labelled set of data based upon frames of data received during the procedure. Applicants note that various embodiments can be used with a variety of procedures.
Click here to view the patent on Google Patents.
