DRL Medical Imaging Literature Review
- 1. Deep Reinforcement Learning in Medical Imaging: A Literature Review JOCELYN BADURIA S. Kevin Zhoua, Hoang Ngan Leb, Khoa Luub , Hien V. Nguyenc , Nicholas Ayached
- 2. Abstract Introduction Basics of Reinforcement Learning Introduction to Deep Reinforcement Learning DRL in Medical Imaging Conclusion Table of Contents
- 3. Abstract What is Deep Reinforcement Learning(DRL)? - It is a subset of Machine Learning called Deep Learning with added top layered Implementation process called Reinforcement learning. What is Reinforcement Learning(RL)? - It is a learning process that uses a game theory analogy, that it learns through a series of actions through level up reward system. Most recent applications of DRL have shown tremendous impact on healthcare and medicine field. DRL is very useful in learning medical images analysis for “object/lesion detection, surgical image segmentation/augmentation, registration of significant medical images, view plane image localization, personal mobile health intervention”.
- 4. Introduction DRL is a (SOTA) State-of-the-Art framework in Artificial Intelligence. - It scales up the learning sequential decision process and can highly learn with delayed supervised information. - It can also deal with maximizing the “non-differentiable” metrics of parametric setting of classification accuracy problem. - It also speed up the searching detection process for optimal transformation of image registration accuracy. - It improves the memory allocation of high-dimensional images. Small images can be used in DRL object detection process. This survey paper expanded its study on many (SOTA) state-of-the-art contents.
- 5. Basics of Reinforcement Learning Markov Decision Process (MDP) - Is the essential foundation theory of Reinforcement Learning, a process “whose the conditional probability distribution of future states only depends upon the present state”. - The introduction of reward system not only depends on the current state but also, on the action that leads to future state. - One good example of this is the famous online game “Fortnite”. Usually, the game player chooses an action while playing according to the set of rules and the environment receives the action and produces a reward. Then the player transfers to the next level until he/she reached the final level of the game which is the future state.
- 6. Basics of Reinforcement Learning MDP is defined into five elements: 1. set of state/observation space 2. set of actions the agents can choose from 3. transition probability function 4. reward function 5. discount factor The main objective of (RL) Reinforcement Learning is to find the prime rules or best- known solution that gives the highest “expected cumulative reward called return - the weighted sum of immediate rewards”.
- 7. Basics of Reinforcement Learning Reinforcement Learning is divided into two functions 1. Model-Free Methods - This method is the learning process of a gamer during the interaction process with other players and the environment to exceed the maximum policy. 2. Model-Based Methods - This method is when the experienced gamer knows such a certain rules or game policy and use it for its own advantage to exceed the maximum policy.
- 8. Autoencoder - Is a representation unsupervised learning used for “dimensionality reduction and feature selection”. Deep Belief Network and Deep Autoencoders - Are unsupervised learning used for network initialization. - Deep autoencoders have one seen input layer and one invisible layer. - Deep belief network is based on “Restricted Boltzmann Machines” which contains of layered input data and a layered invisible units that can gain a knowledge to describe the features that are gathered from the “higher-order of correlations in the data”. Introduction to Deep Reinforcement Learning
- 9. Multi-layer Perceptron (MLP) - Neuron is the simplest form of neural network. - Perceptron is a computational single form of neuron that has one or many inputs, main-processor, and one output. - There are two types of Neural Networks. 1. Convolutional Neural Networks (CNN) 2. Recurrent Neural Networks (RNN) Convolutional Neural Networks (CNN) - A fully joined multi-layer perceptrons that perform weight sharing during the data processing in a grid like topology network. Introduction to Deep Reinforcement Learning
- 10. Recurrent Neural Network (RNN) - A sequential model that performs the same task for each sequence on which the output depends on the previous task computations. - Long Short-Term Memory (LSTM) was born to address the issue of the difficulty of (RNN) training for long-term dependencies. It was abled to maintain the memory that updates and release its content when it is needed during training process. - Gated Recurrent Unit (GRU) was proposed recently to improve the process of capturing dependencies at different time intervals. The difference with LSTM it has no separate memory cells. Introduction to Deep Reinforcement Learning
- 11. Model-free DRL Algorithms - There are three approaches for this algorithms namely 1. Value-Based DRL Methods 2. Policy Gradient DRL Methods 3. Actor-Critic DRL Methods Value-Based DRL Methods 1. Deep Q-Learning Network (DQN) - Is the popular one among the three. This method directly knows the policy knowledge from the higher dimension inputs in a deep neural network. - This uses a regression modelling but with main limitation that Q* values tends to over-estimate because of max values in MSE loss computation. Introduction to Deep Reinforcement Learning
- 12. Value-Based DRL Methods 2. Double DQN is the improved DQN version that uses double implementation of DQN. It addresses the over-estimation issue with its improved estimation but very costly process although the easiest one method. 3. Dueling DQN addresses the issue of DQN on its value V*, instead of lowering the value it remembers only the low reward by updating the Q* value. It introduces the advantage function. Introduction to Deep Reinforcement Learning
- 20. DRL in Medical Imaging DRL is mostly used in medical imaging analysis. - Application varies from these field of analysis : 1. Landmark Detection 2. Image Registration, Segmentation 3. Object Lesion localization and classification 4. View Plane localization 5. Plaque Tracking 6. Vessel Extraction - It is also used in hyperparameter tuning, data augmentation and neural search architecture. - Mostly shared non-differentiable optimization for solution.
- 21. DRL in Medical Imaging DRL for parametric Medical Image Analysis - This model estimated a low dimensional image for analysis. - There are three elements that is required to formulate a DRL framework. 1. Action - The agent takes the image by moving the ith parameter independently in certain value while keeping the other parameters in same position. 2. State - The state defined in environment is image at certain points – centered position. 3. Reward - When the target signal is hit or closer the reward function provides a stimuli signal to the agent.
- 22. DRL in Medical Imaging DRL for parametric Medical Image Analysis - Once the three elements are available the DQL algorithm was invoked and triggered to learn the process. - Then the process learned, the Q-function loss is calculated using the maximum value. - Using the greedy search fashion for path supervision it maximizes the reward for every iteration process. - This eventually leads to convert the Reinforcement Learning (RL) into Supervised Learning model. From there we can start learning a model using classification or regression function.
- 23. Landmark Detection - This landmark anatomical structures plays an important roles in navigating the image sample. It is just like a travelling geographical landmarks in map. - It helps the tourist to explore the places they wanted to visit. 1. Artificial Agent – presents the multi-scale approach for detection of anatomical landmarks in 3Dimensional space. It exhibits perfect detection with no false positive and negative values. DRL in Medical Imaging
- 24. DRL in Medical Imaging Landmark Detection 2. Supervised action classifier – image partitioning of landmark detection. This is taken from the path supervision approach. - This modeling allow four possible action types that helps to achieve the best result accuracy of image detection from a “cardiac arrest or obstetric ultrasound image in two datasets with 1323 and 1642 patients with compared to SOTA - Artificial Agent. Pixel-wise optimal action step with slopes +/-1 crossing the landmark
- 25. DRL in Medical Imaging Image Registration - For comparison in different times and modalities, robust image registration in medical imaging is needed. There are two ways used for this modelling. 1. Rigid Registration – It uses a Deep Convolutional Neural Network (DCNN) that utilizes the path supervision method to end-to-end training. - This modelling is evaluated into two datasets (spine – 87 pairs of images, heart - 97 pairs of images). This artificial agent outperforms SOTA(State of the Art) methods with big margin in terms of accuracy and robustness.
- 26. DRL in Medical Imaging Image Registration 1. Rigid Registration - Updated model using the dueling DQN by learning the Q function instead of path supervision was introduced to previous method achieves the SOTA when compared with another model. 2. Non-rigid Registration - A non-rigid registration comes into full picture when the rigid registration performs an insufficient transformation between two images.
- 27. DRL in Medical Imaging Image Registration 2. Non-rigid Registration - Studying the parametric Statistical deformed pair images for an “organ-centered” MR registered prostate images resulted to better performance when compared to other SOTA (State of the Art) models. The median DICE coefficients is 0.96 for 2D and 0.88 in 3D. The Q function thus learned from this study. - This was also concluded when tested with image registration prostate MR data (41 3D images, 8 testing samples resulting to 56 inter-subject image pairs). Both results are better compared with other state of the art registration model.
- 28. Object/lesion Localization and Detection - Studies were also concluded for this model for detecting breast lesions taken from (DCE-MRI) Dynamic Contrast-Enhanced Magnetic Resonance Imaging machine. - Bounding box was defined with its actions together with signal reward function - Deep Q-Learning Network (DQN) is used based on ResNet Architecture. - Samples were taken from 117 patients with training set 58 patients (annotated 72 lesions ), testing set from 59 patients (69 annotated lesions). - Accuracy shows similar results with state-of-the-art model but with biggest impact on speed with time reduced detection. DRL in Medical Imaging
- 29. DRL in Medical Imaging Object/lesion Localization and Detection - Other studies also took the challenges of big computational histopathological images of breast cancer classification. - Pathologist usually selects an abnormal area of the breast and study its details. - Such mechanisms captures the attention to improve the selection and classification of abnormal breast region. - DRL was used for two task, one for selection and the other one for classification. Selection is training the certain area of image. The image selected is used for classification. - This model achieves its 98% accuracy while consuming only 50% of the old Pathological method, the attention-based approach.
- 30. DRL in Medical Imaging View Plane Localization - New method proposed for detection of canonical view in cardiac and MR brain images. - Studies used the DQN, DDQN, Duel DQN and Duel DDQN. - A 3D plane is parameterized in 4D vector, with signed reward function and action steps coarse-to-fine selection. - Similarly other method added the image augmentation starting warm for better initialization with active module termination. - This added approach improves the efficiency and accuracy of plane localization based on in-house datasets of 430 prenatal 3D ultrasound volume of fetal head.
- 31. DRL in Medical Imaging Plaque Tracking - Plaque Atherosclerosis monitoring is being explored by DRL. - Plaque is composed of cholesterol, fat, calcium, and other substances found in the blood. This buildup can lead to serious health problems or even death. - Still this proposal is having some challenges due to various intravascular morphology. Fig A. Plaque
- 32. DRL in Medical Imaging Vessel Centerline Extraction - DRL using navigation model tracing the vessel centerline. A point to curve measure is defined in two terms - Pulls agent position towards the true center label - The agent is force towards the direction of curve. - This method achieves good performance compared with 3D CNN supervised learning. - The method was updated using DDQN with 3D CNN to improve the centerline accuracy with speed of 7seconds of inference.
- 33. DRL in Medical Imaging Solving Optimization using DRL - DRL is known to handle “non-differentiable metrics” therefore it is widely used for optimizations. 1. Tuning Hyperparameter 2. Image Augmentation 3. Image Classification 4. Searching Neural Network for Augmentation
- 34. DRL in Medical Imaging Image Classification - CNN model integration with question and answering RL-Based method. - DNN agent asked the patient about the skin disease symptom using CNN visual information and answers. - DNN learned from this method based on question and answering the patient. - This approach improves the classification accuracy greater than 20% compared only to CNN using only visualization. - It shortened the diagnosis in average time compared to decision tree-based QA approach.
- 35. DRL in Medical Imaging Image Segmentation - Assigning label to pixels helps in finding the perfect boundaries of anatomical structure of medical image. - This method realized the object image segmentation and mostly used for pre- processing. However, this method is intractable in size and not yet fully utilized to meet the clinical requirements for 3D image segmentation. - State of the art proposed NAS (Neural Architecture Search) improves the image search automation for special application however it is not used often in medical image segmentation.
- 36. DRL in Medical Imaging Image Acquisition and Reconstruction - CT Metal artifacts affects the clinical decision making because of image inconsistency. - Using the iterative CT reconstruction helps to solve the optimization issue in medical image.
- 37. DRL in Medical Imaging Radiotheraphy Planning - DRL weight tuning policy was leverage in Radiotheraphy planning that takes the volume histogram as input and outputs the adjusted weight with reward function sparing the organs at risk. - This method proposed to improve the radiotherapy planning with yields of quality score 10.7% higher than human radiotherapy planner.
- 38. DRL in Medical Imaging Miscellaneous Topics - These topics are for exploration, not related to medical images analysis but mostly uses based Reinforcement Learning. 1. Video Summarization 2. Surgical gesture segmentation and classification 3. Personalized mobile health intervention 4. Computational model personalization
- 39. Conclusion DRL is powerful model in medical imaging analysis. Successfully applied to various applications in landmark localization, object detection, image registration and image-based inferencing. Demonstrated effectiveness in tuning parameter optimization, image augmentation selection, and Neural Architecture Search (NAS). Although some methods are not fully utilized in clinical requirement the current DRL model surpassed my expectations and still needs further exploration in medical imaging classification.
- 40. Thank You!!!