SYNTHETIC-TO-REALISTIC IMAGE CONVERSION USING GENERATIVE ADVERSARIAL NETWORK (GAN) OR OTHER MACHINE LEARNING MODEL

Jonathan H. Goldstein; Lauren E. Turney; Richard W. Ely; Jody D. Verret; Brett N. Appleton

Patent

Atlas literature

Patent

US 12,586,359 B2

SYNTHETIC-TO-REALISTIC IMAGE CONVERSION USING GENERATIVE ADVERSARIAL NETWORK (GAN) OR OTHER MACHINE LEARNING MODEL

Jonathan H. Goldstein, Lauren E. Turney, Richard W. Ely, Jody D. Verret et al.

Raytheon Company, Arlington, VA (US)·Mar. 24, 2026·US

Drawings

Patent drawings and their descriptions. Click a drawing to enlarge it.

FIG. 1

FIG. 1 illustrates an example system supporting synthetic- to-realistic image conversion using a generative adversarial network (GAN) or other machine learning …

FIG. 2

FIGS. 2 through 4 illustrate example images that may be used to support image-based navigation according to this disclosure:

FIG. 3

FIG. 4

FIG. 4 illustrates an example image 400 that may be generated using a trained machine learning model 122. Here, the image 400 may be generated based on a …

FIG. 5

FIG. 5 illustrates an example architecture supporting image-based navigation using synthetic-to-realistic image conversion according to this disclosure;

FIG. 6

FIGS. 6A and 6B illustrate an example machine learning model supporting synthetic-to-realistic image conversion according to this disclosure:

FIG. 7

FIGS. 7 and 8 illustrate example images that may be associated with a trained machine learning model to support synthetic-to-realistic image conversion …

FIG. 8

FIG. 9

FIG. 9 illustrates an example method for training a machine learning model to support synthetic-to-realistic image conversion according to this disclosure; and

FIG. 10

FIG. 10 illustrates an example method for image-based navigation using synthetic-to-realistic image conversion according to this disclosure.

Claims

Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.

3 independent · 17 dependent

1
Independent
A method comprising: obtaining training data comprising ﬁrst image pairs, each of the ﬁrst image pairs comprising (i) a ﬁrst training image and (ii) a ﬁrst ground truth image; training a machine learning model to generate realistic images using the ﬁrst image pairs; obtaining additional training data comprising second image pairs, each of the second image pairs comprising (i) a second training image and (ii) a second ground truth image, wherein at least some of the images in the second image pairs are less aligned or of lower quality than at least some of the images in the ﬁrst image pairs; and B₂ continuing to train the machine learning model to gener-ate the realistic images using the second image pairs; wherein training the machine learning model and con-tinuing to train the machine learning model comprise using a loss function; and wherein the loss function is based on (i) a generative adversarial network loss and (ii) a feature matching loss.
1. 2
  Dependent← claim 1
  The method of claim 1, wherein: the machine learning model comprises a generative adver-sarial network, the generative adversarial network com-prising a generator and at least one discriminator; training the machine learning model and continuing to train the machine learning model comprise using the ﬁrst and second image pairs to train the generator, the generator conﬁgured to generate the realistic images using the ﬁrst and second training images; and the at least one discriminator is conﬁgured to attempt to differentiate between (i) the ﬁrst and second ground truth images and (ii) the realistic images generated by the generator.
2. 4
  Dependent← claim 1
  The method of claim 1, wherein the loss function comprises a sample-based adjustable hyperparameter asso-ciated with the feature matching loss, the adjustable hyper-parameter having a larger value when images in image pairs have better alignment and a smaller value when images in image pairs have poorer alignment.
3. 6
  Dependent← claim 1
  The method of claim 1, further comprising: deploying at least a portion of the trained machine learn-ing model to one or more platforms for use during inferencing.
4. 19
  Dependent← claim 1
  The method of claim 1, wherein the machine learning model is trained to convert images between domains without introducing hallucinations.
7
Independent
An apparatus comprising: at least one memory conﬁgured to store: training data comprising ﬁrst image pairs, each of the ﬁrst image pairs comprising (i) a ﬁrst training image and (ii) a ﬁrst ground truth image; and additional training data comprising second image pairs, each of the second image pairs comprising (i) a second training image and (ii) a second ground truth image, wherein at least some of the images in the second image pairs are less aligned or of lower quality than at least some of the images in the ﬁrst image pairs; and at least one processing device conﬁgured to: train a machine learning model to generate realistic images using the ﬁrst image pairs; and continue to train the machine learning model to gen-erate the realistic images using the second image pairs; wherein, to train the machine learning model and to continue to train the machine learning model, the at least one processing device is conﬁgured to use a loss function; and wherein the loss function is based on (i) a generative adversarial network loss and (ii) a feature matching loss.
1. 8
  Dependent← claim 7
  The apparatus of claim 7, wherein: the machine learning model comprises a generative adver-sarial network, the generative adversarial network com-prising a generator and at least one discriminator; to train the machine learning model and to continue to train the machine learning model, the at least one processing device is conﬁgured to use the ﬁrst and second image pairs to train the generator, the generator conﬁgured to generate the realistic images using the ﬁrst and second training images; and the at least one discriminator is conﬁgured to attempt to differentiate between (i) the ﬁrst and second ground truth images and (ii) the realistic images generated by the generator.
2. 10
  Dependent← claim 7
  The apparatus of claim 7, wherein the loss function comprises a sample-based adjustable hyperparameter asso-ciated with the feature matching loss, the adjustable hyper-parameter having a larger value when images in image pairs have better alignment and a smaller value when images in image pairs have poorer alignment.
13
Independent
A method comprising: obtaining one or more synthetic images of an environ-ment; generating one or more realistic images of the environ-ment based on the one or more synthetic images using a trained machine learning model; obtaining one or more actual images of the environment; and determining at least one characteristic of a ﬂight vehicle based on the one or more realistic images of the environment and the one or more actual images of the environment; wherein the trained machine learning model is trained by: training a machine learning model to generate realistic images using ﬁrst image pairs, each of the ﬁrst image pairs comprising (i) a ﬁrst training image and (ii) a ﬁrst ground truth image; and continuing to train the machine learning model to generate the realistic images using second image pairs, each of the second image pairs comprising (i) a second training image and (ii) a second ground truth image, wherein at least some of the images in B₂ the second image pairs are less aligned or of lower quality than at least some of the images in the ﬁrst image pairs; wherein training the machine learning model and con-tinuing to train the machine learning model comprise using a loss function; and wherein the loss function is based on (i) a generative adversarial network loss and (ii) a feature matching loss.
1. 14
  Dependent← claim 13
  The method of claim 13, wherein the trained machine learning model comprises a generator of a generative adver-sarial network, the generator of the generative adversarial network trained by: obtaining training data comprising the ﬁrst image pairs; training the generator to generate the realistic images using the ﬁrst image pairs; obtaining additional training data comprising the second image pairs; and continuing to train the generator to generate the realistic images using the second image pairs.
2. 15
  Dependent← claim 13
  The method of claim 13, wherein the loss function comprises a sample-based adjustable hyperparameter asso-ciated with the feature matching loss, the adjustable hyper-parameter having a larger value when images in image pairs have better alignment and a smaller value when images in image pairs have poorer alignment.
3. 16
  Dependent← claim 13
  The method of claim 13, wherein: the one or more realistic images of the environment include image data not contained in the one or more synthetic images; and the one or more realistic images of the environment lack at least some artifacts that are contained in the one or more synthetic images.
4. 17
  Dependent← claim 13
  The method of claim 13, wherein the one or more synthetic images are generated based on a three-dimensional (3D) model of the environment.
5. 18
  Dependent← claim 13
  The method of claim 13, wherein the at least one characteristic of the ﬂight vehicle comprises at least one of: an estimated location of the ﬂight vehicle, an estimated orientation of the ﬂight vehicle, and an estimated direction of travel of the ﬂight vehicle.
6. 20
  Dependent← claim 13
  The method of claim 13, wherein the machine learning model is trained to convert images between domains without introducing hallucinations. ∗ ∗ ∗ ∗ ∗

Cited prior art

Patents and literature cited by this patent (applicant and examiner references).

Cited patents · 22

US 9,269,145 B29,269,145 B2 2/2016 Ely et al.
US 9,275,267 B29,275,267 B2 3/2016 Verret
US 9,451,166 B19,451,166 B1 9/2016 Ribardo, Jr. et al.
US 9,767,572 B29,767,572 B2 9/2017 Ely
US 11,042,998 B211,042,998 B2 6/2021 Ely
EP 4134860 A1EP 4134860 A1 * 2/2023............. G06T 17/00examiner

Why these are connected

Related documents with shared materials, methods, properties, or citations.

patent2025cites
UNPAIRED IMAGE-TO-IMAGE TRANSLATION USING A GENERATIVE ADVERSARIAL NETWORK (GAN)
Direct citation

SYNTHETIC-TO-REALISTIC IMAGE CONVERSION USING GENERATIVE ADVERSARIAL NETWORK (GAN) OR OTHER MACHINE LEARNING MODEL | Matter42 Literature

Cited non-patent literature · 8

AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality. Weiquan Liu et al.,“AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality,” Sep. 26, 2019, Remote Sensing 2019, 11,2243, pp. 1-19.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. Ting-Chun Wang et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” Jun. 2018, Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8798-8804.
Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation. R. Barth et al., “Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation,” Apr. 23, 2020, Computers and Electronics in Agriculture 173 (2020), pp. 1-9.* Jon Gauthier, “Conditional generative adversarial nets for convo- lutional face generation,” May 2014, Class project for Stanford CS231N: convolutional neural networks for visual recognition, Winter semester 2014.5 (2014), pp. 1-7.* Alba Nely Are´valo-Verjel et al., “Estimation of the Block Adjust- ment Error in UAV Photogrammetric Flights in Flat Areas,” Jun. 16, 2022, Remote Sens. 2022, 14, 2877,pp. 1-11.
AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality. Liu et al., “AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality,” Remote Sensing, 2019, 23 pages.
An Image Denoising Method Based on Deep Residual GAN. Wang et al., “An Image Denoising Method Based on Deep Residual GAN,” IWAACE, Journal of Physics: Conference Series, 2020, 7 pages. Ackermann, “Combined Adjustment of Airborne Navigation Data and Photogrammetric Blocks,” XVIth ISPRS Congress Technical Commission III, Working Group III/1, Jul. 1988, 13 pages.
Techniques andApplications of UAV-Based Photogram- metric 3D Mapping. Jiang et al., “Techniques andApplications of UAV-Based Photogram- metric 3D Mapping,” MDPI Remote Sensing, 2022, 296 pages. Wikipedia, “Generative adversarial network,” Aug. 2023, 61 pages.
Conditional Generative Adversarial Nets. Mirza et al., “Conditional Generative Adversarial Nets,” Machine Learning (cs.LG), Nov. 2014, 7 pages.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. Wang et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” Computer Vision and Pat- tern Recognition (cs.CV), Aug. 2018, 14 pages.

Cited non-patent literature · 8

AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality. Weiquan Liu et al.,“AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality,” Sep. 26, 2019, Remote Sensing 2019, 11,2243, pp. 1-19.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. Ting-Chun Wang et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” Jun. 2018, Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8798-8804.
Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation. R. Barth et al., “Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation,” Apr. 23, 2020, Computers and Electronics in Agriculture 173 (2020), pp. 1-9.* Jon Gauthier, “Conditional generative adversarial nets for convo- lutional face generation,” May 2014, Class project for Stanford CS231N: convolutional neural networks for visual recognition, Winter semester 2014.5 (2014), pp. 1-7.* Alba Nely Are´valo-Verjel et al., “Estimation of the Block Adjust- ment Error in UAV Photogrammetric Flights in Flat Areas,” Jun. 16, 2022, Remote Sens. 2022, 14, 2877,pp. 1-11.
AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality. Liu et al., “AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality,” Remote Sensing, 2019, 23 pages.
An Image Denoising Method Based on Deep Residual GAN. Wang et al., “An Image Denoising Method Based on Deep Residual GAN,” IWAACE, Journal of Physics: Conference Series, 2020, 7 pages. Ackermann, “Combined Adjustment of Airborne Navigation Data and Photogrammetric Blocks,” XVIth ISPRS Congress Technical Commission III, Working Group III/1, Jul. 1988, 13 pages.
Techniques andApplications of UAV-Based Photogram- metric 3D Mapping. Jiang et al., “Techniques andApplications of UAV-Based Photogram- metric 3D Mapping,” MDPI Remote Sensing, 2022, 296 pages. Wikipedia, “Generative adversarial network,” Aug. 2023, 61 pages.
Conditional Generative Adversarial Nets. Mirza et al., “Conditional Generative Adversarial Nets,” Machine Learning (cs.LG), Nov. 2014, 7 pages.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. Wang et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” Computer Vision and Pat- tern Recognition (cs.CV), Aug. 2018, 14 pages.

Cited non-patent literature · 8

AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality. Weiquan Liu et al.,“AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality,” Sep. 26, 2019, Remote Sensing 2019, 11,2243, pp. 1-19.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. Ting-Chun Wang et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” Jun. 2018, Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8798-8804.
Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation. R. Barth et al., “Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation,” Apr. 23, 2020, Computers and Electronics in Agriculture 173 (2020), pp. 1-9.* Jon Gauthier, “Conditional generative adversarial nets for convo- lutional face generation,” May 2014, Class project for Stanford CS231N: convolutional neural networks for visual recognition, Winter semester 2014.5 (2014), pp. 1-7.* Alba Nely Are´valo-Verjel et al., “Estimation of the Block Adjust- ment Error in UAV Photogrammetric Flights in Flat Areas,” Jun. 16, 2022, Remote Sens. 2022, 14, 2877,pp. 1-11.
AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality. Liu et al., “AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality,” Remote Sensing, 2019, 23 pages.
An Image Denoising Method Based on Deep Residual GAN. Wang et al., “An Image Denoising Method Based on Deep Residual GAN,” IWAACE, Journal of Physics: Conference Series, 2020, 7 pages. Ackermann, “Combined Adjustment of Airborne Navigation Data and Photogrammetric Blocks,” XVIth ISPRS Congress Technical Commission III, Working Group III/1, Jul. 1988, 13 pages.
Techniques andApplications of UAV-Based Photogram- metric 3D Mapping. Jiang et al., “Techniques andApplications of UAV-Based Photogram- metric 3D Mapping,” MDPI Remote Sensing, 2022, 296 pages. Wikipedia, “Generative adversarial network,” Aug. 2023, 61 pages.
Conditional Generative Adversarial Nets. Mirza et al., “Conditional Generative Adversarial Nets,” Machine Learning (cs.LG), Nov. 2014, 7 pages.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. Wang et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” Computer Vision and Pat- tern Recognition (cs.CV), Aug. 2018, 14 pages.

Cited non-patent literature · 8

AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality. Weiquan Liu et al.,“AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality,” Sep. 26, 2019, Remote Sensing 2019, 11,2243, pp. 1-19.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. Ting-Chun Wang et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” Jun. 2018, Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8798-8804.
Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation. R. Barth et al., “Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation,” Apr. 23, 2020, Computers and Electronics in Agriculture 173 (2020), pp. 1-9.* Jon Gauthier, “Conditional generative adversarial nets for convo- lutional face generation,” May 2014, Class project for Stanford CS231N: convolutional neural networks for visual recognition, Winter semester 2014.5 (2014), pp. 1-7.* Alba Nely Are´valo-Verjel et al., “Estimation of the Block Adjust- ment Error in UAV Photogrammetric Flights in Flat Areas,” Jun. 16, 2022, Remote Sens. 2022, 14, 2877,pp. 1-11.
AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality. Liu et al., “AE-GAN-Net: Learning Invariant Feature Descriptor to Match Ground Camera Images and a Large-Scale 3D Image-Based Point Cloud for Outdoor Augmented Reality,” Remote Sensing, 2019, 23 pages.
An Image Denoising Method Based on Deep Residual GAN. Wang et al., “An Image Denoising Method Based on Deep Residual GAN,” IWAACE, Journal of Physics: Conference Series, 2020, 7 pages. Ackermann, “Combined Adjustment of Airborne Navigation Data and Photogrammetric Blocks,” XVIth ISPRS Congress Technical Commission III, Working Group III/1, Jul. 1988, 13 pages.
Techniques andApplications of UAV-Based Photogram- metric 3D Mapping. Jiang et al., “Techniques andApplications of UAV-Based Photogram- metric 3D Mapping,” MDPI Remote Sensing, 2022, 296 pages. Wikipedia, “Generative adversarial network,” Aug. 2023, 61 pages.
Conditional Generative Adversarial Nets. Mirza et al., “Conditional Generative Adversarial Nets,” Machine Learning (cs.LG), Nov. 2014, 7 pages.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. Wang et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” Computer Vision and Pat- tern Recognition (cs.CV), Aug. 2018, 14 pages.