Patent
Atlas literature
Patent
US 11,638,032 B2Patent drawings and their descriptions. Click a drawing to enlarge it.
FIG. 1A shows a simplified system block and dataflow diagram of a GAN architecture for VSR under an unsuper- vised training according to an embodiment of the …
FIG. 2 shows a detailed system block and dataflow diagram of the GAN architecture.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
An apparatus for video super-resolution (VSR) using generative adversarial networks (GAN), comprising: a set of low-resolution (LR) frame generation networks (LFGNet) configured to synthesize a LR video frame of an intermediate LR video sequence from an input high-resolution (HR) video frame of a training HR video sequence during a training of the apparatus; a set of high-resolution (HR) frame estimation networks (HFENet) configured to generate an output HR video frame of a reconstructed HR video sequence from a currently-processing LR video frame and a last-pro-cessed LR video frame of an input LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, wherein the HFENet is trained during the training of the apparatus using the intermediate LR video sequence as the input LR video sequence, and the currently-processing LR video frame and the last-processed LR video frame are video frames of the intermediate LR video sequence; and a set of kernel estimation networks (KENet) configured, during the training of the apparatus, to: extract degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-pro-cessing LR video frame; contract the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degra-dation features; feed the contracted degradation features back to the LFGNet, adding to the training HR video sequence; and judge whether degradation of the intermediate LR video sequence is same as degradation of the testing LR video sequence; wherein the HFENet comprises: a coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain a coarse HR flow; a fine flow estimator configured to: warp a last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate a fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and 50 a HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The apparatus of claim 1, wherein the synthesis of the output LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet during training comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the output LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The apparatus of claim 1, wherein the KENet is further configured to execute a metric learning method in contract-ing the extracted degradation features; wherein the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features.
A method for training an apparatus for video superresolution (VSR) using generative adversarial networks (GAN), comprising: wherein the apparatus comprises: a set of low-resolution (LR) frame generation networks (LFGNet); a set of high-resolution (HR) frame estimation net-works (HFENet); and a set of kernel estimation networks (KENet); wherein the method comprises: synthesizing, by the LFGNet, a currently-processing LR video frame of an intermediate LR video sequence from an input HR video frame of a training HR video sequence; generating, by the HFENet, an output HR video frame of a reconstructed HR video sequence from the currently-processing LR video frame and a last-processed LR video frame of the intermediate LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, extracting, by the KENet, degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-processing LR video frame; contracting, by the KENet, the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degradation features; feeding the contracted degradation features back to the LFGNet, adding to training HR video sequence; and judging, by the KENet, whether degradation of the intermediate LR video sequence is same as degra-dation of the testing LR video sequence; wherein the HFENet comprises: a coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain a coarse HR flow; a fine flow estimator configured to: warp a last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate a fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and a HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The method of claim 4, wherein the synthesis of currently-processing LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the cur-rently-processing LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The method of claim 4, wherein the contracting of the extracted degradation features comprises executing a metric learning method, the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features.
An apparatus for video super-resolution (VSR) using generative adversarial networks (GAN), comprising: a set of low-resolution (LR) frame generation networks (LFGNet) configured to synthesize a LR video frame of an intermediate LR video sequence from an input high-resolution (HR) video frame of a training HR video sequence during a training of the apparatus; a set of high-resolution (HR) frame estimation networks (HFENet) configured to generate an output HR video frame of a reconstructed HR video sequence from a currently-processing LR video frame and a last-pro-cessed LR video frame of an input LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, wherein the HFENet comprises: a coarse flow estima-tor, a fine flow estimator and a HR frame synthesizer, wherein the coarse flow estimator generates a coarse HR flow according to the currently-processing LR video frame and a last-processed LR video frame, the fine flow estimator generates a fine HR flow accord-ing to the last-generated output HR video frame and the currently-processing LR video frame, and the HR frame synthesizer generates the output HR video frame according to the fine HR flow, the coarse HR flow and the last-generated output HR video frame, wherein the HFENet is trained during the training of the apparatus using the intermediate LR video sequence as the input LR video sequence, and the currently-processing LR video frame and the last-processed LR video frame are video frames of the intermediate LR video sequence; and a set of kernel estimation networks (KENet) configured, during the training of the apparatus, to: extract degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-pro-cessing LR video frame; contract the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degra-dation features; feed the contracted degradation features back to the LFGNet, adding to the training HR video sequence; and judge whether degradation of the intermediate LR video sequence is same as degradation of the testing LR video sequence.
The apparatus of claim 7, wherein the synthesis of the output LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet during training comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the output LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The apparatus of claim 7, wherein the HFENet com-prises: the coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain the coarse HR flow; the fine flow estimator configured to: warp the last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate the fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and the HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The apparatus of claim 7, wherein the KENet is further configured to execute a metric learning method in contracting the extracted degradation features; wherein the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features. ∗ ∗ ∗ ∗ ∗
Layer stacks claimed or described, ordered top of device to substrate.
VistGAN video super-resolution apparatus
No layer stack recorded.
Patents and literature cited by this patent (applicant and examiner references).
Cited patents · 2
Patent
Atlas literature
Patent
US 11,638,032 B2Patent drawings and their descriptions. Click a drawing to enlarge it.
FIG. 1A shows a simplified system block and dataflow diagram of a GAN architecture for VSR under an unsuper- vised training according to an embodiment of the …
FIG. 2 shows a detailed system block and dataflow diagram of the GAN architecture.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
An apparatus for video super-resolution (VSR) using generative adversarial networks (GAN), comprising: a set of low-resolution (LR) frame generation networks (LFGNet) configured to synthesize a LR video frame of an intermediate LR video sequence from an input high-resolution (HR) video frame of a training HR video sequence during a training of the apparatus; a set of high-resolution (HR) frame estimation networks (HFENet) configured to generate an output HR video frame of a reconstructed HR video sequence from a currently-processing LR video frame and a last-pro-cessed LR video frame of an input LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, wherein the HFENet is trained during the training of the apparatus using the intermediate LR video sequence as the input LR video sequence, and the currently-processing LR video frame and the last-processed LR video frame are video frames of the intermediate LR video sequence; and a set of kernel estimation networks (KENet) configured, during the training of the apparatus, to: extract degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-pro-cessing LR video frame; contract the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degra-dation features; feed the contracted degradation features back to the LFGNet, adding to the training HR video sequence; and judge whether degradation of the intermediate LR video sequence is same as degradation of the testing LR video sequence; wherein the HFENet comprises: a coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain a coarse HR flow; a fine flow estimator configured to: warp a last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate a fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and 50 a HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The apparatus of claim 1, wherein the synthesis of the output LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet during training comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the output LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The apparatus of claim 1, wherein the KENet is further configured to execute a metric learning method in contract-ing the extracted degradation features; wherein the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features.
A method for training an apparatus for video superresolution (VSR) using generative adversarial networks (GAN), comprising: wherein the apparatus comprises: a set of low-resolution (LR) frame generation networks (LFGNet); a set of high-resolution (HR) frame estimation net-works (HFENet); and a set of kernel estimation networks (KENet); wherein the method comprises: synthesizing, by the LFGNet, a currently-processing LR video frame of an intermediate LR video sequence from an input HR video frame of a training HR video sequence; generating, by the HFENet, an output HR video frame of a reconstructed HR video sequence from the currently-processing LR video frame and a last-processed LR video frame of the intermediate LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, extracting, by the KENet, degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-processing LR video frame; contracting, by the KENet, the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degradation features; feeding the contracted degradation features back to the LFGNet, adding to training HR video sequence; and judging, by the KENet, whether degradation of the intermediate LR video sequence is same as degra-dation of the testing LR video sequence; wherein the HFENet comprises: a coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain a coarse HR flow; a fine flow estimator configured to: warp a last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate a fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and a HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The method of claim 4, wherein the synthesis of currently-processing LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the cur-rently-processing LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The method of claim 4, wherein the contracting of the extracted degradation features comprises executing a metric learning method, the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features.
An apparatus for video super-resolution (VSR) using generative adversarial networks (GAN), comprising: a set of low-resolution (LR) frame generation networks (LFGNet) configured to synthesize a LR video frame of an intermediate LR video sequence from an input high-resolution (HR) video frame of a training HR video sequence during a training of the apparatus; a set of high-resolution (HR) frame estimation networks (HFENet) configured to generate an output HR video frame of a reconstructed HR video sequence from a currently-processing LR video frame and a last-pro-cessed LR video frame of an input LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, wherein the HFENet comprises: a coarse flow estima-tor, a fine flow estimator and a HR frame synthesizer, wherein the coarse flow estimator generates a coarse HR flow according to the currently-processing LR video frame and a last-processed LR video frame, the fine flow estimator generates a fine HR flow accord-ing to the last-generated output HR video frame and the currently-processing LR video frame, and the HR frame synthesizer generates the output HR video frame according to the fine HR flow, the coarse HR flow and the last-generated output HR video frame, wherein the HFENet is trained during the training of the apparatus using the intermediate LR video sequence as the input LR video sequence, and the currently-processing LR video frame and the last-processed LR video frame are video frames of the intermediate LR video sequence; and a set of kernel estimation networks (KENet) configured, during the training of the apparatus, to: extract degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-pro-cessing LR video frame; contract the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degra-dation features; feed the contracted degradation features back to the LFGNet, adding to the training HR video sequence; and judge whether degradation of the intermediate LR video sequence is same as degradation of the testing LR video sequence.
The apparatus of claim 7, wherein the synthesis of the output LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet during training comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the output LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The apparatus of claim 7, wherein the HFENet com-prises: the coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain the coarse HR flow; the fine flow estimator configured to: warp the last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate the fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and the HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The apparatus of claim 7, wherein the KENet is further configured to execute a metric learning method in contracting the extracted degradation features; wherein the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features. ∗ ∗ ∗ ∗ ∗
Layer stacks claimed or described, ordered top of device to substrate.
VistGAN video super-resolution apparatus
No layer stack recorded.
Patents and literature cited by this patent (applicant and examiner references).
Cited patents · 2
Patent
Atlas literature
Patent
US 11,638,032 B2Patent drawings and their descriptions. Click a drawing to enlarge it.
FIG. 1A shows a simplified system block and dataflow diagram of a GAN architecture for VSR under an unsuper- vised training according to an embodiment of the …
FIG. 2 shows a detailed system block and dataflow diagram of the GAN architecture.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
An apparatus for video super-resolution (VSR) using generative adversarial networks (GAN), comprising: a set of low-resolution (LR) frame generation networks (LFGNet) configured to synthesize a LR video frame of an intermediate LR video sequence from an input high-resolution (HR) video frame of a training HR video sequence during a training of the apparatus; a set of high-resolution (HR) frame estimation networks (HFENet) configured to generate an output HR video frame of a reconstructed HR video sequence from a currently-processing LR video frame and a last-pro-cessed LR video frame of an input LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, wherein the HFENet is trained during the training of the apparatus using the intermediate LR video sequence as the input LR video sequence, and the currently-processing LR video frame and the last-processed LR video frame are video frames of the intermediate LR video sequence; and a set of kernel estimation networks (KENet) configured, during the training of the apparatus, to: extract degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-pro-cessing LR video frame; contract the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degra-dation features; feed the contracted degradation features back to the LFGNet, adding to the training HR video sequence; and judge whether degradation of the intermediate LR video sequence is same as degradation of the testing LR video sequence; wherein the HFENet comprises: a coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain a coarse HR flow; a fine flow estimator configured to: warp a last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate a fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and 50 a HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The apparatus of claim 1, wherein the synthesis of the output LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet during training comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the output LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The apparatus of claim 1, wherein the KENet is further configured to execute a metric learning method in contract-ing the extracted degradation features; wherein the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features.
A method for training an apparatus for video superresolution (VSR) using generative adversarial networks (GAN), comprising: wherein the apparatus comprises: a set of low-resolution (LR) frame generation networks (LFGNet); a set of high-resolution (HR) frame estimation net-works (HFENet); and a set of kernel estimation networks (KENet); wherein the method comprises: synthesizing, by the LFGNet, a currently-processing LR video frame of an intermediate LR video sequence from an input HR video frame of a training HR video sequence; generating, by the HFENet, an output HR video frame of a reconstructed HR video sequence from the currently-processing LR video frame and a last-processed LR video frame of the intermediate LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, extracting, by the KENet, degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-processing LR video frame; contracting, by the KENet, the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degradation features; feeding the contracted degradation features back to the LFGNet, adding to training HR video sequence; and judging, by the KENet, whether degradation of the intermediate LR video sequence is same as degra-dation of the testing LR video sequence; wherein the HFENet comprises: a coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain a coarse HR flow; a fine flow estimator configured to: warp a last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate a fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and a HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The method of claim 4, wherein the synthesis of currently-processing LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the cur-rently-processing LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The method of claim 4, wherein the contracting of the extracted degradation features comprises executing a metric learning method, the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features.
An apparatus for video super-resolution (VSR) using generative adversarial networks (GAN), comprising: a set of low-resolution (LR) frame generation networks (LFGNet) configured to synthesize a LR video frame of an intermediate LR video sequence from an input high-resolution (HR) video frame of a training HR video sequence during a training of the apparatus; a set of high-resolution (HR) frame estimation networks (HFENet) configured to generate an output HR video frame of a reconstructed HR video sequence from a currently-processing LR video frame and a last-pro-cessed LR video frame of an input LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, wherein the HFENet comprises: a coarse flow estima-tor, a fine flow estimator and a HR frame synthesizer, wherein the coarse flow estimator generates a coarse HR flow according to the currently-processing LR video frame and a last-processed LR video frame, the fine flow estimator generates a fine HR flow accord-ing to the last-generated output HR video frame and the currently-processing LR video frame, and the HR frame synthesizer generates the output HR video frame according to the fine HR flow, the coarse HR flow and the last-generated output HR video frame, wherein the HFENet is trained during the training of the apparatus using the intermediate LR video sequence as the input LR video sequence, and the currently-processing LR video frame and the last-processed LR video frame are video frames of the intermediate LR video sequence; and a set of kernel estimation networks (KENet) configured, during the training of the apparatus, to: extract degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-pro-cessing LR video frame; contract the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degra-dation features; feed the contracted degradation features back to the LFGNet, adding to the training HR video sequence; and judge whether degradation of the intermediate LR video sequence is same as degradation of the testing LR video sequence.
The apparatus of claim 7, wherein the synthesis of the output LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet during training comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the output LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The apparatus of claim 7, wherein the HFENet com-prises: the coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain the coarse HR flow; the fine flow estimator configured to: warp the last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate the fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and the HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The apparatus of claim 7, wherein the KENet is further configured to execute a metric learning method in contracting the extracted degradation features; wherein the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features. ∗ ∗ ∗ ∗ ∗
Layer stacks claimed or described, ordered top of device to substrate.
VistGAN video super-resolution apparatus
No layer stack recorded.
Patents and literature cited by this patent (applicant and examiner references).
Cited patents · 2
Patent
Atlas literature
Patent
US 11,638,032 B2Patent drawings and their descriptions. Click a drawing to enlarge it.
FIG. 1A shows a simplified system block and dataflow diagram of a GAN architecture for VSR under an unsuper- vised training according to an embodiment of the …
FIG. 2 shows a detailed system block and dataflow diagram of the GAN architecture.
Claims define the patent's legal scope. Independent claims stand alone; dependent claims (nested) narrow them. Click a claim to expand its dependents.
An apparatus for video super-resolution (VSR) using generative adversarial networks (GAN), comprising: a set of low-resolution (LR) frame generation networks (LFGNet) configured to synthesize a LR video frame of an intermediate LR video sequence from an input high-resolution (HR) video frame of a training HR video sequence during a training of the apparatus; a set of high-resolution (HR) frame estimation networks (HFENet) configured to generate an output HR video frame of a reconstructed HR video sequence from a currently-processing LR video frame and a last-pro-cessed LR video frame of an input LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, wherein the HFENet is trained during the training of the apparatus using the intermediate LR video sequence as the input LR video sequence, and the currently-processing LR video frame and the last-processed LR video frame are video frames of the intermediate LR video sequence; and a set of kernel estimation networks (KENet) configured, during the training of the apparatus, to: extract degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-pro-cessing LR video frame; contract the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degra-dation features; feed the contracted degradation features back to the LFGNet, adding to the training HR video sequence; and judge whether degradation of the intermediate LR video sequence is same as degradation of the testing LR video sequence; wherein the HFENet comprises: a coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain a coarse HR flow; a fine flow estimator configured to: warp a last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate a fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and 50 a HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The apparatus of claim 1, wherein the synthesis of the output LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet during training comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the output LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The apparatus of claim 1, wherein the KENet is further configured to execute a metric learning method in contract-ing the extracted degradation features; wherein the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features.
A method for training an apparatus for video superresolution (VSR) using generative adversarial networks (GAN), comprising: wherein the apparatus comprises: a set of low-resolution (LR) frame generation networks (LFGNet); a set of high-resolution (HR) frame estimation net-works (HFENet); and a set of kernel estimation networks (KENet); wherein the method comprises: synthesizing, by the LFGNet, a currently-processing LR video frame of an intermediate LR video sequence from an input HR video frame of a training HR video sequence; generating, by the HFENet, an output HR video frame of a reconstructed HR video sequence from the currently-processing LR video frame and a last-processed LR video frame of the intermediate LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, extracting, by the KENet, degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-processing LR video frame; contracting, by the KENet, the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degradation features; feeding the contracted degradation features back to the LFGNet, adding to training HR video sequence; and judging, by the KENet, whether degradation of the intermediate LR video sequence is same as degra-dation of the testing LR video sequence; wherein the HFENet comprises: a coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain a coarse HR flow; a fine flow estimator configured to: warp a last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate a fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and a HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The method of claim 4, wherein the synthesis of currently-processing LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the cur-rently-processing LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The method of claim 4, wherein the contracting of the extracted degradation features comprises executing a metric learning method, the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features.
An apparatus for video super-resolution (VSR) using generative adversarial networks (GAN), comprising: a set of low-resolution (LR) frame generation networks (LFGNet) configured to synthesize a LR video frame of an intermediate LR video sequence from an input high-resolution (HR) video frame of a training HR video sequence during a training of the apparatus; a set of high-resolution (HR) frame estimation networks (HFENet) configured to generate an output HR video frame of a reconstructed HR video sequence from a currently-processing LR video frame and a last-pro-cessed LR video frame of an input LR video sequence, and a last-generated output HR video frame of the reconstructed HR video sequence, wherein the HFENet comprises: a coarse flow estima-tor, a fine flow estimator and a HR frame synthesizer, wherein the coarse flow estimator generates a coarse HR flow according to the currently-processing LR video frame and a last-processed LR video frame, the fine flow estimator generates a fine HR flow accord-ing to the last-generated output HR video frame and the currently-processing LR video frame, and the HR frame synthesizer generates the output HR video frame according to the fine HR flow, the coarse HR flow and the last-generated output HR video frame, wherein the HFENet is trained during the training of the apparatus using the intermediate LR video sequence as the input LR video sequence, and the currently-processing LR video frame and the last-processed LR video frame are video frames of the intermediate LR video sequence; and a set of kernel estimation networks (KENet) configured, during the training of the apparatus, to: extract degradation features of the currently-processing LR video frame and a LR video frame of a testing LR video sequence corresponding to the currently-pro-cessing LR video frame; contract the extracted degradation features so to reduce the distance among the same degradation features and enlarge the distance among the different degra-dation features; feed the contracted degradation features back to the LFGNet, adding to the training HR video sequence; and judge whether degradation of the intermediate LR video sequence is same as degradation of the testing LR video sequence.
The apparatus of claim 7, wherein the synthesis of the output LR video frame of the intermediate LR video sequence from the input HR video frame of the training HR video sequence by the LFGNet during training comprises: down-sampling the HR video frame of the training HR video sequence to generate a down-sampled HR video frame of the training HR video sequence; and minimizing a content loss in the synthesis of the output LR video frame based on the down-sampled HR video frame, wherein the content loss comprises a pixel loss and a VGG (Visual Geometry Group) loss.
The apparatus of claim 7, wherein the HFENet com-prises: the coarse flow estimator configured to estimate a LR flow between the currently-processing LR video frame and the last-processed LR video frame, and up-scale the estimated LR flow to obtain the coarse HR flow; the fine flow estimator configured to: warp the last-generated output HR video frame of the reconstructed HR video sequence and the coarse HR flow to produce a first warped HR video frame; space-to-depth map the first warped HR video frame to produce a first warped LR video frame; and generate the fine HR flow, an occlusion mask matrix, and a residual frame from the first warped LR video frame and the currently-processing LR video frame; and the HR frame synthesizer configured to: warp the fine HR flow and the first warped HR video frame to produce a second warped HR video frame; and synthesize the output HR video frame of a recon-structed HR video sequence by fusing the second warped HR video frame and the residual frame according to the occlusion mask matrix; wherein the occlusion mask matrix comprises one or more fusion weight for fusing the second warped HR video frame and the residual frame; and wherein the residual frame comprises high-frequency details from the currently-processing LR video frame.
The apparatus of claim 7, wherein the KENet is further configured to execute a metric learning method in contracting the extracted degradation features; wherein the metric learning method comprises computing and minimizing a contrastive loss of the extracted degradation features. ∗ ∗ ∗ ∗ ∗
Layer stacks claimed or described, ordered top of device to substrate.
VistGAN video super-resolution apparatus
No layer stack recorded.
Patents and literature cited by this patent (applicant and examiner references).
Cited patents · 2
