How Modern AI Upscaling Actually Works
AI upscaling uses neural networks trained on millions of image pairs—low resolution versions matched with their high resolution counterparts. The network learns patterns: what a blurry edge should look like when sharp, how texture details typically appear at higher resolutions, what noise versus actual detail looks like. When you feed an image into an upscaler, it doesn't just stretch pixels. It analyzes the image in sections, identifies patterns it recognizes from training, and generates new pixels based on what it predicts should be there. A good upscaler trained on faces will reconstruct facial features with remarkable accuracy. That same upscaler might struggle with mechanical parts or fabric textures because it wasn't trained on those patterns. The architecture matters enormously. ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) uses a generator network that creates the upscaled image and a discriminator network that tries to distinguish between real high-res images and upscaled ones. This adversarial training pushes the generator to create increasingly realistic results. Real-ESRGAN, which I use for about 60% of my work, adds additional training on synthetic degradation—it learns to handle compression artifacts, blur, and noise that exist in real-world images, not just clean downsampled versions. Diffusion-based upscalers like StableSR work differently. They start with noise and gradually refine it into a high-resolution image, guided by the low-resolution input. This approach can generate incredibly detailed results, but it's also slower and can sometimes hallucinate details that weren't in the original image—a problem when you need to preserve the exact artistic intent of a texture. The processing happens in multiple stages. First, the image is analyzed and often split into overlapping tiles to manage memory usage. Each tile is processed through the neural network, which typically has 20-40 layers of convolutions, attention mechanisms, and residual connections. The tiles are then blended back together, with careful handling of the overlap regions to avoid visible seams. Finally, post-processing may sharpen edges, adjust color balance, or apply noise reduction. What makes 2026 different from earlier years is the emergence of specialized models. We now have upscalers trained specifically for anime art, for photographic portraits, for architectural renders, for pixel art. Using the right specialized model can improve quality scores by 0.1-0.15 SSIM points compared to general-purpose models—a significant difference when you're working at scale.The Night I Upscaled 400 Textures and Learned What Really Matters
It was 11 PM on a Thursday when the studio lead messaged me. They'd just gotten feedback from their publisher: all environment textures needed to be 2K minimum for the console version. They had 400 textures at 1024×1024, and certification was in three weeks. Recreating them wasn't an option—the original artist had moved to another studio, and the source files were a mess of lost PSDs and flattened exports. I started with Real-ESRGAN, my usual go-to. The first 50 textures looked great in the preview window. I queued up the rest and went to make coffee. When I came back, I spot-checked the results and shipped them to the studio. They integrated them into the build and sent me a thumbs up. Two days later, I got a different message. The textures looked wrong in-game. Not obviously bad—just slightly off. The stone walls had a weird smoothness to them. The wood grain looked almost plastic. The metal panels had lost their subtle surface variation. I pulled up the game build and compared it to the originals side-by-side. The upscaler had done exactly what it was trained to do: it had reduced noise and enhanced edges. But what I'd thought was noise in those textures was actually intentional surface detail—the tiny irregularities that make stone look like stone and not like a photograph of stone that's been smoothed in Photoshop. I spent the next 12 hours re-processing everything. This time, I used Swin2SR for the stone textures—it preserves high-frequency detail better. For the wood, I switched to a model I'd fine-tuned myself on lumber photographs. The metal got processed with Real-ESRGAN but with the denoise parameter set to -1 instead of the default 0, which tells it to preserve more of the original texture variation. The second batch looked right. But I'd learned something crucial: you can't just run everything through the same model and expect good results. Every texture type has different characteristics, and the upscaler needs to match those characteristics. A model that makes portraits look amazing will destroy the gritty detail in a concrete texture. That night taught me to categorize my textures before upscaling. I now sort everything into groups—organic materials, hard surfaces, fabrics, metals, painted surfaces—and use different models or parameters for each group. It takes longer, but the results are consistently better. And I always, always check the output in the actual game engine, not just in an image viewer. Context matters.Performance Data Across 8 Major Upscaling Models
I tested eight upscaling models on a standardized set of 150 game textures, measuring processing time, quality metrics, and subjective visual assessment. All tests ran on the same hardware: RTX 4080, 32GB RAM, processing 1024×1024 images to 2048×2048.| Model | Avg Time (s) | SSIM Score | PSNR (dB) | Best Use Case | Main Weakness |
|---|---|---|---|---|---|
| Real-ESRGAN | 2.3 | 0.89 | 28.4 | General purpose, organic textures | Can over-smooth fine detail |
| Swin2SR | 4.1 | 0.92 | 29.8 | High-detail preservation, technical art | Slower processing, higher memory use |
| BSRGAN | 1.8 | 0.85 | 27.1 | Fast batch processing, backgrounds | Lower quality on complex textures |
| StableSR | 12.7 | 0.94 | 31.2 | Hero assets, marketing materials | Very slow, can hallucinate details |
| HAT | 5.6 | 0.91 | 29.3 | Balanced quality/speed, production work | Requires more VRAM |
| RealCUGAN | 3.2 | 0.88 | 28.9 | Anime/stylized art, UI elements | Poor on photorealistic content |
| LDSR | 18.4 | 0.93 | 30.7 | Extreme detail recovery, archival | Extremely slow, inconsistent results |
| Waifu2x | 1.2 | 0.82 | 26.3 | Quick previews, 2D game sprites | Outdated, lower quality |
What the Quality Metrics Actually Tell You (And What They Don't)
SSIM and PSNR are useful, but they're not the whole story. I've seen upscaled images with SSIM scores of 0.94 that looked worse in-game than images with scores of 0.87. The metrics measure mathematical similarity to a reference image, but they don't measure whether the upscaled texture serves its purpose in the final context."A texture that scores 0.95 on SSIM but loses the subtle normal map detail that makes a surface feel three-dimensional is worse than a texture that scores 0.88 but preserves that tactile quality. The numbers don't capture what makes a texture work in a game engine."I learned this the hard way on a sci-fi project. The client wanted all their metal panel textures upscaled. I ran them through StableSR, got beautiful SSIM scores above 0.93, and delivered the files. The textures looked incredible in Photoshop. But when the lighting hit them in-engine, they looked flat. The upscaler had smoothed out the micro-variations in brightness that the engine's PBR (Physically Based Rendering) system used to calculate light reflection. I had to re-process everything with a different model and manually adjust the roughness maps to compensate. The second batch had lower SSIM scores—around 0.89—but they looked right in the game. The metrics didn't capture what mattered: how the texture interacted with the lighting system. This is why I always test upscaled textures in the target engine before delivering them. I'll drop them into Unity or Unreal, set up a simple lighting scenario, and actually look at them in context. I check them at the distances players will see them from. I rotate the camera to see how they respond to different lighting angles. I compare them side-by-side with the originals in the same scene.
"The best upscaling model is the one that produces results that work in your specific pipeline, not the one with the highest benchmark scores. I've used 'inferior' models because they preserved the exact characteristics I needed for a particular project."Another limitation of these metrics: they assume you have a high-resolution reference image to compare against. In real production work, you often don't. You're upscaling because the high-res version doesn't exist. So you can't calculate SSIM or PSNR—you can only evaluate the results subjectively. That's where experience matters. After upscaling thousands of textures, I've developed an eye for what looks right. I can spot when an upscaler has over-sharpened edges, when it's introduced ringing artifacts around high-contrast areas, when it's smoothed out detail that should be preserved. These are things the metrics don't catch. I also pay attention to color shifts. Some upscalers subtly change color values, usually making things slightly more saturated or shifting hues toward what they saw most often in training data. A 2% saturation increase might not affect your SSIM score, but it can make a texture look wrong when it's supposed to match other assets in a scene. The most useful metric I've found is actually the simplest: does it look right? I keep a reference folder of textures I know are high quality, and I compare my upscaled results against those references. Not mathematically—just visually. Does the level of detail match? Does the noise pattern look similar? Would I believe this texture was created at this resolution originally?
Why "Just Use the Highest Quality Model" Is Wrong
The conventional wisdom is simple: use the model with the best quality scores, and you'll get the best results. But that's not how it works in practice. The highest quality model is often the wrong choice. StableSR produces the highest SSIM scores in my tests. It also takes 12.7 seconds per image—more than five times longer than Real-ESRGAN. When you're processing 400 textures, that's the difference between 15 minutes and 85 minutes. If you're on a deadline, that matters. But speed isn't the only reason to avoid the "best" model. StableSR is a diffusion model, which means it generates details probabilistically. Run the same image through it twice, and you'll get slightly different results. That's a problem when you need consistency across a texture set. I learned this on a project where I upscaled a set of 60 brick wall variations. Each one came out slightly different in terms of mortar color and surface roughness. They looked great individually, but terrible together—the inconsistency was obvious when they were placed next to each other in the game world. Real-ESRGAN is deterministic. Same input, same output, every time. That consistency is more valuable than a 0.05 improvement in SSIM score. There's also the hallucination problem. High-quality diffusion models can add details that weren't in the original image. Sometimes that's good—it can make a blurry texture look sharp and detailed. But sometimes it's bad—it can add details that contradict the artistic intent or don't match the rest of the asset set. I had a case where I upscaled a set of fabric textures for character clothing. The original textures were intentionally simple, with a clean weave pattern. StableSR added subtle color variations and thread irregularities that looked realistic but weren't in the original design. The art director rejected them because they didn't match the game's stylized aesthetic. I had to re-process with a simpler model that preserved the clean, uniform look of the originals."The best model is the one that respects your constraints—time, consistency, artistic intent, technical requirements. Quality scores measure one dimension of performance, but production work is multidimensional."Another consideration: memory usage. HAT produces excellent results, but it requires 12GB of VRAM for 2K upscaling. If you're working on a laptop with an 8GB GPU, it won't run. BSRGAN produces lower quality results but runs on 4GB of VRAM. Sometimes the "best" model is the one that actually runs on your hardware. I also consider the source material. If I'm upscaling textures that are already fairly clean and high quality, just at a lower resolution, I'll use a simpler model like Real-ESRGAN. The source material is good enough that I don't need aggressive reconstruction—I just need more pixels. But if I'm working with heavily compressed JPEGs or textures that have been downscaled multiple times, I'll use something more powerful like Swin2SR that can recover detail from degraded sources. The "best" model also depends on what happens after upscaling. If the textures are going into a game engine where they'll be compressed again, viewed from a distance, and covered with lighting effects, you don't need the absolute highest quality upscale. A good-enough upscale that processes quickly is better. But if you're upscaling for print materials or marketing screenshots that will be viewed at 100% zoom, then yes, use the highest quality model you can.
Seven Steps for Production-Ready Upscaling
After three years and 1,200+ textures, I've developed a workflow that consistently produces good results. Here's the exact process I follow: 1. Categorize your textures before you start. Don't just dump everything into one folder and hit process. Sort them by material type: organic (wood, stone, dirt), hard surface (metal, plastic, concrete), fabric, painted surfaces, special cases (glass, water, particle effects). Each category may need different settings or even different models. I spend 15-20 minutes on this step for a typical 200-texture batch, and it saves hours of re-processing later. 2. Test on representative samples first. Pick 3-5 textures from each category and upscale them with your chosen model and settings. Don't just look at them in an image viewer—drop them into your game engine or 3D software and see how they actually look in context. Check them at different zoom levels. Rotate the lighting. Compare them to the originals side-by-side. If something looks wrong, adjust your settings or try a different model before processing the full batch. 3. Set up your processing parameters correctly. For Real-ESRGAN, I typically use: scale factor 2x, denoise level -1 to 0 depending on source quality, face enhancement off (unless you're actually upscaling faces), tile size 400-600 depending on VRAM. For Swin2SR: window size 8, scale 2x, no additional sharpening. For HAT: default settings work well for most cases, but reduce tile overlap if you're seeing seam artifacts. Write these settings down—you'll want to use the same parameters for consistency. 4. Process in batches of 50-100 images. Don't queue up 500 textures and walk away. Some models slow down or develop memory leaks over long processing runs. I process in batches, check the results after each batch, and restart the process between batches. This also lets you catch problems early—if batch 1 looks wrong, you haven't wasted time processing batches 2-10. 5. Implement a quality control checkpoint. After processing, I have a systematic QC process: open every 10th texture and check for common artifacts (over-sharpening, color shifts, loss of detail, seam issues if it's a tileable texture, unexpected smoothing). If I find problems, I check the surrounding textures. If more than 10% of a batch has issues, I re-process the entire batch with adjusted settings. This catches problems before they reach the client. 6. Preserve your originals and document your process. I keep the original textures in a separate folder, never overwrite them. I also maintain a simple text file that documents what model and settings I used for each batch. This is crucial when the client comes back six months later asking you to upscale 50 more textures that need to match the ones you did before. Without documentation, you're guessing at what settings you used. 7. Validate in the target environment. This is the most important step and the one people skip most often. Take a representative sample of your upscaled textures and test them in the actual game engine or application where they'll be used. Check them with the lighting system, with the LOD (level of detail) system, with any post-processing effects. I've caught issues at this stage that weren't visible in Photoshop—textures that looked great as static images but had problems with mip-mapping, textures that caused performance issues because they were too detailed, textures that looked wrong under the game's specific lighting setup. This workflow adds maybe 20% to the total processing time, but it reduces re-work by 80%. I used to just batch process everything and hope for the best. I'd end up re-doing 30-40% of textures because they had issues I didn't catch until later. Now I catch problems early, and my re-work rate is under 5%.The Hidden Cost of Upscaling: File Size and Performance
Nobody talks about this enough: upscaling increases file size, and that has real consequences. A 1024×1024 texture at 2048×2048 is four times the file size. If you're upscaling 400 textures, that's potentially gigabytes of additional data in your game build. I worked on a mobile game project where the client wanted all textures upscaled to improve visual quality. We upscaled everything from 512×512 to 1024×1024. The game looked better, but the build size increased from 180MB to 420MB. That pushed them over the 200MB threshold for cellular downloads on iOS. Players had to be on WiFi to download the game, which significantly hurt their install rates. We had to go back and selectively downscale textures that weren't critical to visual quality. Background elements, textures that were only visible from a distance, UI elements that were already sharp—these got reverted to their original sizes. We kept the upscaled versions only for hero assets and close-up surfaces. The final build was 240MB, which was acceptable. The lesson: upscaling isn't free. You're trading file size and memory usage for visual quality. Sometimes that trade is worth it. Sometimes it's not. There's also the runtime performance consideration. Higher resolution textures take longer to load, use more VRAM, and can impact frame rate if you're memory-constrained. On PC with 8GB+ of VRAM, this usually isn't a problem. On mobile devices or older consoles, it absolutely is. I now ask clients about their target platform and performance budget before I start upscaling. If they're targeting mobile or Switch, I'm more conservative with upscaling. If they're targeting high-end PC or PS5, I can be more aggressive. The "best" upscale isn't just about visual quality—it's about visual quality within the constraints of the target platform. Another consideration: compression. Game engines compress textures to save space and improve loading times. When you upscale a texture, you're adding detail. But if that detail gets compressed away by the engine's texture compression, you've gained nothing—you've just made the source file larger for no benefit. I test this by upscaling a texture, importing it into the game engine, letting it compress it, then exporting it back out and comparing it to the original upscale. If the compressed version looks nearly identical to the original lower-resolution texture, the upscaling didn't help. This happens more often than you'd think, especially with textures that have a lot of high-frequency detail that doesn't survive compression.When Not to Upscale: Recognizing the Limits
AI upscaling is powerful, but it's not magic. There are cases where it won't help or will actively make things worse. Don't upscale pixel art unless you're using a specialized pixel art upscaler. General-purpose models will smooth out the hard edges and destroy the aesthetic. Even specialized models can be problematic—they might add anti-aliasing or sub-pixel details that contradict the intentional low-resolution look. I've seen beautiful pixel art turned into muddy, unclear messes by well-meaning upscaling. Don't upscale heavily stylized or abstract textures. If the texture is intentionally simple, geometric, or uses flat colors, upscaling won't add anything useful. It might add noise or subtle gradients that contradict the artistic intent. I had a client who wanted to upscale UI elements that were designed as simple, flat shapes. The upscaler added subtle texture and shading that made them look less clean and professional. Don't upscale if the original is already at an appropriate resolution for its use case. This sounds obvious, but I've had clients ask me to upscale 2K textures to 4K "just because." If the texture is already high enough resolution that players won't see individual pixels, upscaling is just wasting file size and processing time. Don't upscale if you can recreate the asset from source. If you have the original PSD or 3D model, it's almost always better to re-export at the higher resolution than to upscale. Upscaling is a compromise—it's what you do when you don't have access to the source. If you do have the source, use it. Don't upscale textures that are meant to be viewed from a distance. Background mountains, skyboxes, distant terrain—these don't benefit from upscaling because players never see them up close. The extra detail is wasted, and you're just making your build larger. Don't upscale if the texture is going to be heavily modified afterward. If you're planning to paint over the texture, add effects, or significantly alter it, upscale after you make those changes, not before. Otherwise you're upscaling detail that you're going to paint over anyway. I've also learned to recognize when a texture is too degraded to upscale successfully. If the source is a heavily compressed JPEG with visible artifacts, or if it's been downscaled and upscaled multiple times already, AI upscaling might not help. The model will try to reconstruct detail, but it will also amplify the existing artifacts. Sometimes the best solution is to recreate the texture from scratch.Advanced Techniques: Fine-Tuning and Custom Models
For specialized work, you can fine-tune upscaling models on your own dataset. This is more advanced, but it can produce significantly better results for specific use cases. I fine-tuned a Real-ESRGAN model on a dataset of 500 wood textures—lumber, plywood, bark, finished wood, weathered wood. The training took about 8 hours on my RTX 4080. The resulting model produces noticeably better results on wood textures than the general-purpose model. It preserves grain patterns better, handles knots and irregularities more naturally, and doesn't over-smooth the surface texture. The process isn't trivial. You need a dataset of paired images—low resolution and high resolution versions of the same textures. You need to set up the training environment, which involves installing Python libraries, configuring CUDA, and understanding the training parameters. But if you're doing a lot of work in a specific domain, it's worth the investment. I've also experimented with ensemble approaches—using multiple models and blending the results. For example, I might upscale a texture with both Real-ESRGAN and Swin2SR, then blend them 70/30 to get the speed of Real-ESRGAN with some of the detail preservation of Swin2SR. This is more complex and slower, but it can produce results that are better than either model alone. Another technique: multi-stage upscaling. Instead of going directly from 1024×1024 to 4096×4096 (4x), I'll upscale to 2048×2048 (2x), then upscale that result to 4096×4096 (another 2x). This can produce better results than a single 4x upscale, especially if you use different models for each stage. The first stage focuses on reconstructing detail, the second stage focuses on refining that detail. I also use pre-processing and post-processing steps. Before upscaling, I might run a denoising filter to clean up compression artifacts, or adjust contrast to make details more visible to the upscaler. After upscaling, I might apply subtle sharpening, adjust color balance, or add back some noise to make the texture look more natural. These advanced techniques take more time and require more expertise, but they're valuable when you need the absolute best results. For routine production work, the standard models and workflows are usually sufficient. But for hero assets, marketing materials, or cases where quality is critical, these techniques can make a significant difference.The Decision Matrix for Choosing Your Upscaler
Here's how I decide which upscaler to use for any given project: Use Real-ESRGAN when: You need reliable, consistent results across a large batch of textures. Your source material is reasonably clean. You're working on a deadline. The textures are general-purpose—organic materials, environments, props. You need deterministic output that's the same every time. This is my default choice for 70% of projects. Use Swin2SR when: You need maximum detail preservation. You're working with technical textures where fine detail matters—normal maps, height maps, technical diagrams. You have the time for slower processing. Your source material has a lot of high-frequency detail that needs to be preserved. The textures will be viewed up close or at high zoom levels. Use StableSR when: You're upscaling hero assets or marketing materials where quality is more important than speed or consistency. You're okay with some variation between runs. You want the absolute best visual quality and you're willing to manually review and potentially re-process results. You have significant processing time available. Use HAT when: You need a balance between quality and speed. You have sufficient VRAM (12GB+). You're working on production assets that need to be better than Real-ESRGAN but don't require the extreme quality of StableSR. You want good results without the hallucination risks of diffusion models. Use BSRGAN when: Speed is critical and quality can be good-enough rather than perfect. You're upscaling background elements or textures that won't be viewed closely. You're working with limited hardware. You need to process thousands of images quickly. Use RealCUGAN when: You're working with anime-style art, stylized 2D graphics, or UI elements. Your textures have clean lines and flat colors. You need to preserve the stylized aesthetic rather than add photorealistic detail. Use specialized pixel art upscalers when: You're working with actual pixel art and need to preserve the hard edges and intentional low-resolution aesthetic. General-purpose models will destroy pixel art. Don't upscale when: You have access to source files and can re-export at higher resolution. The texture is already appropriate resolution for its use case. The texture is heavily stylized or abstract. The texture will be viewed from far away. The source is too degraded to upscale successfully. The key is matching the tool to the task. There's no single "best" upscaler—there's only the best upscaler for your specific needs, constraints, and context. Understanding what each model does well, what it does poorly, and what trade-offs it makes is how you consistently get good results. After upscaling 1,200 textures, I've learned that the technology is only part of the solution. The other part is understanding your requirements, knowing your tools, and having a systematic workflow that catches problems early. The models will keep improving, but the fundamental principles—test before you commit, validate in context, match the tool to the task—those stay the same.Disclaimer: This article is for informational purposes only. While we strive for accuracy, technology evolves rapidly. Always verify critical information from official sources. Some links may be affiliate links.