Page 1 of 1

Experiments in AI Upscaling

Posted: Fri Jan 17, 2020 8:29 pm
by DaveW
Hello all

I've recently been tinkering with AI Upscaling (again - I released an AI upscale pack for Deus Ex last year) and decided to try it on Unreal textures. Not sure if anyone's done this yet.

So far I think the results are pretty interesting. This is using ESRGAN with the 'Manga109attempt' model, which seems to be best suited to game textures:

https://imgsli.com/MTExMzY

https://imgsli.com/MTExMzc

https://imgsli.com/MTExMzg

https://imgsli.com/MTExMzk

https://imgsli.com/MTExNDA


The results are less dramatic than the URP, which is also the case with New Vision - but it just adds an extra bit of sharpness when playing at high res without changing the aesthetic / style of the original.

Currently exporting the textures is a bit of a pain and I'm not sure how to do the textures in .u files (mutators seem to be the best way? No idea how to do that) but if nothing else I think this is pretty cool!

Re: Experiments in AI Upscaling

Posted: Sat Jan 18, 2020 2:42 am
by Pitbull
Looks impressive.

Re: Experiments in AI Upscaling

Posted: Sat Jan 18, 2020 11:36 am
by Hellkeeper
Let me get this straight, this is all computer-generated upscaling via an AI? The result is close to prefet and needs no hand work? Very cool.

Re: Experiments in AI Upscaling

Posted: Sat Jan 18, 2020 12:00 pm
by DaveW
Exactly, each texture takes approximately 5 seconds to process on my PC (Ryzen 3700x / RTX 2070), since most of the process time is in actually building the models in the first place. In this case, I'm using someone else's trained model - which I think is the best approach. A model is basically a set of instructions telling the AI how to upscale particular details, trained on a massive dataset of other textures.

It might be possible to train a model on Unreal's textures themselves, using the official high resolution textures that were released with UT, but I think that might give more inconsistent results (being a small sample size). So far, these have been pretty good.

The great thing about this technique is it can also be applied to any custom textures quite easily. I will upload some more examples later.

Re: Experiments in AI Upscaling

Posted: Sun Jan 19, 2020 5:19 am
by Lightning_Hunter
I don't mean to steal your thunder, but Ahaigh01 did exactly what you are describing with the UnrealHD texture pack 2.0! He first ran all the textures through various AI upscalers, then tweaked the small details by hand:
https://www.moddb.com/games/unreal/addo ... d-textures
The only textures not upscaled were the very best ones by Diehard for the original Unreal HD pack, but lots of the lower quality ones were redone.

As for the player skins, I personally tackled those based on AI upscaled skins by Ahaigh01, but I almost completely reworked those by hand for my UnrealHD mod. The rest of the game skins I did without upscalers:
https://www.moddb.com/mods/high-resolut ... nrealhd-v2

Re: Experiments in AI Upscaling

Posted: Fri Jan 24, 2020 4:55 pm
by DaveW
I think if I created a full pack out of this, it would be pure AI upscale and no manual tweaking, to get as close to the original look as possible. At least then there's the option of a 'purist' release, and the existing ones with some manual texture updates.

The problem is artifacting on some textures, but I'm getting a bit better at reducing those now.

Re: Experiments in AI Upscaling

Posted: Sat Feb 08, 2020 3:51 am
by Lightning_Hunter
I think if I created a full pack out of this, it would be pure AI upscale and no manual tweaking, to get as close to the original look as possible. At least then there's the option of a 'purist' release, and the existing ones with some manual texture updates.

The problem is artifacting on some textures, but I'm getting a bit better at reducing those now.
I don't wish to de-motivate you, but AI Upscaling has all sorts of problems beyond artifacts that, if not manually reworked, looks worse than the original textures in many cases. Ahaigh01 has been working with upscalers for quite some time now, and I have seen him run thousands of textures through them.  I personally pointed out hundreds of textures that turned out kind of awful looking without manual tweaks.  The biggest example of this is when upscalers attempt to upsize screws, decals, and other sharp details. The upscaler will often guess what these details are, and butcher them in the process.  The result is a decal or screw that no longer looks like a decal or screw. Such details always need manual reworking to look like they are supposed to.  Otherwise, they look terrible and you are better off sticking to the original textures.

At least, that is my two cents, but don't let me stop you from working on textures of course.  We could always use more texture artists around here. 

Re: Experiments in AI Upscaling

Posted: Sat Feb 08, 2020 4:39 pm
by yrex .
I have no idea if this would work and look good, and there's probably no tool that would do that, but if you care about accuracy, you could simply keep the original texture for the lower-res mipmaps.

Re: Experiments in AI Upscaling

Posted: Fri Aug 28, 2020 1:14 am
by SolivaN
I think if I created a full pack out of this, it would be pure AI upscale and no manual tweaking, to get as close to the original look as possible. At least then there's the option of a 'purist' release, and the existing ones with some manual texture updates.

The problem is artifacting on some textures, but I'm getting a bit better at reducing those now.
I don't wish to de-motivate you, but AI Upscaling has all sorts of problems beyond artifacts that, if not manually reworked, looks worse than the original textures in many cases. Ahaigh01 has been working with upscalers for quite some time now, and I have seen him run thousands of textures through them.  I personally pointed out hundreds of textures that turned out kind of awful looking without manual tweaks.  The biggest example of this is when upscalers attempt to upsize screws, decals, and other sharp details. The upscaler will often guess what these details are, and butcher them in the process.  The result is a decal or screw that no longer looks like a decal or screw. Such details always need manual reworking to look like they are supposed to.  Otherwise, they look terrible and you are better off sticking to the original textures.

At least, that is my two cents, but don't let me stop you from working on textures of course.  We could always use more texture artists around here. 
I don't think so, for example:
Here the hair have an abrupt contrast and less soft degraded colors between skin and hair:
Image

And here, we have the contrary, more soft degraded between skin and hair:
Image

I understand that you mean with "When upscalers attempt to upsize screws, decals, and other sharp details. The upscaler will often guess what these details are, and butcher them in the process."

You're right and I prefer the manual tweak that you refer, for example:
Image

This upscaled texture looks terrible.

So, I think that the idea of DaveW is great, only needs take care with the textures with screws and/or with small decals that will be a bad converted texture.

So, go ahead dude, I've makin the upscale for Quake III Team Arena and the result are impressive!

I've finished the tirst tournament map The House of Decay.

Original:
Image

Upscaled:
Image

This take me a lot of time of search the images into the files of the game, and select the right image to convert.
So, I know thats take a hard work to do.

Go ahead dude, I hope you can release you work soon.
Cheers.

Re: Experiments in AI Upscaling

Posted: Mon Dec 27, 2021 10:56 pm
by qduaty
Well, it inspired me a bit and I tested a newer anime model, released 2 weeks ago (xinntao Real-ESRGAN). Skins are not very sharp (as can be judged from weapons) but deals well with mostly everything else.

https://ibb.co/album/vZpTZM
https://ibb.co/album/1ZQyqW

Skins:
Anime model:
https://ibb.co/album/kgTJXg

PSNR model for comparison:
https://ibb.co/album/mDcnsJ

Re: Experiments in AI Upscaling

Posted: Sun Jan 02, 2022 12:29 am
by ahaigh01
Yeah, it looks like one of the nicest upscale models so far. Pretty cool.

Re: Experiments in AI Upscaling

Posted: Sat Feb 05, 2022 4:27 pm
by Kajgue
While the AI upscale method does look impressive from time to time, i have to admit that aside from artefacts, upscaling small features on small 256x256 textures will create a very broad and vague margin of interpretation for the AI upscaler, which often is very different to the way a human can interpret the features of a texture. I would also argue that creating new textures from 3d models is the best way to go as the workflow can be easy to manage and the lighting results will be consistent. I've had past experience with creating new textures in GIMP for example, which was good for a while; but after learning to do this with 3d models now instead, it's hard to imagine a substitute, unless one is intentionally creating low res textures for something like, Quake for example (I've seen some really neat texture artwork done solely either by hand or in Substance Painter being on some Quake modding discords) and trying to be consistent with such a game's artstyle.

3d models to textures are also the way to go especially if one is deciding to use PBR materials in their project, as the results on the relevant texture maps (diffuse, normal, roughness, metallic) will be in sync.

Still imo some of these AI upscales are pretty cool.