Thoughts on AI Upscaling?

Koei’s Romance of the Three Kingdoms game series—discuss it here.

Thoughts on AI Upscaling

It needs to be handled with care.
1
33%
It's cool!
2
67%
I don't like it.
0
No votes
Meh.
0
No votes
 
Total votes: 3

User avatar
ZL181
Budding Translator
Posts: 166
Joined: Tue Apr 07, 2020 8:16 am
Location: Underworld
Contact:

Thoughts on AI Upscaling?

Unread post by ZL181 »

AI Upscaling is a way people upscale low-quality images. Is it moral to the Koei artists? Is it cool to see the quality? Is it even necessary? Does it even work well? Those are some questions I thought about.
Xiahou Shang ROTK 11 Normal Portrait
Image
Xiahou Shang ROTK 11 Upscaled Portrait
Image
Wan Yu ROTK 11 Normal Portrait
Image
Wan Yu ROTK 11 Upscaled Portrait
Image
Cui Yan ROTK 11 Normal Portrait
Image
Cui Yan ROTK 11 Upscaled Portrait
Image
Wu Gang ROTK 11 Normal Portrait
Image
Wu Gang ROTK 11 Upscaled Portrait
Image
Yan Rou ROTK 11 Normal Portrait
Image
Yan Rou ROTK 11 Upscaled Portrait
Image
Wang Yun ROTK 13 Normal Portrait
Image
Wang Yun ROTK 13 Upscaled Portrait
Image
Meng Da ROTK 14 Normal Portrait
Image
Meng Da ROTK 14 Upscaled Portrait
Image
Last edited by ZL181 on Tue May 16, 2023 10:26 am, edited 3 times in total.
I look forward to discussing the things surrounding the historical Three Kingdoms era and contributing translations and information of the personages who roamed China. They are human after all.

Currently active.
DragonAtma
Scholar of Shen Zhou
Posts: 1125
Joined: Wed Aug 15, 2012 2:19 pm
Location: Long Island, NY

Re: Thoughts on AI Upscaling?

Unread post by DragonAtma »

Xiahou Shang: debatable; it's not bad, but feels as if they're a somewhat different style of art.

Wang Yun/Meng Da: SoSZ has a limit on image sizes, so they're barely bigger; you should upside some more RoTK11 portraits instead.
Unless I specifically say otherwise, assume I am talking about historical Three Kingdoms, and not the novel.

In memory of my beloved cats, Anastasia (9/30/06-9/18/17, illness), Josephine (1/19/06-9/23/17, cancer), and Polgar (4/8/07-3/22/23, illness).
User avatar
ZL181
Budding Translator
Posts: 166
Joined: Tue Apr 07, 2020 8:16 am
Location: Underworld
Contact:

Re: Thoughts on AI Upscaling?

Unread post by ZL181 »

I've amended the original post to include more ROTK 11 portraits.
I look forward to discussing the things surrounding the historical Three Kingdoms era and contributing translations and information of the personages who roamed China. They are human after all.

Currently active.
DragonAtma
Scholar of Shen Zhou
Posts: 1125
Joined: Wed Aug 15, 2012 2:19 pm
Location: Long Island, NY

Re: Thoughts on AI Upscaling?

Unread post by DragonAtma »

Yeah, the art style differences are definitely there. For example, compare the lines on Wan Yu's mustache and Yan Rou's fur to the upscaled versions.
Unless I specifically say otherwise, assume I am talking about historical Three Kingdoms, and not the novel.

In memory of my beloved cats, Anastasia (9/30/06-9/18/17, illness), Josephine (1/19/06-9/23/17, cancer), and Polgar (4/8/07-3/22/23, illness).
User avatar
R.P. Gryphus
Student
Posts: 102
Joined: Tue Jan 31, 2023 8:03 pm
Location: Canada
Contact:

Re: Thoughts on AI Upscaling?

Unread post by R.P. Gryphus »

This is neat. What are you using to achieve these results?
User avatar
James
Sausaged Fish
Sausaged Fish
Posts: 18487
Joined: Fri Jun 14, 2002 3:21 pm
Location: Happy Valley, UT
Contact:

Re: Thoughts on AI Upscaling?

Unread post by James »

What a fun idea to play with, ZL181!

I have access to numerous machine learning upscaling programs between my work and my long-standing love for photography, so had to hop in and play around with this a little bit. One neat thing to consider is that a more advanced program for this sort of thing may have modes and configuration options which you can use to define parameters based on what you are working with. A low-resolution image with a lot of compression? Artwork? Line drawings? Photography? Product photography and architecture? So options like that can help a lot to decide how true you want to be to the source artwork, or how much you might want to depart.

In this case I used Topaz Gigapixel. It’s one of the leading programs for this sort of thing. There are now similar capacities baked into a lot of programs, although sometimes they are purpose-built for the use of that program (e.g. for photography).

I didn’t upload them here. I uploaded them to another website and linked them, and controlled the max side using the max_width tag I set up for this forum by request a short time ago. You can open the images in a new window to view them in full resolution. But sorry for the file sizes if anyone here is still using a slow connection. :)

One fun thing is that, with settings that were generally well-behaved across a dataset, it would be possible to create a high-resolution version of an entire collection of portraits.

- - -

Guan Yu (RTK VIII)
Image

Here is an upscale in a mode which I have set up to hold fairly close to the original piece. It really ends up bringing out the solid, heavy lines and delineations. It’s kind of funny to see how badly this program fails on the gold bit on his headpiece. It has no idea what it is. Because it is machine learning based, it is trying to make heads or tails of it based on its training.
Image

This one is fun. This upscaling mode is focused on photography and the like, and it understands faces. It recognizes Guan Yu’s face and is making choices (in solving the “ill-posed problem” part of upscaling) based on what it understands of human faces and anatomy. Without hesitation it departs from the original art in doing this, other than honoring quite precisely dimensions and separation, and starts to introduce things like a transition from less hair growth at the top of the beard into thinker hair on down. It has also gone out of its way to expand the fabric of his clothing into something that looks quite impressive, and has tried to flesh out the detail in the clothing around his neck.
Image

- - -

Pretty Pretty Pei (RTK XIII)
Image

This time I focused the upscaling on remaining reasonably true to the source piece. The standard modes which start to incorporate some knowledge of faces, hair, clothing seem to do well on these newer portraits, as they have moved a bit more into realistic depictions and textures. There is very little going on, here, which feels like a clear failure. One thing I do notice is the stronger texture and separation on the band of his leather armor where it is catching highlights. This is a common upscaling artifact. Because the separation of lights and darks is so much stronger in the original artwork, this ends up being amplified in upscaling in a manner that feels a bit removed.
Image

A crop on his splendor.
Image

- - -

Jinhuan Sanjie (RTK XIII)
Image

This upscaling is just legendary. The art style, here, just feeds in so nicely to what the machine learning understands. And there are numerous fun textures for it to play with, from teeth, hair, clothing, fur, skin, metal, and it really highlights what these programs can do.
Image

It was possible to upscale this one pretty considerably without things materially breaking down. And it is just such a delight. Really makes me chuckle. You could make a really amazing (lol!) wallpaper out of this one.
Image
User avatar
James
Sausaged Fish
Sausaged Fish
Posts: 18487
Joined: Fri Jun 14, 2002 3:21 pm
Location: Happy Valley, UT
Contact:

Re: Thoughts on AI Upscaling?

Unread post by James »

Thoughts on AI upscaling?

It’s a very cool thing, but has some clear limitations. Probably the biggest is that these programs are driven by machine learning, so what they do is a product of the dataset they are trained on. They generally do not focus on a simple universal upscaling, but rather on trusting the machine learning algorithm to make its own connections in how details may be upscaled, so you end up with certain types of textures and shapes being treated in a certain way. This can mean something you feed it, which contains data that looks like something the machine learning understands, but which it is not, can end up being upscaled in a manner that made more sense for what the machine learning “thought” it “recognized.” This very easily creates artifacts in what is being upscaled, which can limit its application where you want to be authentic to the source material.

It can be useful or photography, but I rarely want to upscale something more than 2x or so because artifacts are quickly introduced. I will sometimes blend the result in Photoshop where I can use layer masking to choose a bit how and where to apply the upscaling (e.g. keep it away from out-of-focus details where it can be highly selective, and careful with it on something like an eye or a face).

When working with certain types of artwork, it can be pretty amazing. As you can see in the post above, it can fail in some ways or deviate in some ways from what might be ideal, but with the right options it can do a pretty good job of producing what someone is after. And if they wanted to put a bit of work into it, artifacts could be addressed (e.g. as described above).

One of my most common uses for it is for a professional website I work on—main source of income—where I frequently need to add products that are old or have relatively iffy product photos. I can use tools like Gigapixel to upscale them a bit, deal with JPG artifacts, so the website broadly keeps a professional and polished look even when working with old graphics.

For photography, I tend not to use these upscaling tools too much, and where I do, not very aggressively. But the noise-reduction/removal tools can be pretty amazing, as can be, in some cases, sharpening tools. But both of those also have their machine learning limits.

I once ordered a really lovely piece of koi fish artwork from an artist who produced some really lovely work, and what was delivered to me, through a third-party that takes the source images provided and prints from them, was not very good artwork. It was not high enough resolution to display well at the ordered print size, and the original image the artist provided was noticeably compressed, so the actual artwork featured noticeable JPG compression. Might not have stood out to many people, but I see that like a sore thumb with my work background. So, after deliberation, I scanned the artwork in pieces, stitched it together, and used machine learning programs to remove the compression and upscale it. Then I re-printed it on metallic paper which caused the now-high-resolution and fine-resolution swimming koi to pop out as though they were in water. Then I made a frame for it, put the “original” artwork behind it in the frame, and now I have a really amazing piece of art in my office. But… while I did inform the artist that the resources they sent to the company were inferior or being used poorly, I didn’t share those details, and also didn’t ask for a refund (as I ended up using the artwork anyhow).

I don’t know that they would appreciate learning of what I did. :lol:

I also have a really lovely piece of Totoro artwork which I prepared in a similar way. It looks amazing.
Post Reply