Restoring Skyboxes

[PikaCommando]

I've already talked about this before back on Facepunch, but I'm posting it here again so the information won't be lost. (first of all, credits to YaTebyaNashel for doing the handy list of HL2 leak skyboxes thread)

Have you guys ever wondered why most HL2 leak skybox textures appear unnaturally white as if a white filter was placed over it?


sky_c17_03

Notice how there's some green underneath all the whiteness?

The reason is (probably) because in a haste to make new skyboxes to fit the game's new direction post-pollution-era, the artist repurposed the old skybox textures by changing the exposure (or more accurately, gamma) to make them look gray-cloudy to get a feel of how the skies would look like in the new direction before actually creating the new skybox textures. Hence, I tried to reverse the process and here's what I end up with:


sky_c17_03

I believe this is how the texture looked like when it was first created to fit the direction of the game at the time, before being photoshopped to fit the new direction (and also looks like a solid box was added in later to hide the sea in the bottom half). Of course, it's not an exact 1:1 restoration as the process of photoshopping them twice (first by the artist to make them white, second by me to revert the first) has irreversibly damaged them, causing ugly pixelated artifacts and such.

This does not appear to be a coincidence or a misunderstanding of any kind. Changing gamma values normally does not turn white things green unless they were already green in the first place. For textures that appear to have been created after they decided to remove the pollution aesthetic such as sky_bloomtest01 (which has also undergone a gamma change to make it more white for some reason), changing their gamma does not turn them green; instead, it just makes the colors look more natural.

(above is leak version, bottom is gamma-adjusted by me)


sky_bloomtest01

I have no doubts that the bottom version is how the skybox looked like when it was first created, and that it wasn't created as it is in the leak or else this would not have been possible.

Anyway, here's the rest of the whitened skybox textures being restored to their original forms:

sky_c17_01



sky_c17_02


(the city appears to be photoshopped into the texture later on which is why it doesn't match the other textures)

sky_c17_04



sky_c17_05



sky_citadel01



sky_citadel02



sky_depot01



sky_station01



sky_trainstation01



sky_wasteland01



sky_wasteland02



sky_wasteland03



sky_wasteland04


(this has been brightened way too much, so the restoration is worse than the rest)

sky_wasteland05


(this has been brightened way too much, so the restoration is worse than the rest)


You know what they say, the simplest explanation is most often the correct one. I'd be happy to hear any alternative reasonings as to how this could have happened otherwise. What do you guys think?

[Shift]

Here is a random photo of a cloudy day sky from google

Here is the same photo, but with lowered gamma.


I don't see how your method proves anything, just on the basis it doesn't make skyboxes look particularly ugly,  and I certainly don't see it as "restoration".

And why would Antonov be in any haste in 2001-2002, when HL2 wasn't even slightly ready, or why didn't he edit the old skyboxes we know about?
Simplest explanations are just the easiest ones, and we exist in a complex reality.

[PikaCommando]

Here is a random photo of a cloudy day sky from google
 
Here is the same photo, but with lowered gamma.
 

I don't see how your method proves anything, just on the basis it doesn't make skyboxes look particularly ugly,  and I certainly don't see it as "restoration".

And why would Antonov be in any haste in 2001-2002, when HL2 wasn't even slightly ready, or why didn't he edit the old skyboxes we know about?
Simplest explanations are just the easiest ones, and we exist in a complex reality.

By "haste" I meant quickly whipping out something new using existing assets to get a feel of how the actual thing would look like aka drafting a placeholder, not that they were in a rush or anything.

As for why he didn't edit the other skyboxes such as the AirEx or Borealis ones, it could simply be because they did not want to reuse the AirEx one (since it was cut alongside pollution) and their vision for the Borealis sky did not change much up until the time of its scrapping.

My explanation for the missing older sky_c17_04 and 05 with the green skies (shown in the envmap vert_atrium01 and in leak CSS with purple crosses respectively) is that they were possibly edited to be whiter at some point after the envmap was made/copied into leak CSS, but were also the first to be overwritten by the newer skyboxes we see here with the same names. Since sky_c17_04 appears nearly identical to the retail's sky_day03_02, it's safe to say the new sky_c17_04 was among the first of the final skyboxes to be finished and implemented at the time of the leak, which is why the old sky_c17_04 does not exist in a whitened form in the leak as it being a placeholder was no longer needed anymore.

I agree that you do have a point regarding that simply making it look better does not mean it really was how they originally looked like. However, it's still noteworthy that this process makes them look just like other skyboxes the artist has previously done for Kingpin: Life of Crime (namely in color palette and style) and therefore incredibly likely to be their original state as the artist may have created these early HL2 skyboxes in the same manner as Kingpin's since Viktor Antonov was hired right after the release of Kingpin in 1999 together with the level designer Aaron Barber.


One of Kingpin's stylistically similar skyboxes.

[Shift]

My point is that you can gamma-lower literally any picture of bright sky, and it will look like any other picture of a dark sky, which is why "this process makes them look just like other skyboxes the artist has previously done for Kingpin".

And explaining that "we don't have any evidence that skyboxes were actually ever edited, because Valve conviniently cleaned up all traces of it" doesn't convince me.

[PikaCommando]

My point is that you can gamma-lower literally any picture of bright sky, and it will look like any other picture of a dark sky, which is why "this process makes them look just like other skyboxes the artist has previously done for Kingpin".

And explaining that "we don't have any evidence that skyboxes were actually ever edited, because Valve conviniently cleaned up all traces of it" doesn't convince me.

Not necessarily. For instance, the new sky_c17_04 and 05 are noticeably in a different style compared to the gamma-lowered 02 and Kingpin skyboxes as it was made to be gray cloudy in the first place.

Also, I disagree that you can gamma-lower literally any picture of bright sky to get a Kingpin-style sky; most skies are like the example you have shown above bluish-purple, occasionally orange. But none; no matter how much you gamma-lower them; can emulate the art style (and the green-yellow color palette) shown in Kingpin's skyboxes or even other older HL2 skyboxes like the old sky_c17_05.

Valve did not "conveniently clean up all traces of pre-edited skyboxes"; it is simply the result of opening the file up in photoshop, editing it, and then saving the file again, thus overwriting the original texture. Why would Valve leave traces of the original pre-edited textures since the goal is to replace them in the first place?

If anything, the evidence that they have been edited is staring us right in the face; the fact that the ocean/terrain in the bottom half is also whitened, losing most of the details as a result. Take the strange pixel artifacts in the ocean of sky_citadel02 for example; if it was meant to be a cloudy sky in the first place, the sea would've accurately reflected the sky's tone and not have these blotches like in other unedited skyboxes. Much like how lowering the gamma of a less-edited sky like the new sky_c17_04 introduced unnatural blotches in the terrain, I'd say that these blotches are what happens when the gamma of the original's night sea is raised, which messes up the dark ripples/refraction of the night sea and resulted in these blotches that we see now in the brightened version in the leak.

If the skies were created to be cloudy white in the first place like the new sky_c17_04 and 05, then why is the 3D-generated terrain/sea almost completely white and unnaturally lacking the natural dark ripples/refractions of a sea under a bright cloudy sky? Here's a gamma-raised sky_quarry03 for you to ponder:

(above is gamma-adjusted by me, below is leak version)

[Cvoxalury]

The problem with that theory is leak/HL2's bright daytime skies look normal, but your darkened 'restored' versions look optically wrong. Like, "if there's that sun then the clouds wouldn't be that dark" kind of wrong. Your sky_wasteland02 and sky_wasteland03 are example of that.

Or, with your last example, "gamma-raised sky_quarry03", your turned the moon into the sun, but it remains too small (because moon is smaller) and when the "sun" is that low on the sky, the colour and level of light would not be completely different, basically here you got sunset sun height but noon light levels.

So, that sky is just wrong, even if that kinda proves that you could take a dark texture and make it a light texture. But then again - leak and HL2 skies look appropriate for why they were made - and also we have old light values on the maps that perfectly corespond with those bright skyboxes, not with polluted skyboxes. Your restored skyboxes only kinda look like the concepts but not the same, so that's not what happened to them.

So, the polluted skyboxes for the City, canals and the wasteland just never existed. There was nothing to recycle.
Palace skybox is a moody weather sunset skybox but not polluted looking.
Old vertigo lighting is reddish sunset against green clouds, that's arguably also normal.
Old Citadel skybox (green c17_05) is just atmospheric sunset in a particular palette.
Airex is the most polluted but it's meant for the industrial hellscape region, also if you try and brighten it up you get crap image.

[Shift]

"I disagree that you can gamma-lower literally any picture of bright sky to get a Kingpin-style sky"
I only mentioned getting convincingly-looking dark skies.

"it is simply the result of opening the file up in photoshop, editing it, and then saving the file again, thus overwriting the original texture. Why would Valve leave traces of the original pre-edited textures since the goal is to replace them in the first place?"
Why do we have any of the old skyboxes left, then? And why do these old skyboxes are conviniently unedited, while all "edited" skyboxes conviniently exist only in their "edited" forms? Not to mention, we know ow TWO versions of sky_c17_05 - yellow sunset one and black-and green one and none of three versions are edited versions of each other.

And if we take an example of sky_station01, it was specifically used for 2002 terminal, and we don't have any versions of terminal, where light environment doens't fit the bright skybox, while we have versions of e3_strider with yellow light environment that fits old yellow sky_c17_05 (as an example).

"why is the 3D-generated terrain/sea almost completely white and unnaturally lacking the natural dark ripples/refractions of a sea under a bright cloudy sky?"
Who said it's sea, when it can be just a generic color filling from the generator (just like there are skyboxes with terrain, or ground color, it's just a setting), that isn't supposed to reflect anything, that there were no such details in the first place?

Speaking of 3d generation, why not quickly 3d-generate new placeholder skyboxes a-la "shadertest" instead of editing old ones?

And what if artifacts in sky_citadel02 are instead introduced by DXT compression combined with hue shifting (that creates these kinds of issues due to digital color processing having some trouble with subtle blue/cyan hues) - because that's what citadel 02 is, hue-shifted citadel01. And brand-new retail skyboxes have similiar loss of quality and artifacting, but less pronounced, one ugly-edited skybox isn't a proof.
And you lowering gamma on citadel02 doesn't turn the sun into the moon properly, because value range is simply compressed down, so while gamma-raised sky_quarry may look like supposedly-gamma-raised citadel02, I already said - lowering or raising gamma on any random sky picture doesn't make it unconvincing. Unless there is sun or moon, that instantly become off.

[PikaCommando]

The problem with that theory is leak/HL2's bright daytime skies look normal, but your darkened 'restored' versions look optically wrong. Like, "if there's that sun then the clouds wouldn't be that dark" kind of wrong. Your sky_wasteland02 and sky_wasteland03 are example of that.

Or, with your last example, "gamma-raised sky_quarry03", your turned the moon into the sun, but it remains too small (because moon is smaller) and when the "sun" is that low on the sky, the colour and level of light would not be completely different, basically here you got sunset sun height but noon light levels.

So, that sky is just wrong, even if that kinda proves that you could take a dark texture and make it a light texture. But then again - leak and HL2 skies look appropriate for why they were made - and also we have old light values on the maps that perfectly corespond with those bright skyboxes, not with polluted skyboxes. Your restored skyboxes only kinda look like the concepts but not the same, so that's not what happened to them.

So, the polluted skyboxes for the City, canals and the wasteland just never existed. There was nothing to recycle.
Palace skybox is a moody weather sunset skybox but not polluted looking.
Old vertigo lighting is reddish sunset against green clouds, that's arguably also normal.
Old Citadel skybox (green c17_05) is just atmospheric sunset in a particular palette.
Airex is the most polluted but it's meant for the industrial hellscape region, also if you try and brighten it up you get crap image.

I think that's the point of the old skyboxes, to show that the clouds are dirty and dark even in the presence of the sun as the result of the pollution.

The point of my gamma-raised sky_quarry03 isn't the sky but the effects of raising the gamma on the sea in the bottom half of the texture, which is consistent with the pixelated artifacts on the leak version of the brightened skyboxes like sky_citadel02. My point is that those artifacts wouldn't exist if the gamma wasn't edited in the first place. Since they do exist, that means they have had their gamma raised.

If anything, sky_wasteland01 and 04 with similar "sunset sun height but noon light levels" shows that they were indeed darker skyboxes that were adjusted to appear brighter.

"I disagree that you can gamma-lower literally any picture of bright sky to get a Kingpin-style sky"
I only mentioned getting convincingly-looking dark skies.

"it is simply the result of opening the file up in photoshop, editing it, and then saving the file again, thus overwriting the original texture. Why would Valve leave traces of the original pre-edited textures since the goal is to replace them in the first place?"
Why do we have any of the old skyboxes left, then? And why do these old skyboxes are conviniently unedited, while all "edited" skyboxes conviniently exist only in their "edited" forms? Not to mention, we know ow TWO versions of sky_c17_05 - yellow sunset one and black-and green one and none of three versions are edited versions of each other.

And if we take an example of sky_station01, it was specifically used for 2002 terminal, and we don't have any versions of terminal, where light environment doens't fit the bright skybox, while we have versions of e3_strider with yellow light environment that fits old yellow sky_c17_05 (as an example).

"why is the 3D-generated terrain/sea almost completely white and unnaturally lacking the natural dark ripples/refractions of a sea under a bright cloudy sky?"
Who said it's sea, when it can be just a generic color filling from the generator (just like there are skyboxes with terrain, or ground color, it's just a setting), that isn't supposed to reflect anything, that there were no such details in the first place?

Speaking of 3d generation, why not quickly 3d-generate new placeholder skyboxes a-la "shadertest" instead of editing old ones?

And what if artifacts in sky_citadel02 are instead introduced by DXT compression combined with hue shifting (that creates these kinds of issues due to digital color processing having some trouble with subtle blue/cyan hues) - because that's what citadel 02 is, hue-shifted citadel01. And brand-new retail skyboxes have similiar loss of quality and artifacting, but less pronounced, one ugly-edited skybox isn't a proof.
And you lowering gamma on citadel02 doesn't turn the sun into the moon properly, because value range is simply compressed down, so while gamma-raised sky_quarry may look like supposedly-gamma-raised citadel02, I already said - lowering or raising gamma on any random sky picture doesn't make it unconvincing. Unless there is sun or moon, that instantly become off.

Like I've said before, the reason why the edited skyboxes only exist in their edited forms in the leak is because they are saved with same file names after being edited. It's the same reason why we only have the purple-crossed version of the old green/yellow sky_c17_05 from the leak CSS; because they opened it, edited it, and then saved it with the same file name so the original un-crossed version as seen in the citadel_001 envmap does not exist anymore. And yes, as I have mentioned before, the reason why the leak HL2's sky_c17_05 is different from leak CSS's is because the old green/yellow sky_c17_05 in the hl2 folder was overwritten by the finalized one we see in leak HL2 today.

Also, I doubt that sky_citadel02 is a hue-shifted sky_citadel01. While they indeed are pretty much the same thing just arranged differently, if they were already created to be bright and white in the first place, then hue-shifting wouldn't make much of a difference (since what colors are going to change if they're already near-white?). As you can see in the gamma-lowered versions, they are identical in hue save for some brightness differences (which is affected by gamma, not hue). This means that the artifacts are caused by gamma changes, not hue-shifting.

As for why they chose to raise the gamma of older skyboxes as placeholders for the new style instead of quickly generating a new one such as shadertest, the reasons are twofold:

1. Raising their gamma in photoshop to achieve a bright cloudy sky is faster than generating a new skybox in Terragen (or whatever it was they used)

2. The skies of 3d-generated skyboxes such as shadertest are noticeably poorer in quality (and in a completely different art style) than Viktor's painted ones like the rest. The skies are created by the artist painting onto 3d-generated skyboxes without painting the terrain/sea (which is normally obscured during gameplay anyway). This is why they raised and adjusted the gamma of older skyboxes rather than generating a new one for placeholders as the brightened old skybox textures still retain some of Viktor's painted details.

As for the light_environment, let me just remind you that the envmaps for c17_04_24 and testroom_light_outside shows skybox textures that are darker and greener than the current leak ones (although not as dark as my attempts, which means I made them darker than they should've been) still shows yellow lighting since the yellow sun is what gives off light, not the dark green clouds. That's why the clouds near the sun are also yellow.

Then again, the envmaps for c17_01_17 and the wasteland maps also show them in their current state, so you're probably right.

[Cvoxalury]

The point of my gamma-raised sky_quarry03 isn't the sky but the effects of raising the gamma on the sea in the bottom half of the texture, which is consistent with the pixelated artifacts on the leak version of the brightened skyboxes like sky_citadel02. My point is that those artifacts wouldn't exist if the gamma wasn't edited in the first place. Since they do exist, that means they have had their gamma raised.

But we have same pixelated artifacts on sky_airexchange01. Does that prove that it's also an edited skybox and that it has "ocean" under it? It's the same wavy bottom half, is that supposed to be the sea now? Or is the more likely explanation, is that it's just the software default ground effect. And the artifacts/banding are there because compression. They're on almost every skybox, some more some less depending on how "lucky" it got with compression (mostly depends on original palette, Shift mentioned how DXT algorhitm is biased towards certain hues).

[Cvoxalury]

I'll ask this however. Where's any trace of supposed old-non gamma raised-polluted skyboxes? There's no maps that have dark lighting. There's no screenshots. There's no cubemaps. The other old textures that we have, Airex, Palace, Spire, Citadel, they're accounted for now. The concept art often shows skyboxes that can be identified (the wasteland railroad uses sky_wasteland03, Citadel uses sky_c17_05)

You said simplest explanations are most likely right, but yours is opposite of simplest because you're assuming they went through this whole process but we instead have evidence that oldest C17 levels were pretty much same back then as they are today if you compile them. (save for some building textures but that's another topic).

And by the way.

2. The skies of 3d-generated skyboxes such as shadertest are noticeably poorer in quality (and in a completely different art style) than Viktor's painted ones like the rest. The skies are created by the artist painting onto 3d-generated skyboxes without painting the terrain/sea (which is normally obscured during gameplay anyway). This is why they raised and adjusted the gamma of older skyboxes rather than generating a new one for placeholders as the brightened old skybox textures still retain some of Viktor's painted details.

This is getting difficult to follow, but basically: you can totally tell those skyboxes ARE terragen skyboxes because the patterns in them look generated. And what do you even mean "the skies are painted without bottom half"? Well yeah, they take photos and paste them into the program so it can build the scene for them. And then maybe paint some effects, but not like paint them from scratch, by hand. And the program fills the bottom because obviously they didn't shoot it as some sphere, they shoot from the ground level. (after 2007 they stopped even using square skyboxes because bottom isn't needed at all).

[PikaCommando]

The point of my gamma-raised sky_quarry03 isn't the sky but the effects of raising the gamma on the sea in the bottom half of the texture, which is consistent with the pixelated artifacts on the leak version of the brightened skyboxes like sky_citadel02. My point is that those artifacts wouldn't exist if the gamma wasn't edited in the first place. Since they do exist, that means they have had their gamma raised.

But we have same pixelated artifacts on sky_airexchange01. Does that prove that it's also an edited skybox and that it has "ocean" under it? It's the same wavy bottom half, is that supposed to be the sea now? Or is the more likely explanation, is that it's just the software default ground effect. And the artifacts/banding are there because compression. They're on almost every skybox, some more some less depending on how "lucky" it got with compression (mostly depends on original palette, Shift mentioned how DXT algorhitm is biased towards certain hues).

Whether or not it's a sea or terrain is not the point, the point is that the bottom half of the skybox texture is unnaturally brightened. Default ground effect or not, it should not be that white if it was rendered with a cloudy sky preset much like how the ground is still visibly brown in sky_c17_04 (because it was already rendered with a cloudy sky in mind).

And yes, sky_airexchange also was edited albeit not for brightening, but to add in that image of the airex concept art in the background and it seems like the top half of the ground was hue-shifted to match the colors in the sky while the bottom half of the ground was gamma darkened.

This is getting difficult to follow, but basically: you can totally tell those skyboxes ARE terragen skyboxes because the patterns in them look generated. And what do you even mean "the skies are painted without bottom half"? Well yeah, they take photos and paste them into the program so it can build the scene for them. And then maybe paint some effects, but not like paint them from scratch, by hand. And the program fills the bottom because obviously they didn't shoot it as some sphere, they shoot from the ground level. (after 2007 they stopped even using square skyboxes because bottom isn't needed at all).

I'll ask this however. Where's any trace of supposed old-non gamma raised-polluted skyboxes? There's no maps that have dark lighting. There's no screenshots. There's no cubemaps. The other old textures that we have, Airex, Palace, Spire, Citadel, they're accounted for now. The concept art often shows skyboxes that can be identified (the wasteland railroad uses sky_wasteland03, Citadel uses sky_c17_05)

You said simplest explanations are most likely right, but yours is opposite of simplest because you're assuming they went through this whole process but we instead have evidence that oldest C17 levels were pretty much same back then as they are today if you compile them. (save for some building textures but that's another topic).

And by the way.

2. The skies of 3d-generated skyboxes such as shadertest are noticeably poorer in quality (and in a completely different art style) than Viktor's painted ones like the rest. The skies are created by the artist painting onto 3d-generated skyboxes without painting the terrain/sea (which is normally obscured during gameplay anyway). This is why they raised and adjusted the gamma of older skyboxes rather than generating a new one for placeholders as the brightened old skybox textures still retain some of Viktor's painted details.

This is getting difficult to follow, but basically: you can totally tell those skyboxes ARE terragen skyboxes because the patterns in them look generated. And what do you even mean "the skies are painted without bottom half"? Well yeah, they take photos and paste them into the program so it can build the scene for them. And then maybe paint some effects, but not like paint them from scratch, by hand. And the program fills the bottom because obviously they didn't shoot it as some sphere, they shoot from the ground level. (after 2007 they stopped even using square skyboxes because bottom isn't needed at all).

This whole process isn't that complicated at all; it's literally just raising gamma until they're white enough. Doesn't even take 10 seconds in Photoshop.

And yes, I meant basically what you said: they are generated via Terragen first and then the skies are edited and painted onto the sky part of the generated textures, which means the bottom half of the generated textures (aka the ground) should look just as they did when they were rendered.

[Shift]

"as I have mentioned before, the reason why the leak HL2's sky_c17_05 is different from leak CSS's is because the old green/yellow sky_c17_05 in the hl2 folder was overwritten by the finalized one we see in leak HL2 today."
Doesn't answer why neither of three are edited versions of each other.

"if they were already created to be bright and white in the first place, then hue-shifting wouldn't make much of a difference"
Why are you guessing if it would change anything because "colors are near white", when you already have one skybox with slightly greener hue and another with more pure-blue hue, and if you overlay them on top of each other and compare, you can see one is based on another?

"Raising their gamma in photoshop to achieve a bright cloudy sky is faster"
As was already said, they weren't in a haste, no deadline around the corner, can afford to spent 10 minutes on a skybox instead of 5.

"The skies of 3d-generated skyboxes such as shadertest are noticeably poorer in quality"
What does it matter for placeholders?

[PikaCommando]

"as I have mentioned before, the reason why the leak HL2's sky_c17_05 is different from leak CSS's is because the old green/yellow sky_c17_05 in the hl2 folder was overwritten by the finalized one we see in leak HL2 today."
Doesn't answer why neither of three are edited versions of each other.

"if they were already created to be bright and white in the first place, then hue-shifting wouldn't make much of a difference"
Why are you guessing if it would change anything because "colors are near white", when you already have one skybox with slightly greener hue and another with more pure-blue hue, and if you overlay them on top of each other and compare, you can see one is based on another?

"Raising their gamma in photoshop to achieve a bright cloudy sky is faster"
As was already said, they weren't in a haste, no deadline around the corner, can afford to spent 10 minutes on a skybox instead of 5.

"The skies of 3d-generated skyboxes such as shadertest are noticeably poorer in quality"
What does it matter for placeholders?

I don't know how else to explain "we don't see a brightened old sky_c17_05 because it has already been replaced by the final one" to you so let's leave it at that.

Yes, I did say that sky_citadel01 and 02 are pretty much the same thing that are just arranged differently. I am also saying that their hues aren't that different; green and blue are literally right next to each other in the hue scale so hue-shifting would not cause artifacts as damaging as that seen in sky_citadel02 as getting to completely fit each other would only require a rather small shift of around +15 or -15 (out of 180); certainly not enough to cause the blackness/whiteness to produce artifacts.

As I've already said, it's not about rushing for a deadline, it's about not wanting to spend more effort than necessary to make a placeholder. Brightening soon-to-be-scrapped skybox textures is faster and takes less effort to do. Therefore, why bother generating another placeholder in Terragen when you can just photoshop it to look like the desired result while looking better and takes less time/effort to accomplish?

[Johnnywick]

some of these seem like they did do that but most look like they were made with sunny-ness in mind