11GB of extracted (?) cache files, (+ 30GB GigaCache released)

[Carbon]

The Dropbox archives extracted without issue using Winrar.

Thanks again guys.

[gopostal]

Also extracted just fine for me. Bob and I are working on this but if someone else is too, here is a total listing of the files contained in the cache:
https://www.dropbox.com/s/bm972oe9aueuw ... t.zip?dl=0

The text file looks like this

Code: Select all

FF81FFA641D6B2377E4EE8B8944F1D3C.uxx
FF86986011D2DCE82EE6DF97A09F5863.uxx
FF91A7B14AFD34720A7CD6A67FA78EB6.uxx
FF91B39C4DFE22555539EF8D74C9D019.uxx
FF9276FB457D42BDC3ECBD97BE00108C.uxx
FF952CF5419AF8A400AD9FB23952E75A.uxx
FF95E5B74278AED1ED014F8D1ECCE31F.uxx

but at least it has every file (17,423 of them!). The cache.ini only lists 1470 of them so a TON of these are somewhat orphan.

Over the next week Bob and I will try to identify as much as we can and correctly sort them into folders. Once this is done I'll make a torrent where you can select the files you want and download only those instead of having to get the entire thing as a zip.

I haven't been this excited about Unreal in a long time....Wonder if I can call in sick for the next week?

BTW, if you have a large cache file(s) and you have some free time, could you send me your cache.ini file? It would make ID'ing some of these way faster if I had a master cache list to compare it to. Putting mine and Bob's and this one together I was able to create one that closes in on 10,000 lines but I could always use more to enhance it. You can email it to agutgopostal at hotmail dot com or via PM. Much appreciated!

[[]KAOS[]Casey]

yeah...

this cache deobfuscator is going to take a while.

I totally forgot about what if the maps dependencies.... have dependencies?... that have dependencies? IT's even possible to have circular dependencies (I did it once on accident. Oops.)  because of unrealed, but that's probably not going to be a big deal.

Going to have to make some storage classes that figure all this jazz out, OR figure out how I can know if a packages export table matches what a package wants on its import table..

Code: Select all

Package (00216D0948A3059AAF0A20B58D48B476) : Import [0]->ObjectName == 'DecayedS', Package = (1)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [1]->ObjectName == 'Engine', Package = (1)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [2]->ObjectName == 'Base', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [3]->ObjectName == 'Starship', Package = (1)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [4]->ObjectName == 'Base', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [5]->ObjectName == 'Floor', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [6]->ObjectName == 'Dterbas1', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [7]->ObjectName == 'UnrealShare', Package = (1)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [8]->ObjectName == 'Grey', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [9]->ObjectName == 'Brush', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [10]->ObjectName == 'Dark1', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [11]->ObjectName == 'Polys', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [12]->ObjectName == 'Model', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [13]->ObjectName == 'sh_b5', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [14]->ObjectName == 'Mine', Package = (1)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [15]->ObjectName == 'FLOOR8E', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [16]->ObjectName == 'Deco', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [17]->ObjectName == 'Blakm_31', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [18]->ObjectName == 'PlayrShp', Package = (1)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [19]->ObjectName == 'Base', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [20]->ObjectName == 'MMCRATE2', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [21]->ObjectName == 'Deco', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [22]->ObjectName == 'PathNode', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [23]->ObjectName == 'Trim', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [24]->ObjectName == 'Spawnpoint', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [25]->ObjectName == 'dmFlor3c', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [26]->ObjectName == 'Ext_bs', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [27]->ObjectName == 'FLOOR8F', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [28]->ObjectName == 'Light', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [29]->ObjectName == 'dTrim3', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [30]->ObjectName == 'GenFX', Package = (1)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [31]->ObjectName == 'LensFlar', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [32]->ObjectName == '3', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [33]->ObjectName == 'UnrealI', Package = (1)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [34]->ObjectName == 'Deco', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [35]->ObjectName == 'Pillar', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [36]->ObjectName == 'sh_bh', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [37]->ObjectName == 'SF', Package = (1)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [38]->ObjectName == 'Light', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [39]->ObjectName == 'dHull2', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [40]->ObjectName == 'Mover', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [41]->ObjectName == 'Wall', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [42]->ObjectName == 'dBase2b', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [43]->ObjectName == 'Light5', Package = (0)
Package (00216D0948A3059AAF0A20B58D48B476) : Import [44]->ObjectName == 'General', Package = (0)
{hundred or so more lines}

00216D0948A3059AAF0A20B58D48B476 = some random map I picked out of my cache (I knew about it from cache.ini)
Package = (0) = Not a "base" package, i.e. not the root of a package, say, GenFX.lensflar.4, GenFX is the base package, and GenFX.lensflar is a sub package. 4 is the actual class.
Package = (1)  = a base package, like GenFX.

As you can see, the import table seems to be added on-demand. Taht is, if a mapper adds a skaarj right after adding some random texture the new latest indexes will be those two (assuming those classes have not been used before, and their base packages are already loaded), it's not sorted. It's just added.

So right now, I have to figure out if there's some way to map the random imports and base packages together so I can mass load packages until I find an export table that matches an import table for another package.

There needs to be more time in a day...


Edit: thanks to a little help from smirf.. I made a crazy function to magically get the root package name of an import from an import table.

No, I have no idea why it works like this.

Code: Select all

static FString GetImportPackage(ULinkerLoad *Linker, INT i )
{
      guard(GetImportPackage);
      const TCHAR *S;
      int x = i;
      while (true)
      {
            S = *Linker->ImportMap(x).ObjectName;
            x = -Linker->ImportMap(x).PackageIndex-1; //wtf is this. I don't even.
            if (x

[Carbon]

Thanks for your work Casey. I PMed gopostal the cache.ini I used; likely lots of double entries. If there is anything I can do to help, let me know.

[gopostal]

I'm still working on this, just had a six day work week was all. Anyway, I grabbed every cache.ini I could beg, borrow, plead, steal and made a master ini file. I then stripped duplicates and made some judgement calls on obscure mods that had been renamed four or five times. I ran it all through every compressed Unreal file I could find and extracted out all the matches. The resulting folder is almost 15GB. I made text lists of all the complete files in each folder and here is a zip of those:
https://www.dropbox.com/s/en1gk8q1vutqk ... t.zip?dl=0

Sadly there are still around 13,000 lines still in the cache that don't match anything I could find. This is a good start though. If you see anything in those lists you want right now just message me and I'll get them to you.

I'm working out now a way to batch ID files by the flags. I think I can separate a bunch out that are music tracks, etc. Bob and I should be able to make another big dent in this over the next couple of days.

[[]KAOS[]Casey]

Well some progress has been made. I can almost correctly detect first-level dependency package. Some Map -> some file it wants that it thinks is named "SF.u".. when I manually compare using choco package edit it all looks good, but then I noticed that I was getting twice as many matches as I should have against my listed import table and the export table of "SF.u"

So then I looked into the indexes, and I found out that every single unrealscript class was getting doubled in positive matches. It turns out that every unrealscript class has a duplicate export with the same name that's a text buffer. No problem, I thought. (besides that not really making sense since I thought you couldn't have two things with the same name)

I'll just use this function

class ULinkerLoad : public ULinker, public FArchive
{
...
FName GetExportClassName( INT i );
..
}

Code: Select all

if(Linker->GetExportClassName(y) == NAME_TextBuffer)
     continue; //skip TextBuffers since they inflate the match count, preventing false positives.

And then visual studio decided it would be ok to not be able to link to that function. WTF?

One of ULinkerLoad's parent classes (ULinker) has a function named GetExportFullName that actually works(??? thanks Obama!!!!!!!!!!!), but I can't directly compare the name without extracting it with string functions from there. So I want to reverse engineer (what GetExportClassName) would do for more efficiency.

[jaden88]

Well some progress has been made. I can almost correctly detect first-level dependency package. Some Map -> some file it wants that it thinks is named "SF.u".. when I manually compare using choco package edit it all looks good, but then I noticed that I was getting twice as many matches as I should have against my listed import table and the export table of "SF.u"

So then I looked into the indexes, and I found out that every single unrealscript class was getting doubled in positive matches. It turns out that every unrealscript class has a duplicate export with the same name that's a text buffer. No problem, I thought. (besides that not really making sense since I thought you couldn't have two things with the same name)

I'll just use this function

class ULinkerLoad : public ULinker, public FArchive
{
...
FName GetExportClassName( INT i );
..
}

Code: Select all

if(Linker->GetExportClassName(y) == NAME_TextBuffer)
     continue; //skip TextBuffers since they inflate the match count, preventing false positives.

And then visual studio decided it would be ok to not be able to link to that function. WTF?

One of ULinkerLoad's parent classes (ULinker) has a function named GetExportFullName that actually works(??? thanks Obama!!!!!!!!!!!), but I can't directly compare the name without extracting it with string functions from there. So I want to reverse engineer (what GetExportClassName) would do for more efficiency.

:|

[[]KAOS[]Casey]

First-level dependencies work now. Still hardcoded as to which cache-package I am starting with to figure out dependencies.

Code: Select all

E:\UnrealGold\227iSystem>ucc.exe CacheDeobfuscator.CAcheDeobfuscator
Failed loading package: Can't find file 'SF'
Failed loading package: Can't find file 'SFUTWeapons'
Failed loading package: Can't find file 'NWPakSF'


Attempting to Cache-Bruteforce packagename 'DecayedS',RealPackageName 'DecayedS'
Package 'DecayedS' already exists as file '..\Textures\DecayedS.utx'


Attempting to Cache-Bruteforce packagename 'Engine',RealPackageName 'Engine'
Package 'Engine' already exists as file '..\227iSystem\Engine.u'


Attempting to Cache-Bruteforce packagename 'Starship',RealPackageName 'Starship'
Package 'Starship' already exists as file '..\Textures\Starship.utx'


Attempting to Cache-Bruteforce packagename 'UnrealShare',RealPackageName 'UnrealShare'
Package 'UnrealShare' already exists as file '..\227iSystem\UnrealShare.u'


Attempting to Cache-Bruteforce packagename 'Mine',RealPackageName 'Mine'
Package 'Mine' already exists as file '..\Textures\Mine.utx'


Attempting to Cache-Bruteforce packagename 'PlayrShp',RealPackageName 'PlayrShp'
Package 'PlayrShp' already exists as file '..\Textures\PlayrShp.utx'


Attempting to Cache-Bruteforce packagename 'GenFX',RealPackageName 'GenFX'
Package 'GenFX' already exists as file '..\Textures\GenFX.utx'


Attempting to Cache-Bruteforce packagename 'UnrealI',RealPackageName 'UnrealI'
Package 'UnrealI' already exists as file '..\227iSystem\UnrealI.u'


Attempting to Cache-Bruteforce packagename 'SF',RealPackageName 'SF'
Probably found package 'SF' as file '21205E6B492FAE8AC80DB3B712691C5F.uxx'
Probably found package 'SF' as file '817F3F534936705D424FD2A21EDE49EE.uxx'


Attempting to Cache-Bruteforce packagename 'Doorsmod',RealPackageName 'Doorsmod'
Package 'Doorsmod' already exists as file '..\Sounds\DoorsMod.uax'


Attempting to Cache-Bruteforce packagename 'Ancient',RealPackageName 'Ancient'
Package 'Ancient' already exists as file '..\Textures\Ancient.utx'


Attempting to Cache-Bruteforce packagename 'SFUTWeapons',RealPackageName 'SFUTWeapons'
Probably found package 'SFUTWeapons' as file '7E93911149739752368044B43C44F1FF.uxx'
Probably found package 'SFUTWeapons' as file 'EF3C6971481CC45F6266F58C61B9D8EF.uxx'


Attempting to Cache-Bruteforce packagename 'UPak',RealPackageName 'UPak'
Package 'UPak' already exists as file '..\227iSystem\UPak.u'


Attempting to Cache-Bruteforce packagename 'ATMisc',RealPackageName 'ATMisc'
Package 'ATMisc' already exists as file '..\Textures\ATMisc.utx'


Attempting to Cache-Bruteforce packagename 'ALFAFX',RealPackageName 'ALFAFX'
Package 'ALFAFX' already exists as file '..\Textures\AlfaFX.utx'


Attempting to Cache-Bruteforce packagename 'GenFluid',RealPackageName 'GenFluid'
Package 'GenFluid' already exists as file '..\Textures\GenFluid.utx'


Attempting to Cache-Bruteforce packagename 'LavaFX',RealPackageName 'LavaFX'
Package 'LavaFX' already exists as file '..\Textures\LavaFX.utx'


Attempting to Cache-Bruteforce packagename 'sm07crateriafall',RealPackageName 'sm07crateriafall'
Package 'sm07crateriafall' already exists as file '..\Music\sm07crateriafall.umx'


Attempting to Cache-Bruteforce packagename 'Core',RealPackageName 'Core'
Package 'Core' already exists as file '..\227iSystem\Core.u'


Attempting to Cache-Bruteforce packagename 'AmbModern',RealPackageName 'AmbModern'
Package 'AmbModern' already exists as file '..\Sounds\AmbModern.uax'


Attempting to Cache-Bruteforce packagename 'NWPakSF',RealPackageName 'NWPakSF'
Probably found package 'NWPakSF' as file 'EDA863A648F0B809FEA91D800CEDBB3E.uxx'


Attempting to Cache-Bruteforce packagename 'sm15hostile',RealPackageName 'sm15hostile'
Package 'sm15hostile' already exists as file '..\Music\sm15hostile.umx'


Attempting to Cache-Bruteforce packagename 'mp35pirateswintro',RealPackageName 'mp35pirateswintro'
Package 'mp35pirateswintro' already exists as file '..\Music\mp35pirateswintro.umx'


Attempting to Cache-Bruteforce packagename 'mp34piratesambience',RealPackageName 'mp34piratesambience'
Package 'mp34piratesambience' already exists as file '..\Music\mp34piratesambience.umx'


Attempting to Cache-Bruteforce packagename 'mp33piratestheme',RealPackageName 'mp33piratestheme'
Package 'mp33piratestheme' already exists as file '..\Music\mp33piratestheme.umx'


Attempting to Cache-Bruteforce packagename 'sm10itemamb',RealPackageName 'sm10itemamb'
Package 'sm10itemamb' already exists as file '..\Music\sm10itemamb.umx'


Attempting to Cache-Bruteforce packagename 'sm23ambientsilence',RealPackageName 'sm23ambientsilence'
Package 'sm23ambientsilence' already exists as file '..\Music\sm23ambientsilence.umx'


Attempting to Cache-Bruteforce packagename 'sm16kraid',RealPackageName 'sm16kraid'
Package 'sm16kraid' already exists as file '..\Music\sm16kraid.umx'


Attempting to Cache-Bruteforce packagename 'mp32piratesapproach',RealPackageName 'mp32piratesapproach'
Package 'mp32piratesapproach' already exists as file '..\Music\mp32piratesapproach.umx'

Keep in mind this is off a cache with a 100% complete cache.ini that I am not using for a lookup table, but using it as a manual reference to the automated output, I get a 1:1 match when it finds 2 matches. So it looks good.

[[]KAOS[]Casey]

Well, it looks like dependency of dependency works now.

Except that in my test-case I seem to have run across a circular dependency, and since its past midnight, I am too damned lazy to fix the bug(? I didn't do it! I blame obama) right now. MAybe tomorrow.

Code: Select all

Probably found package 'ModPlusTex' as file '47C0E08E4D458D09EB372F9F9DD042CB.uxx'
Failed loading package: Can't find file 'ModPlusTex2'

followed by:

Code: Select all

Probably found package 'ModPlusTex2' as file 'D01EC9EB4E11C1354280A880932070D6.uxx'
Failed loading package: Can't find file 'ModPlusTex'

Yeah. Whoops!!!!!!!!!!!!!

[Carbon]

Thanks for the work Casey! Some kinda magic...

[gopostal]

I have a question for you guys....In my version of the cache I still have a ton of files with no way of accessing them. I've done everything I can to uncache as much as possible. I'm still left with a lot of files. I mean a LOT a lot.

I was thinking of mass decompiling everything left into scripts, wav, textures, etc. so it can be useful at least as a 'junkpile' that devs can sift through to find interesting tidbits.

Obfuscated/protected code won't be improved and I'll remove any of those I find but otherwise do you guys see an issue with doing this? I was going to batchexport everything and then post it as distinct torrents (or dropbox if small enough). This work really needs to see the light of day somehow.

[Carbon]

Brother, I won't pretend that half of your post is proverbial Greek to me, but if you can make even some of that stuff available, I can't imagine anyone who would be against it.

The real question is, do you think it is worth your time? I for one appreciate any and all efforts within this community, even for things that I can't understand.

If and when you do have at it and for those of us living in script kiddie paradise, a BTSync share would be awesome.

[gopostal]

I got to roughly halfway last night but it will be the weekend before I finish this first pass. There's about 12,000 unique uscript classes I've extracted and GB's of textures and sounds. I've been too busy to really dig into what I've gotten (the batch scripting is very hard for me) but I sent a chunk to Bob last night. He went from "Well, I'll take a look but there's probably not a lot in there for me" to "Oh, wow...." in about 5 minutes. Yeah, there's some good stuff in it.

I'll set up a Sync this evening and start sharing what I've gotten done. Bob is working on a better extraction method to try and preserve the packages as workable/usable but that's above my pay grade. I'll just be happy to have access to the assets because those are what people will be interested in right off.

Current totals at the nearly halfway-
UC classes: 12828
Textures (bmp): 20289
Sounds (wav): 4161


More after work....

[[]KAOS[]Casey]

I am still working on the deobfuscator, but it looks like I'll have to trash some/all of my algorithms and reinvent them after analyzing some more info I get from even more reverse engineering.

Something appears to be broken, such as it thinking UTeamFix imports "Mod"... and it's a circular dependency. That "Mod" import UTeamFix and UTeamFix imports "Mod" which, unless a custom UTeamFix was made specifically for this, is simply not true. I don't see why this would be the case. I didn't bother opening it in choco package edit or utpt to verify this, as some other random zora package also supposedly did the same thing...



In the meantime, I could make a much simpler mod that just dumps every map in the cache and gives you the title/entry text, and dependencies so that some manual reverse engineering could occur. Some maps don't use custom assets at all or use common ones that maps still use today, so might be able to just magically get some working.

[BobIsUnreal]

so here is my plan ( half complete)
its just a simple bs was to attack it and is same as you but worsey.

-batch export every file.
-on export failure that has missing dependence, save missing name  and move for stage2.
-take all export and user class/asset name to build a reference table
it looks like this:
FFA002EA4A84028DFB31D3AA9F577CBF.WaveBelt
FFA002EA4A84028DFB31D3AA9F577CBF.WaveBeltEffect
FFA002EA4A84028DFB31D3AA9F577CBF.Weps
FFA047EB4FF7334626863CB7E503FF87.DarkTrooper
FFA047EB4FF7334626863CB7E503FF87.RustTrooper
FFA047EB4FF7334626863CB7E503FF87.WireTrooper
FFA7E9E311D302754000CAA7EC6B5905.MyLevel
FFAA118311D8A5BC040076AB5437865A.Solid.Blue

also sort anything with level (batchexport t3d ) into a subfolder, along with mylevel.t3d's title/auther info

^^ that parts done

stage2:
take those files you couldn't open before , provide a dummy.u package that matches name,
that run ucc batch export a second time, it will throw cant file  class X in package  B etc.
take that X and find it in the reference table , find the file and rename to B.

repeat as necessary programaticly.

here is the reference table
https://www.dropbox.com/s/diww796z83zr2 ... s.txt?dl=0

[[]KAOS[]Casey]

Just FYI, I keep brickwalling on any tools related to cache because ULinkerLoad isn't exported in headers

I've asked smirf to add exports for ULinkerLoad for 227j. The class is there, in public headers as it always has been, it's just not exported for... um.. some reason. It needs to be exported in order to actually use it.

I'm just tired of trying to reverse engineer more and more functions when I should just be able to call them.

Stay tuned...

[[]KAOS[]Casey]

I was able to download and unzip the entire thing Gex. It's up to you if you want to try again or not. I'm currently 7zipping the collection and then I'll post it as a torrent (if you want to wait). I'll post back as soon as I get done.

Edit: Here's the torrent
https://dl.dropboxusercontent.com/u/366 ... ip.torrent

I'll keep superseeding for a few days or until everyone gets on. The entire collection ultra compressed with 7z to 4.7GB. I ran through it all before compression and files looked good and intact. Thank you Carbon!

Someone's trying to download this from me at a very slow rate, but I only have 90% downloaded.

[Smirftsch]

Just FYI, I keep brickwalling on any tools related to cache because ULinkerLoad isn't exported in headers

I've asked smirf to add exports for ULinkerLoad for 227j. The class is there, in public headers as it always has been, it's just not exported for... um.. some reason. It needs to be exported in order to actually use it.

I'm just tired of trying to reverse engineer more and more functions when I should just be able to call them.

Stay tuned...

yes, sorry, not forgotten. Will come asap.

[.oldPL]

Thanks for upload, I added these cache folders to my archives.

If someone has cache files, redirects or other Unreal & UT & Tactical Ops files, please contact. Thanks