logo
Main

Forums

Downloads

Unreal-Netiquette

Donate for Oldunreal:
Donate

borderline

Links to our wiki:
Wiki

Walkthrough

Links

Tutorials

Unreal Reference

Usermaps

borderline

Contact us:
Submit News
Page Index Toggle Pages: 1 [2] 3  Send TopicPrint
Very Hot Topic (More than 25 Replies) Is there a proper fix for TArray<Byte> linkage? (Read 6863 times)
han
Global Moderator
Unreal Rendering Guru
Developer Team
*****
Offline


Oldunreal member

Posts: 565
Location: Germany
Joined: Dec 10th, 2014
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #15 - Sep 7th, 2016 at 10:00pm
Print Post  
UE1 doesn't use the STL stuff or any newer C++ additions, and you loose all integration into the UE1 framework when using std::vector. So what are in your opinion the shortcommings of TArray?  Also because it wont use templates as much you dont run into the issues msvc6 has with templates, etc.

As for the optimizations, I do a lot of benchmarking on the rendering stuff, and the first non neglectable performance improvement turned out to be for acos, but in this case the advantage is just temporary as weighting face normals is regardlesd of the compiler too expansive at runtime on the cpu, and once this ends up in a compute shader, the perfhingsorman e advantage again vanishes.

Don't get me wrong, for anything new I would  not use MSVC6, but imho for my work on UE1 games it is imho the best solution.
  

HX on Mod DB. Revision on Steam.
Back to top
 
IP Logged
 
[]KAOS[]Casey
Developer Team
Betatester
Offline


nedm

Posts: 3196
Joined: Aug 7th, 2011
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #16 - Sep 8th, 2016 at 4:13am
Print Post  
After inserted an edited UnTemplate.h provided by han, i got fewer linker errors

The code in question

Buffer is just a buffer of the whole .DDS texture as a const BYTE*

Code
Select All
Result *UTexture =  CastChecked<UTexture>(StaticConstructObject(UTexture::StaticClass(), 0, TEXT("TEMP"), 0 ) );
...
some DDS checking here
...
Result->Init(Width, Height);
Result->PostLoad();

...
more DDS checking, look for DXT1, etc.
...
new(Result->Mips)FMipmap( UBits, VBits,offset);
Result->Mips(i).Clear();
Result->Mips(i).DataArray.SetSize(offset);
appMemcpy( &Result->Mips(i).DataArray(0), Buffer, offset );
 




I can probably pastebin the whole code since this was written by smirf.

This also causes some linker errors:

Code
Select All
	UTexture* Result = ImportObject<UTexture>(T->GetOuter(),TEXT("TEMP"),0,*FoundFilename,this,0,0,GWarn);
	if (Result)
	{
		Fmt = (ETextureFormat)Result->Format;
		T->Init(Result->USize,Result->VSize);
		if(Fmt != T->Format)
		{
			T->Mips = Result->Mips;
			T->Format = Fmt;
		}
		else T->Mips = Result->Mips;
		T->Palette = Result->Palette;
		Result->Destroy();
	}
 



which is weird, because it didn't before the untemplate change...
  
Back to top
 
IP Logged
 
Masterkent
Developer Team
Offline



Posts: 1330
Location: Russia
Joined: Apr 5th, 2013
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #17 - Sep 8th, 2016 at 10:25am
Print Post  
han wrote on Sep 7th, 2016 at 10:00pm:
UE1 doesn't use the STL stuff or any newer C++ additions, and you loose all integration into the UE1 framework when using std::vector.

A part of program, that does not directly invoke UE functions, may store its data in any suitable containers and use libraries that have no any relationship to UE.

Quote:
So what are in your opinion the shortcommings of TArray?

It imposes specific requirements on data types which can be used as element types: zero-initialization of object's memory must be sufficient for its default construction, bitwise copy of object's memory must be sufficient for moving the object to a new memory location. Attempts to insert or relocate objects that do not satisfy these requirements may lead to appearance of objects with invalid internal state. std::vector is free of such issues, because it uses value-initialization (that invokes default constructor for objects having a non-trivial default ctor) instead of dumb filling memory with zero bytes for the purposes of default construction, and it uses high-level copy/move operation (that may invoke object's copy/move constructor) in order to relocate an element, so in any case an element can correctly manage its internal state. Also you have to be careful with TArray::SetSize - it may lead to memory leaks, because it does not invoke destructors when the given function is supposed to remove some elements (in case if new size is less than the initial size). IIRC, 227i UScript dynamic arrays of strings had huge memory leaks when performing array downsizing, and it's not hard to guess where that error comes from.

[]KAOS[]Casey wrote on Sep 8th, 2016 at 4:13am:
This also causes some linker errors
....
which is weird, because it didn't before the untemplate change...

Some (not mentioned) linker errors after some (not mentioned) changes in program... M-m-m, OK, good luck with fixing.
  
Back to top
 
IP Logged
 
han
Global Moderator
Unreal Rendering Guru
Developer Team
*****
Offline


Oldunreal member

Posts: 565
Location: Germany
Joined: Dec 10th, 2014
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #18 - Sep 8th, 2016 at 2:05pm
Print Post  
Masterkent wrote on Sep 8th, 2016 at 10:25am:
A part of program, that does not directly invoke UE functions, may store its data in any suitable containers and use libraries that have no any relationship to UE.

This tends to be usually not the case.

Quote:
zero-initialization of object's memory must be sufficient for its default construction

I haven't yet encountered any situation in UE1 where this wasn't the case.

Quote:
bitwise copy of object's memory must be sufficient for moving the object to a new memory location.

This isn't a huge issue, this would just be an issue when you allocate subobjects which would point back to the object inside your array, and i haven't encountered this situation in UE1.

It seems a bit of a wild mix on TArray, whether a constructor/destructor is called on insert/remove, and is probably worth to clean up a bit.

Quote:
IIRC, 227i UScript dynamic arrays of strings had huge memory leaks when performing array downsizing, and it's not hard to guess where that error comes from.

UObjects + TArray<TArray> are certainly a pain in the ass, and there are many troubles associated with it. Especially the "template" default object for classes where these arrays are not empty, the property import/export functions (e.g. if you set EditPackages defprops inside a UEditorEngine subclass, which is a TArray<FString> those will give you hell..). So there is certainly a lot of work involved to properly handle these things...

[]KAOS[]Casey wrote on Sep 8th, 2016 at 4:13am:
This also causes some linker errors
....
which is weird, because it didn't before the untemplate change...

Some (not mentioned) linker errors after some (not mentioned) changes in program... M-m-m, OK, good luck with fixing.
[/quote]
My crystal ball says you didn't remove the ENGINE_API macro on FSoundData, FMipMapBase and FMipMap... Wink
  

HX on Mod DB. Revision on Steam.
Back to top
 
IP Logged
 
[]KAOS[]Casey
Developer Team
Betatester
Offline


nedm

Posts: 3196
Joined: Aug 7th, 2011
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #19 - Sep 8th, 2016 at 4:12pm
Print Post  
Except I did remove those. And the linker errors are one of the many mentioned already here, so I thought it was unnecessary to even paste them

edit: this is the untemplate.h that I was given www.oldunreal.com/cgi-bin/nopaste/?19

Previously I just used 227i public headers untemplate.h

Pretty much just going with the default plan of VC6, because unless we can fenagle the compiler to use the symbols to match what the .lib has, no amount of crappity smacking around will fix it.


edit2: FSoundData isn't even used in my code so why would changing that be necessary?
  
Back to top
 
IP Logged
 
han
Global Moderator
Unreal Rendering Guru
Developer Team
*****
Offline


Oldunreal member

Posts: 565
Location: Germany
Joined: Dec 10th, 2014
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #20 - Sep 8th, 2016 at 6:41pm
Print Post  
You could try adding
Code
Select All
template class TArray<BYTE>;
template class TArray<TCHAR>; 


At the end of UnTemplate.h as MK suggested.
  

HX on Mod DB. Revision on Steam.
Back to top
 
IP Logged
 
Masterkent
Developer Team
Offline



Posts: 1330
Location: Russia
Joined: Apr 5th, 2013
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #21 - Sep 8th, 2016 at 9:14pm
Print Post  
han wrote on Sep 8th, 2016 at 2:05pm:
Masterkent wrote on Sep 8th, 2016 at 10:25am:
A part of program, that does not directly invoke UE functions, may store its data in any suitable containers and use libraries that have no any relationship to UE.

This tends to be usually not the case.

I have no idea what you're implementing there, but I can imagine a mod that would implement some sort of database in C++, and the only part that would really do something with UE functions/classes would be the interface of queries to that database. Implementation of the database itself can be completely independent from UE.

han wrote on Sep 8th, 2016 at 6:41pm:
You could try adding
Code
Select All
template class TArray<BYTE>;
template class TArray<TCHAR>; 


At the end of UnTemplate.h as MK suggested.

That shouldn't be necessary after removal of the dllimport specifier. The compiler should implicitly instantiate everything that needs instantiation unless you either explicitly suppress such instantiation or hide the templated definition of a function from the compiler:

Quote:
[temp.inst] / 2:

Unless a member of a class template or a member template has been explicitly instantiated or explicitly specialized, the specialization of the member is implicitly instantiated when the specialization is referenced in a context that requires the member definition to exist;

Quote:
[temp] / 6:

A function template, member function of a class template, variable template, or static data member of a class template shall be defined in every translation unit in which it is implicitly instantiated (14.7.1) unless the corresponding specialization is explicitly instantiated (14.7.2) in some translation unit; no diagnostic is required.
  
Back to top
 
IP Logged
 
[]KAOS[]Casey
Developer Team
Betatester
Offline


nedm

Posts: 3196
Joined: Aug 7th, 2011
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #22 - Sep 9th, 2016 at 12:45am
Print Post  
forward declaring the TArray<TCHAR> caused a linker error, the other appears to have no effect since TARRAY<BYTE> is used in untemplate.h

removing ENGINE_API adds linker errors to TLazyArray<unsigned char> -

The additions with ENGINE_API being removed, or the forward declaration of TArray<TCHAR> add these linker errors.

Code
Select All
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __thiscall TLazyArray<unsigned char>::TLazyArray<unsigned char>(class TLazyArray<unsigned char> const &)" (__imp_??0?$TLazyArray@E@@QAE@ABV0@@Z)
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __thiscall TLazyArray<unsigned char>::~TLazyArray<unsigned char>(void)" (__imp_??1?$TLazyArray@E@@QAE@XZ)
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __thiscall TLazyArray<unsigned char>::TLazyArray<unsigned char>(int)" (__imp_??0?$TLazyArray@E@@QAE@H@Z) 



one of my linker errors was caused by SetSize not existing in UT99 -- looks like I jumped the gun on using 227i public header originally. I fixed up UT99's original header now. All I had to do was add a few "Typename" declarations.

I've currently reduced it down to

Code
Select All
FMipmap *M = new FMipmap( 0, 0,0);
M->DataArray(0); 



where accessing DataArray(0) causes the linker error.

Code
Select All
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: unsigned char & __thiscall TArray<unsigned char>::operator()(int)" (__imp_??R?$TArray@E@@QAEAAEH@Z)
1>..\..\System\TextureMerger.dll : fatal error LNK1120: 1 unresolved externals 



I'll also need to figure out how to reimplement or workaround SetSize being missing.
  
Back to top
 
IP Logged
 
Masterkent
Developer Team
Offline



Posts: 1330
Location: Russia
Joined: Apr 5th, 2013
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #23 - Sep 9th, 2016 at 7:02am
Print Post  
[]KAOS[]Casey wrote on Sep 9th, 2016 at 12:45am:
forward declaring the TArray<TCHAR> caused a linker error

What do you mean by forward declaring the TArray<TCHAR>? The language rules forbid declarations like

Code
Select All
class TArray<TCHAR>; 


You can forward-declare the entire template

Code
Select All
template <class T>
    class TArray; 


but not its specializations.

If you're referring to explicit instantiation of TArray<TCHAR>

Code
Select All
template class TArray<TCHAR>; 


then it could make sense only in DLL that is supposed to export member functions of TArray<TCHAR> that can later might be imported by other module.

When you tell the compiler that some function shall be imported from a DLL, you obviously need to ensure that some DLL really exports such a function. Otherwise, where should the linker obtain that function from?

Since it is impossible to export the entire template, you can export only a finite set of functions that can be generated from the template. The presence of generic definition of a member function like TArray::Remove won't magically make all its possible specializations exported. You can instantiate TArray with billions of types - TArray<char>, TArray<short>, TArray<MyClass1>, TArray<MyClass2>, ..., TArray<MyClass100500>, and so on, and every such a specialization TArray<BlaBlaBla> will have its own distinct member function TArray<BlaBlaBla>::Remove. Hence we have a reasonable question: what set of functions should be exported? Obviously, the compiler needs some hint regarding what set of functions we need.

However, all these things with exporting/importing may be unnecessary (you should know better the specifics of the UE API, because you were working with it; I'm just speaking about the general theory - templates & DLLs). Then you could remove dllimport specifiers and rely on implicit instantiation of templates from headers.

Quote:
The additions with ENGINE_API being removed, or the forward declaration of TArray<TCHAR> add these linker errors.

Code
Select All
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __thiscall TLazyArray<unsigned char>::TLazyArray<unsigned char>(class TLazyArray<unsigned char> const &)" (__imp_??0?$TLazyArray@E@@QAE@ABV0@@Z)
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __thiscall TLazyArray<unsigned char>::~TLazyArray<unsigned char>(void)" (__imp_??1?$TLazyArray@E@@QAE@XZ)
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __thiscall TLazyArray<unsigned char>::TLazyArray<unsigned char>(int)" (__imp_??0?$TLazyArray@E@@QAE@H@Z) 


These messages clearly indicate that dllimport specifier is present and applied. The linker could previously omit issuance of these errors after detection of other errors.

Quote:
All I had to do was add a few "Typename" declarations.

What is a typename declaration?

Quote:
I've currently reduced it down to

Code
Select All
FMipmap *M = new FMipmap( 0, 0,0);
M->DataArray(0); 



where accessing DataArray(0) causes the linker error.

Code
Select All
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: unsigned char & __thiscall TArray<unsigned char>::operator()(int)" (__imp_??R?$TArray@E@@QAEAAEH@Z)
1>..\..\System\TextureMerger.dll : fatal error LNK1120: 1 unresolved externals 


The same history - the compiler sees that TArray<unsigned char>::operator() is odr-used, but does not instantiate the function definition from what you have in your header because of dllimport specifier. The linker fails to find the given function in any DLL and issues an error message.
  
Back to top
 
IP Logged
 
han
Global Moderator
Unreal Rendering Guru
Developer Team
*****
Offline


Oldunreal member

Posts: 565
Location: Germany
Joined: Dec 10th, 2014
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #24 - Sep 9th, 2016 at 8:07am
Print Post  
Code
Select All
Result->Mips(i).Clear();
Result->Mips(i).DataArray.SetSize(offset); 


to...
Code
Select All
Result->Mips(i).Clear();
Result->Mips(i).DataArray.Add(offset); 


  

HX on Mod DB. Revision on Steam.
Back to top
 
IP Logged
 
han
Global Moderator
Unreal Rendering Guru
Developer Team
*****
Offline


Oldunreal member

Posts: 565
Location: Germany
Joined: Dec 10th, 2014
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #25 - Sep 9th, 2016 at 8:09am
Print Post  
[]KAOS[]Casey wrote on Sep 9th, 2016 at 12:45am:
forward declaring the TArray<TCHAR> caused a linker error, the other appears to have no effect since TARRAY<BYTE> is used in untemplate.h

removing ENGINE_API adds linker errors to TLazyArray<unsigned char> -

The additions with ENGINE_API being removed, or the forward declaration of TArray<TCHAR> add these linker errors.

Code
Select All
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __thiscall TLazyArray<unsigned char>::TLazyArray<unsigned char>(class TLazyArray<unsigned char> const &)" (__imp_??0?$TLazyArray@E@@QAE@ABV0@@Z)
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __thiscall TLazyArray<unsigned char>::~TLazyArray<unsigned char>(void)" (__imp_??1?$TLazyArray@E@@QAE@XZ)
1>TextureMergerMain.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __thiscall TLazyArray<unsigned char>::TLazyArray<unsigned char>(int)" (__imp_??0?$TLazyArray@E@@QAE@H@Z) 



How about a template class TLazyArray<TCHAR>; afterwards?
  

HX on Mod DB. Revision on Steam.
Back to top
 
IP Logged
 
[]KAOS[]Casey
Developer Team
Betatester
Offline


nedm

Posts: 3196
Joined: Aug 7th, 2011
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #26 - Sep 9th, 2016 at 8:16am
Print Post  
by "typename", I mean

from
Code
Select All
TI& Add( TTypeInfo<TK>::ConstInitType InKey, TTypeInfo<TI>::ConstInitType InValue ) 



to

Code
Select All
TI& Add( typename TTypeInfo<TK>::ConstInitType InKey, typename TTypeInfo<TI>::ConstInitType InValue ) 



for a few functions

I did actually end up checking if the .lib exports TArray<TCHAR> or BYTE, and it doesn't. SO I think I understand the issue better that my compiler is trying to link to something that's not actually exported, as explained.


And yes, I mean explicit instantiation of TArray<TCHAR>
in UnTemplate.h.

So then, its pretty much just a hunt for what has the CORE/ENGINE_API macro at this point, it seems?
  
Back to top
 
IP Logged
 
Masterkent
Developer Team
Offline



Posts: 1330
Location: Russia
Joined: Apr 5th, 2013
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #27 - Sep 9th, 2016 at 12:41pm
Print Post  
[]KAOS[]Casey wrote on Sep 9th, 2016 at 8:16am:
by "typename", I mean

The reason why new VC++ issues an error message in case of the variant without the typename keyword is that it better follows the language rules that require presence of typename in such situations:

Quote:
14.6 Name resolution

When a qualified-id is intended to refer to a type that is not a member of the current instantiation (14.6.2.1) and its nested-name-specifier refers to a dependent type, it shall be prefixed by the keyword typename, forming a typename-specifier.

Quote:
1.4 Implementation compliance

— If a program contains a violation of any diagnosable rule or an occurrence of a construct described in this Standard as “conditionally-supported” when the implementation does not support that construct, a conforming implementation shall issue at least one diagnostic message.


[]KAOS[]Casey wrote on Sep 9th, 2016 at 8:16am:
And yes, I mean explicit instantiation of TArray<TCHAR>
in UnTemplate.h.

Explicit instantiation has nothing common with forward declaration except a similar syntax. The purpose of a forward declaration is to declare an entity (so it could be referenced later in the same translation unit) while the purpose of an explicit instantiation is to manually generate the definition of particular entity from some generic definition (template) parametrized with certain template arguments. In the vast majority of situations, implicit instantiation (which is performed automatically) is sufficient and explicit instantiation is not needed.

F.e., here

Code
Select All
#include <iostream>

template <class T1, class T2>
    auto min_value(T1 const &x1, T2 const &x2) ->
        decltype(x1 < x2 ? x1 : x2)
{
    return x1 < x2 ? x1 : x2;
}

int main()
{
    std::cout << min_value(3.14, 2) << std::endl;
} 


expression min_value(3.14, 2) causes implicit instantiation of min_value<double, int>, so we don't need to do anything special to make linker happy in this case.

If we had something like this

Code
Select All
// translation unit 1
#include <iostream>

template <class T>
    T square(T const &x);
// no definition of square is available in this translation unit

int main()
{
    std::cout << square(3.14) << std::endl;
} 


then some other translation unit would have to explicitly instantiate the definition of square<double>:

Code
Select All
// translation unit 2

template <class T>
    T square(T const &x)
{
    return x * x;
}

template double square(double const &); // explicit instantiation of square<double> 



[]KAOS[]Casey wrote on Sep 9th, 2016 at 8:16am:
So then, its pretty much just a hunt for what has the CORE/ENGINE_API macro at this point, it seems?

yes, for whatever that expands into __declspec(dllimport) and is used without a reason.

  
Back to top
 
IP Logged
 
han
Global Moderator
Unreal Rendering Guru
Developer Team
*****
Offline


Oldunreal member

Posts: 565
Location: Germany
Joined: Dec 10th, 2014
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #28 - Sep 14th, 2016 at 7:30pm
Print Post  
Okay, I'm back at home, if you sent me the msvc2013 stuff which caused these build errors I can test my luck with them. Smiley
  

HX on Mod DB. Revision on Steam.
Back to top
 
IP Logged
 
[]KAOS[]Casey
Developer Team
Betatester
Offline


nedm

Posts: 3196
Joined: Aug 7th, 2011
Gender: Male
Re: Is there a proper fix for TArray<Byte> linkage?
Reply #29 - Sep 15th, 2016 at 3:04am
Print Post  
Sent you the whole kit&kaboodle for it in PM.

Edit:

Note that SetSize is commented on the TArray because UT99 doesn't have it and I never got around to implementing a workaround for that yet.
  
Back to top
 
IP Logged
 
Page Index Toggle Pages: 1 [2] 3 
Send TopicPrint
Bookmarks: del.icio.us Digg Facebook Google Google+ Linked in reddit StumbleUpon Twitter Yahoo