logo
Main

Forums

Downloads

Unreal-Netiquette

Donate for Oldunreal:
Donate

borderline

Links to our wiki:
Wiki

Walkthrough

Links

Tutorials

Unreal Reference

Usermaps

borderline

Contact us:
Submit News
Page Index Toggle Pages: [1] 2  Send TopicPrint
Hot Topic (More than 10 Replies) In serious need of some help fixing server crashes (Read 7240 times)
Tim-_-
New Member
*
Offline


Oldunreal member

Posts: 11
Joined: Jan 16th, 2013
In serious need of some help fixing server crashes
Jan 16th, 2013 at 3:29pm
Print Post  
Hello everyone!

I'd like to start off by saying thank you for all your work on 227.  The work you've done is amazing.

I used to play Unreal avidly a few years ago, pretty much every day for hours a day.  I wish I had a time machine so I could go back and do it again.

Nowadays, I've actually joined the UT99 community as its DM/CTF scene is more active.  And I'm going to go ahead and apologize in advance: this topic probably doesn't belong here in the 227 section, as my problems are specific to a mod I've been developing for UT99 the past 7 months or so.  But I'm hoping asking for help here will get maybe a few more eyes on it since I know those of you working on 227 are much more well-versed in the engine and all the different things that cause it to crash.

The mod I've been working on is essentially "newnet" for UT99.  It attempts to simulate 0 ping by tracking players' positions over the previous second and "rewinding" the server to what the shooter saw according his/her ping.  This actually works pretty well and that part of it is essentially finished.

It also simulates 0 ping with movement, so there's no more movement lag like you'd to see when dodging with a high ping with the old net code.  This also works pretty well, but I believe there's something wrong with it that causes the server to crash.  The server only crashes when players are teleporting; i.e., through portals (very rare) or when using the translocator (crashes every time in CTF).  I've honestly tried everything I could think of over the past 3 months to fix it, but nothing works.

I've actually probably read at least a dozen topics on these forums regarding similar crashes, hoping to figure out some kind of pattern; but I have yet to find any definitive cause or solution.

Here are some recent logs and some snippets of code of which I believe to be the culprit.

Quote:
NetComeGo: Open MyLevel 01/15/13 21:16:36 74.91.124.125:63513
NetComeGo: Open MyLevel 01/15/13 21:18:02 74.91.124.125:63517
NetComeGo: Close TcpipConnection100 01/15/13 21:18:36
NetComeGo: Close TcpipConnection101 01/15/13 21:20:02
Critical: FCollisionHash::ActorLineCheck
Critical: CheckWithActors
Critical: ULevel::MultiLineCheck
Critical: ULevel::MoveActor
Critical: AActor::physFalling
Critical: AActor::performPhysics
Critical: AActor::Tick
Critical: TickAllActors
Critical: ULevel::Tick
Critical: (NetMode=1)
Critical: TickLevel
Critical: UGameEngine::Tick
Critical: UpdateWorld
Critical: UServerCommandlet::Main
Exit: Executing UObject::StaticShutdownAfterError
Exit: Exiting.
Uninitialized: Name subsystem shut down


Quote:
NetComeGo: Open MyLevel 01/15/13 00:17:02 23.22.139.41:49688
NetComeGo: Close TcpipConnection61 01/15/13 00:17:17
Critical: FCollisionHash::ActorLineCheck
Critical: CheckWithActors
Critical: ULevel::MultiLineCheck
Critical: ULevel::MoveActor
Critical: APawn::physWalking
Critical: APawn::performPhysics
Critical: UObject:: ProcessEvent
Critical: (bbTMale2 CTF-Bleak-CE100.bbTMale3, Function GlobalUnrealv0_3s.bbPlayer.xxServerMove)
Critical: RemoteCall
Critical: HandleStream
Critical: UActorChannel::ReceivedBunch
Critical: (Actor bbTMale3)
Critical: UChannel::ReceivedSequencedBunch
Critical: Direct
Critical: UChannel::ReceivedRawBunch
Critical: DispatchDataToChannel
Critical: BunchData
Critical: UNetConnection::ReceivedPacket
Critical: UNetConnection::ReceivedRawPacket
Critical: UTcpNetDriver::TickDispatch
Critical: UpdatePreNet
Critical: ULevel::Tick
Critical: (NetMode=1)
Critical: TickLevel
Critical: UGameEngine::Tick
Critical: UpdateWorld
Critical: UServerCommandlet::Main
Exit: Executing UObject::StaticShutdownAfterError
Exit: Exiting.
Uninitialized: Name subsystem shut down



And here's the code that I believe to be the culprit.  I've tried a dozen different things but this is the most recent:
Code
Select All
if (!bTooLong && !bJustTeled && !bPktLoss)
		return;

	LastUpdateTime = ServerTimeStamp;

	OldBase = Base;

	if (bNewNet)
		MaxPosError = Class'UTPure'.Default.MaxPosError;
	else
		MaxPosError = 3.0;

	LocDiff = Location - ClientLoc;
	ClientErr = LocDiff Dot LocDiff;

	if (!bJustTeled && (ForceUpdateUntil > 0 || ClientErr > MaxPosError))
	{
		bForceUpdate = true;
		if (ServerTimeStamp > ForceUpdateUntil)
			ForceUpdateUntil = 0;
	}

	if (bJustTeled)
	{
		//StoreCollision();
		LocDiff = Location;
		if (SetLocation(ClientLoc))
		{
			if (Weapon.IsA('ST_Translocator'))
			{
				if ( !Region.Zone.bWaterZone )
					SetPhysics(PHYS_Falling);
				//Velocity.X = 0;
				//Velocity.Y = 0;
			}
			bForceUpdate = false;
			ForceUpdateUntil = 0;
			NN_Encroach();
		}
		else
		{
			Log("!!!!SetLocation FAILED!!!!");
			SetLocation(LocDiff);
		}

		bJustTeled = false;
	}
	else if (bForceUpdate)
	{
		bForceUpdate = zzMyState == 'Dying';

		if ( Mover(Base) != None )
			ClientLoc = Location - Base.Location;
		else
			ClientLoc = Location;
		//log("Client Error at "$TimeStamp$" is "$ClientErr$" with acceleration "$Accel$" LocDiff "$LocDiff$" Physics "$Physics);

		if (zzMyState == 'PlayerWalking')
		{
			if (Physics == PHYS_Walking)
			{
				if (Base == Level)
				{
					xxCAPWalkingWalkingLevelBase(TimeStamp, ClientLoc.X, ClientLoc.Y, ClientLoc.Z, Velocity.X, Velocity.Y, Velocity.Z);
				}
				else
					xxCAPWalkingWalking(TimeStamp, ClientLoc.X, ClientLoc.Y, ClientLoc.Z, Velocity.X, Velocity.Y, Velocity.Z, Base);
			}
			else
			{
				xxCAPWalking(TimeStamp, Physics, ClientLoc.X, ClientLoc.Y, ClientLoc.Z, Velocity.X, Velocity.Y, Velocity.Z, Base);
			}
		}
		else
		{
			if (Base == Level)
				xxCAPLevelBase(TimeStamp, zzMyState, Physics, ClientLoc.X, ClientLoc.Y, ClientLoc.Z, Velocity.X, Velocity.Y, Velocity.Z);
			else
				xxCAP(TimeStamp, zzMyState, Physics, ClientLoc.X, ClientLoc.Y, ClientLoc.Z, Velocity.X, Velocity.Y, Velocity.Z, Base);
		}

		return;

	}
	else if (ClientErr > Class'UTPure'.Default.MinPosError)
	{
		/*
		ForEach VisibleCollidingActors(class'bbPlayer', bbP)
		{
			if ( bbP != self && bbP.bBlockPlayers && VSize(bbP.Location - ClientLoc) < (bbP.CollisionRadius + CollisionRadius) * CollisionHeight )
			{
				bTooClose = true;
				break;
			}
		}
		*/
		if (Mover(Base) != None)
		{
		}
		else if (FastTrace(ClientLoc)) // || bTooClose)
		{
			MoveSmooth(ClientLoc - Location);
		}
		else
		{

			Carried = CarriedDecoration;
			OldLoc = Location;

			bCanTeleport = false;
			//StoreCollision();
			SetLocation(ClientLoc);
			bCanTeleport = true;

			if ( Carried != None )
			{
				CarriedDecoration = Carried;
				CarriedDecoration.SetLocation(ClientLoc + CarriedDecoration.Location - OldLoc);
				CarriedDecoration.SetPhysics(PHYS_None);
				CarriedDecoration.SetBase(self);
			}

		}
	}

	xxFakeCAP(TimeStamp); 

  
Back to top
 
IP Logged
 
Hyper
Board Moderator
Betatester
*****
Offline


It's Unreal.

Posts: 2833
Joined: Oct 11th, 2002
Re: In serious need of some help fixing server crashes
Reply #1 - Jan 16th, 2013 at 8:27pm
Print Post  
"FCollisionHash::ActorLineCheck" has been fixed in Unreal 227 so you might consider running your server with Unreal 1 v227 instead of UT99 if that is possible for your use case.

I don't see any developments in UT99 so I'm afraid you won't see a patch for this bug in UT.

In the past there has been quite some reports of this kind of crashes, you may want to do some searching for "FCollisionHash" and/or "FCollisionHash::ActorLineCheck" to research it more.
  

"Darkness cannot drive out darkness; only light can do that.
Hate cannot drive out hate; only love can do that."

- Martin Luther King, Jr.

http://www.hypercoop.tk
unreal://hypercoop.tk
Back to top
WWW  
IP Logged
 
Tim-_-
New Member
*
Offline


Oldunreal member

Posts: 11
Joined: Jan 16th, 2013
Re: In serious need of some help fixing server crashes
Reply #2 - Jan 17th, 2013 at 12:25am
Print Post  
Hey Hyper!  It's been a while.

Thank you for the quick reply.

I've actually googled "FCollisionHash" and/or "FCollisionHash::ActorLineCheck" a dozen times but every thread on it seems to come to a dead with no explanation on how to fix it.

Do you know if there's a way to prevent it in unrealscript or is it a core issue?  And if it's a core issue, do you know if it'd be feasible to make UT99 work on top of 227?
  
Back to top
 
IP Logged
 
Hyper
Board Moderator
Betatester
*****
Offline


It's Unreal.

Posts: 2833
Joined: Oct 11th, 2002
Re: In serious need of some help fixing server crashes
Reply #3 - Jan 17th, 2013 at 8:54am
Print Post  
As far as I know it is a core issue as it was never fully fixable with uScript mods.

You cannot fully run UT on Unreal 227 and these are not network compatible. But quite a bit of content of UT has been ported to Unreal over the years, including CTF and Botpack.

I don't know current download links.

  

"Darkness cannot drive out darkness; only light can do that.
Hate cannot drive out hate; only love can do that."

- Martin Luther King, Jr.

http://www.hypercoop.tk
unreal://hypercoop.tk
Back to top
WWW  
IP Logged
 
Tim-_-
New Member
*
Offline


Oldunreal member

Posts: 11
Joined: Jan 16th, 2013
Re: In serious need of some help fixing server crashes
Reply #4 - Jan 17th, 2013 at 3:53pm
Print Post  
Thanks again for the reply!

Would you happen to know what causes the crash?  Or if you could direct whoever fixed it in 227 to this thread, I would be eternally grateful.

Maybe if someone can tell me exactly what causes it, I can figure out some kind of workaround or optimization.  I seem to recall from reading other threads that the issue revolves around spawning too many projectiles or something?  Maybe the engine is attempting to reference something in memory that no longer exists because its already been overwritten by something else (another projectile?) because not enough memory has been allocated?

I'm nearly out of ideas here, so even the slightest nudge in the right direction would be incredibly helpful.  It would really be a shame to see the few hundred hours I've put into the mod go to waste.
  
Back to top
 
IP Logged
 
Hyper
Board Moderator
Betatester
*****
Offline


It's Unreal.

Posts: 2833
Joined: Oct 11th, 2002
Re: In serious need of some help fixing server crashes
Reply #5 - Jan 17th, 2013 at 9:25pm
Print Post  
I've had some of these crashes with my server for a long time. It was mostly related to teleporting monsters (pawns) on some maps like skytown. But I don't have nor know the tech details of the crash.

I was able to reduce the amount of crashes by eliminating monsters that teleport around back in the time, but this did not fully fix the issue. Fortunately, it has now been fixed for Unreal with u227.
  

"Darkness cannot drive out darkness; only light can do that.
Hate cannot drive out hate; only love can do that."

- Martin Luther King, Jr.

http://www.hypercoop.tk
unreal://hypercoop.tk
Back to top
WWW  
IP Logged
 
Tim-_-
New Member
*
Offline


Oldunreal member

Posts: 11
Joined: Jan 16th, 2013
Re: In serious need of some help fixing server crashes
Reply #6 - Jan 17th, 2013 at 9:46pm
Print Post  
Assuming the fix lies within a dll/so file, would anyone here be so kind as to share the change you made to fix the issue?  Maybe I can patch the same/similar dll on the UT99 servers.
  
Back to top
 
IP Logged
 
Zombie
Forum Administrator
The One Who Wanted To Have A Special Title In Forum
*****
Offline


Living Dead

Posts: 323
Joined: May 9th, 2002
Re: In serious need of some help fixing server crashes
Reply #7 - Jan 18th, 2013 at 2:13am
Print Post  
So do you confirm that code segment is the trigger of the crash by commenting out the teleporting parts? At least you have source control over this suspected code so that means if it is the trigger of the crash then you can implement a uscript workaround to avoid it.

Be safe about what conditions right before and right after SetLocation() is called for a pawn. For Unreal v225 it is sensitive when calling traces towards pawns just in the process of teleporting, and when pawns are teleporting into preoccupied or non-fitting locations. At least these are the situations I documented, and informed Hyper with. SetLocation() like Spawn() was not completely safe alone even when they returned successfully. If UT collision system still behaves much like Unreal v225 these situations might apply.

My suggestions to try:
-Disable collision before the pawn attempts to teleport, and re-enable again right after. This should also prevent chances of telefragging which could be a trigger for the crash.
-Delay calling SetPhysics() by one tick after the pawn teleports if there is tolerance, but also make sure the pawn still requires the intended physics change after the delay.
-Set the pawn physics to PHYS_None before a teleport attempt, and reset again right after to what it should be.
-When a player is about to logout (disconnect) prevent any chance of auto-teleporting since the pawn will be destroyed.

If you have source control over the gametype you can also make use of GameInfo.PlayTeleportEffect() to intercept moments when pawns are about to teleport by methods not directly in your control (mods that use the function; Teleporter actor, etc) so that you can briefly disable/re-enable collision. Plus, if Bots/ScriptedPawn are in play you can stop them from attacking a targeted player (thus stopping traces to) just as the player teleports, but as a consequence this would break their pursuit too.


-Zombie
  
Back to top
 
IP Logged
 
Tim-_-
New Member
*
Offline


Oldunreal member

Posts: 11
Joined: Jan 16th, 2013
Re: In serious need of some help fixing server crashes
Reply #8 - Jan 18th, 2013 at 3:55am
Print Post  
Hey Zombie!  Thank you so very much for the detailed reply.

I had already tried disabling collision when teleporting for a few ticks before, but I just re-implemented it to try it one more time.  I also just attempted to implement setting the physics to PHYS_None then setting it to the desired physics after a couple of ticks, but it causes a lot of warping because it kills the velocity, even when I store the velocity and reset it to the previous value when setting the desired physics.

If you have any more suggestions, please advise.

In the meantime, I'm gonna get a pickup game going to test the changes I've made so far.
  
Back to top
 
IP Logged
 
Smirftsch
Forum Administrator
*****
Online



Posts: 7392
Location: at home
Joined: Apr 30th, 1998
Gender: Male
Re: In serious need of some help fixing server crashes
Reply #9 - Jan 18th, 2013 at 7:45am
Print Post  
sorry for late reply, but having a very busy time at the moment. Some technical information about
FCollisionHash::ActorLineCheck

From what we found out it is usually caused if a pawn is placed in an invalid location, just as Zombie already said. But not invalid in place or if the pawn fits there- more that one location point returns nothing or garbage.
So interestingly an invalid location for the pawn usually causes no problem, except it returns an invalid value.
For older 227 versions I placed an is_nan check for each X Y and Z location. It is located in the Engine.dll and can't be applied standalone.
However, we noticed it happens (sometimes) to return an invalid value for example if the pawn is placed between BSP and a waterline. Also possible that BSP errors and BSP cuts could cause that. Only fact known is, that one or more of the X/Y/Z is definitely not returning a valid number, just some garbage, which can't be calculated anymore and causes the crash.
I have no idea if this could be checked via UScript - or rather, if a check for this, if implemented, wouldn't trigger another crash then.

So the fix you are seeking for is native and currently only available for 227. Newer 227 versions also have a new hashing system (octree), avoiding a lot of trouble also. Sorry that it can't be taken over easily for UT.
But maybe the information can be of help a bit.
  

Sometimes you have to lose a fight to win the war.
Back to top
WWWICQ  
IP Logged
 
Tim-_-
New Member
*
Offline


Oldunreal member

Posts: 11
Joined: Jan 16th, 2013
Re: In serious need of some help fixing server crashes
Reply #10 - Jan 18th, 2013 at 4:18pm
Print Post  
Hey Smirftsch!  Thanks for taking the time to provide some (very helpful) details on the matter!

That's strange that the engine would set some location value to something other than a number.

Does the issue only apply to pawns or would it happen to any actor?  If it's only pawns, I wonder if it would work to spawn a temporary actor of the same collision size where the pawn would be teleporting, then check its Location.X/Y/Z values, and if they're all good, we know it's okay to set the pawn's location to that point.

Even if that isn't feasible, there must be some kind of workaround in uscript that prevents the crashes.
  
Back to top
 
IP Logged
 
Smirftsch
Forum Administrator
*****
Online



Posts: 7392
Location: at home
Joined: Apr 30th, 1998
Gender: Male
Re: In serious need of some help fixing server crashes
Reply #11 - Jan 18th, 2013 at 4:47pm
Print Post  
it returns nothing or garbage if it can't determine the location. What's really causing it remains unclear, might be just some tracing issue, or some bsp problem, as said. It was very hard to track it down, unfortunately we had no way to check for this bug other than with pawns- and even that took quite some time. Wish we had your mod back then, it would have saved us hours, if not days Tongue
It is very unlikely an actor will behave any different though.

Zombie's suggestions to work around this are the first things I'd try.
  

Sometimes you have to lose a fight to win the war.
Back to top
WWWICQ  
IP Logged
 
DocHoliday
New Member
*
Offline


Oldunreal member

Posts: 20
Joined: Jun 29th, 2012
Gender: Male
Re: In serious need of some help fixing server crashes
Reply #12 - Jan 18th, 2013 at 7:10pm
Print Post  
Would the Garbage Collector be the solution?
  
Back to top
 
IP Logged
 
Smirftsch
Forum Administrator
*****
Online



Posts: 7392
Location: at home
Joined: Apr 30th, 1998
Gender: Male
Re: In serious need of some help fixing server crashes
Reply #13 - Jan 18th, 2013 at 7:58pm
Print Post  
lol, sorry if I was unclear. No, garbage collection has nothing to do with it. It's just that the value which is returned is invalid, no real number if you wish, causing the engine to crash, because you can't calculate with a "non existing" number. Was just calling it garbage, because that fits very well Smiley


  

Sometimes you have to lose a fight to win the war.
Back to top
WWWICQ  
IP Logged
 
Zombie
Forum Administrator
The One Who Wanted To Have A Special Title In Forum
*****
Offline


Living Dead

Posts: 323
Joined: May 9th, 2002
Re: In serious need of some help fixing server crashes
Reply #14 - Jan 18th, 2013 at 10:43pm
Print Post  
Tim-_- wrote on Jan 18th, 2013 at 4:18pm:
Does the issue only apply to pawns or would it happen to any actor?  If it's only pawns, I wonder if it would work to spawn a temporary actor of the same collision size where the pawn would be teleporting, then check its Location.X/Y/Z values, and if they're all good, we know it's okay to set the pawn's location to that point.

As a fact, there are monster spawner mods that implement location testing like what you describe because of this type of crash. Of one method that has been commonly relied on is a tester actor of same or slightly larger collision size that is spawned at the location first. This tester actor has all collision properties enabled by default, including 'bCollideWhenPlacing' and 'bMovable'. If the tester actor fails to spawn the location is avoided for that specific case, and if not the actor is destroyed so the pawn can then be placed.

If one wants check if location X/Y/Z values are invalid one must convert them to strings (Ex. string(Location.X) ). Within the string check for ".#IN" or anything not numerical (yes this works).

I do not have a full understanding of how your mod operates so it is difficult to know what further actions you could take. The code looks somewhat like your own version of ServerMove which I assume is used to simulate non-latent movement. For SetLocation() to possibly be called very often may be risky, and testing locations there with a tester actor may be impractical. Still, you can try every idea and safety and maybe learn something new.


-Zombie
  
Back to top
 
IP Logged
 
Page Index Toggle Pages: [1] 2 
Send TopicPrint
Bookmarks: del.icio.us Digg Facebook Google Google+ Linked in reddit StumbleUpon Twitter Yahoo