Mythruna

Those of you who follow Mythruna on Facebook, Twitter, or G+ undoubtedly have already heard a bit about this, but I met a networking milestone last night... finally.

I was able to test full networked physics in a loop-back mode.  All of the major components are now in place and it's a matter of cleaning up and fleshing out the additional features a bit.  Overall, if the protocol doesn't have bugs it's a much tighter networking scheme than I'd even originally imagined.

The next step is to add some fake "connection quality" settings and a simple UI to control them.  This will allow me to test the protocol under conditions like higher latency, 50% packet loss, etc... and vary these settings while the app runs.  That will be the final test of the protocol.  I know it works right when no messages are lost so now I need to test it with lost messages.  It's designed to handle this but the proof is in the pudding, as they say.

To sort of understand what all of this means, I will try to walk through what happens when the physics engine moves an object.

1) Physics engine loops through all objects and adjusts velocity, rotation, position, etc. based on the current state of the objects.  New objects that are contacted are "woken up" and objects that have been still for some period of time are "put to sleep".  The physics engine only considers objects that are awake.

2) The physics engine then notifies the zone manager about all of the objects that have moved or been removed (put to sleep).

3) The zone manager figures out which zones the objects are overlapping and then delivers updates to those zones.

4) Zones collect these updates into a buffer and at the end of a "frame" moves them into a buffered history that holds some number of iterations.  The physics engine operates at a high rate, like 60 frames per second but state is only sent 20 times a second or so.  That's why these two things (the state generation and the state collection) are decoupled.

5) A separate "collector" thread, goes through all of the zones and empties their history... which generally includes three frames of "movement".  This history is then delivered to the zone listeners.

Zone listeners in this case represent the player.  Each player may be watching a different set of zones, usually the zone they are in and the immediately surrounding zones (9 zones total).  The collector thread delivers the 3-frame history of physics state to each player's listener if they are watching that zone.  (Note: remember that an object can be in multiple zones at once if it overlaps.)

6) These player-level zone listeners pack up the state for each object, disambiguating multiple zones... so if they get multiple updates for the same object from different zones then it picks one.

7) These listeners also keep track of which network messages that it knows the client has already received.  It does this so that it can avoid sending redundant information.  If the position, rotation, parent, etc. hasn't changed for some object then a very minimal data block is sent.  Still, the biggest general data block is about 14 bytes per object.  Smallest is about 3 bytes.

8 ) All data blocks for a frame are packed into a message, and may be split across multiple messages if the frame exceeds the configured MTU.  If multiple frames will fit in the same message then they are packed together.

[7 and 8 are the hardest part of the whole protocol and represent a solid two weeks of my life, I think.]

9) In the prototype, I package these data blocks up into an actual read-to-go network message but instead of sending it over the network, I pass it off to code that is pretending to be a network connection.

10) A network message handler gets the message and unpacks it.  This handler represents the real "client".

11) The client sends an ACK message back to the "Server" saying it received the message.

12) The current object delta-baseline is updated to reflect the messages that the server knows that the client has received (ACK-ACK).  Every message from the server to the client includes a list of the ACKs that it's gotten from the client.  In other words, it's sort of acknowledging the acknowledges.  It's mind-warping but necessary.  The client needs to know which baseline they agree on.

13) The deltas from the message are applied to the current objects on the client as part of a sliding window history buffer.

14) The code that handles the visuals has been looping at FPS this whole time interpolating object position between the last known value and the next known value... and now it has new values to interpolate between.

Someday maybe I will make a diagram.

The prototype started out by just strapping the object position directly to the physics engine.  The next tiny milestone was decoupling the frame timings as described in step 14.  Then I implemented zone management and everything looped through that, basically adding a zone listener that talked directly to the buffers mentioned in step 14.  (basically step 5 talked directly to step 14)

...and the most recent huge milestone, of actually looping through the network layer is a pretty big deal as that added all of the other steps.

Sending the messages over an actual network connection is only a formality at this point.  The only thing it will add to the test is unreliability... and I will be testing that in the prototype first.

Running a single process to test is much easier than launching a server, waiting for it to load, and then launching a client... every time I want to test.  Also, I can more readily test specific scenarios as I find issues.

That sounds great paul :] so will this also change how objects move when we drag them? im just wondering because of the issues i have with moving objects and lag hehe

Paul, you do so much, and we (I) appreciate the hard work you put into it. But for some java beginners, (me) and ones interested, your favor, why not stream yourself coding? there is a website called twitch.tv (http://twitch.tv/) that does streaming, but i believe you need to get a software yourself to allow streaming.

It will change.  It may or may not improve lag since it still requires a round trip... but it will be a round trip through UDP instead of TCP.  In theory it could be faster.

I still haven't worked out player dragging an object yet, though.  That's not strictly a networking problem but a physics engine problem so I haven't tackled it yet.

No.  I cannot predict when I will be productive and coding tends to alternate between note taking and change, run, change, run, change.

When I'm designing, nothing is happening... or certainly nothing that I'd want people to see. ;)  When I'm really going then I'm flipping between files and windows too fast to see what's going on without narration.  And narrating would destroy productivity.

Bottom line: It would slow me down and I already don't have enough time.

I didn't say to narrate. :P

I would just love to watch you work.
Also: twitch provides it's own software, some free for the broadcast stream.

This is what my desktop looks like:
(http://i.imgur.com/iZS6G.png)

When I'm actually working, I flip back and forth between tabs and windows so fast it's probably impossible to follow.

At any rate, I don't want people watching over my shoulder.

I can't profess to understanding this Paul but it sounds like a lot of mind bending work, well done :)

Paul do it :D

Smartiklez - i need to see an example, but I wouldn't be able to steal code if i wanted xD

This still went slightly off topic.

Yeah, but I'm not going to name names.

*facepalm

unnatural life just so you know.... twitch.tv is terrible and https://join.me/ is 100% better, iv watched a few people drawing stuff and thats cool watching them draw things and watching how they do it but that is alot diffident to coding so i get why you wouldn't want someone watching and sending messages asking what your doing and slowing your work rate down :P

glad to hear there is progress being made before the holidays, paul. good luck testing!

Wow, you just spoiled my hopes :'(

So how significant is this milestone, relative to the big scheme? It sure sounds like great progress, but I can't tell how great.

This is a major step for getting the real physics engine integrated into the engine.  It's one of the big things holding back more progress.  Code-size-wise, Physics and AI are a couple of the single biggest pieces.  And Physics is especially hard because it takes so long to re-wrap my brain around it every time I have to do something major.  So progress of any kind is pretty monumental already.

I'm really pushing hard to get the physics engine integrated.  Ships, proper water, lava, fire, falling trees, and even character animation are sort of all depending on this.  Even things like crafting and the magic system depend on how well this goes.  It's kind of a cornerstone of lots of things.


Tonight's update: (actually cut and pasted from the JME forum so may be a little more technical than even normal)

Just added some support for adjusting the up and down channel packet loss rates. I can move a slider from 0 to 100% loss and so packets are randomly dropped within the threshold. This is good because sometimes Math.random() is bursty and you’ll lose a whole bunch of packets in a row… just like in a real high-drop connection.

At 50% drop rate with a minimal amount of objects moving, the simulation is still playable. There is some jerkiness when there is a stream of drops but for the most part the protocol handles it. I will note that there is no latency built in yet… that will make things much worse because the latency will eat up most of the view-delay window that currently allows non-visible recovery for a few lost packets.

At 15% drop rate, it’s hardly noticeable at all.

At 75% drop rate, you can still play but it’s very frustrating as there is a lot of teleporting. I certainly wouldn’t want to do any combat that way. ;) It will be worse with latency.

After I add simulated latency adjustment (and requisite view delay sensing) then I may post a video.

Tomorrow I hope to add the latency tests with GUI controls for playing with different latency.  I will also add the connection stats support so that the server and client know how well a particular connection is doing (or not) hopefully with some kind of nice visual.  This is also necessary to figure out how far to lag the visuals to keep things looking relatively smooth.  Assuming I get all of that done tomorrow night then I will probably make and post a video and also post a new release of the prototype up to the donators group.

Then I could actually make a client server version for testing but after all of the above, I'll consider networking basically done... at least enough to really integrate.  So I'm not sure yet if I'll bother creating a true multiplayer version of this prototype unless I see some inconsistency in my tests that I think only a real network connection will properly test.  Though there is a part of me that wants to do it anyway so that we can shove each other around. ;)

math.random loses packets sometimes?? i'll have to remember that.

in any case, this is very good news! hope to see that video soon, and maybe start shoving you in multiplayer (don't forget the balls)

No.  I use it to determine when to drop packets in the simulator.  If Math.random() returns a value under the drop packet percentage then I drop the packet.  But Math.random() will frequently return several low values in a row, etc..

i feel like there will be an update soon :P

For various definitions of soon. :)

I'm off from work all next week for the holidays and hope to put some time into getting closer to a release.  No idea what it will contain yet but I'm going to try and redo the menuing system from scratch using my new UI library and stuff... add mythruna.com login, maybe some more formal plugin support, etc..  We'll see.

The new main-menu will also seem alot more easier to use than the current buggy and sometimes confusing one. (Not exactly buggy but un-responsive at times)

ah ok. i wondered if i read that wrong. hope the definition of "soon" is soon  :)

Network prototype as it stands now:
(http://i.imgur.com/eSMRs.jpg)

Working simulated network connection quality and real stats tracking based on the messages.  I think I will also include average message size next.

Cant wait to see how this looks in the game.

The other interesting thing is that those stats panels and controls are using my new GUI library... obviously with a different style than I'd use for Mythruna but they are functional enough to use now.  That's what I'll mostly be concentrating on over the beginning of holidays... the new menuing system, better plugin management, etc.... foundations.

its all about a good foundation.

Is that going to be during pre-alpha too?

Alpha is still six months away, Unn. Or did it change to five?

Yes.

Physics for pre-alpha please? Haha. If I can't buy Mythruna itself I'd at least like to play with physics like creating catapults. It would be the new Garry's mod. (It feels like I said that before)

I don't think you have, but you have probably thought it plentiful :)

Haha. I remember, I did. A few months ago about the physics demo video Paul made.