Mythruna

First, to catch folks up who might just be joining...

What seems recent but has been a couple years now, I wrote a high-performance network object synching library for Mythruna... in prototype form.  It's zone based, has internal delta compression, and so on.  All very fancy.

This past year, I got the idea to open source this library for several reasons.  It will force me to clean up the design and decouple it from some Mythruna specific things.  It will also expose its bugs/fixes/patches to a wider audience.  The clever parts of this code are just about at the limits of my cleverness which means I'm probably not able to debug it properly on my own anyway (according to conventional wisdom).  At any rate, I use it in a specific way.  Others will bend it in different ways and discover bugs under rocks I never overturned.

Everyone wins.

In that light, for the past few weeks, I've been recoding this library one bit at a time... testing what I could when I could.  It's been a long road.

This open source library will be called Sim-Ethereal.  That should catch everyone up.

Fear of Failure

I suffer from a kind of subtle fear of failure sometimes that can be insidious.  Perfectionism is one common root of procrastination and the 'fear of failure' can be one of the gateway links.  When time and motivation are already balancing on the head of a pin, any little upset can have far reaching consequences and so I try to find strategies to overcome it.

A few factors contribute to this effect for me and a particular project:
1) is this an important thing for which failure ruins a bunch of other stuff,
2) is it a job that's big enough that it's not easy to see the whole thing all at once,
3) and is it complicated enough that there will be numerous major and fundamental design crossroads for which turning back will be expensive.

Any of these on their own can be demotivational but for various reasons, in me, the combination is deadly.

Sim-Ethereal definitely falls into this category.  It's a fundamental missing building block in almost every game project idea I ever conceive of.  Furthermore, without it, a bunch of Mythruna's planned features vanish in a puff of smoke.  The job has stretched over the better part of a month and will continue to do so.  The delta compression over unreliable linkages has stretched the limits of my brain's internal state tracking and there have been several major design decisions with far-reaching ramifications.  It's got it all.

Now, first, let me remind the humble reader that a "fear of failure" is not always logical.  A networking library is not an impossible thing.  Game developers much less experienced than I am have come up with perfectly functional networking libraries.  My requirements may be kind of extreme but they are no more onerous than Valve's or any other AAA studio's... and as they say, it's not exactly rocket science.

It might be a big job for one guy with lots of life issues tugging at his fringes but it's not an impossible job.  It just isn't.  Even a crappy or partial implementation would probably be ok, even.

Anyway, this feedback loop has the potential to be deadly to a project:
-hit "yet another design issue"
-day dream a bit about possible solutions
-these thoughts lead to reminders of the ancillary requirements of the 50 other projects you have in mind for this library
-this leads to thinking about all of the "cool stuff" you'll be able to do when the project has finished... and possibly even more stuff.
-meanwhile, the design decisions has gotten no easier, the project has gotten no smaller, and no progress has been made.

At least half the time, thinking about the problem may lead to more discovered issues.  "Aw man, this assumption I made last week is wrong... isn't it?  Maybe, yes?  No.  Well, yes..."  Then I sleep on it, fitfully, dream about horrible things, and hopefully have some new ideas in the morning.

Rinse, repeat.

Wrestling with a Dilema

Truthfully, some problems just plain require worrying about them.  There might be a logical way to divide and conquer or some other way to systematically approach a solution but way way way too often, a problem just requires endlessly gnawing at it until a solution presents itself.  I can't count the number of times Mythruna-related development has worked this way.  

For days there will seem to be no way in until an idea blossoms.  It's almost never a lightening bolt.  It will be some thing I didn't notice.  Some way of looking at an issue from a different angle, or masking out some part that was distracting me, or... too many things to describe.  Slowly, the knot comes untied and the solution is so clear that it almost seems obvious.

Rational me understands this but fear-of-failure me needs constant reminding.  If it's been a while since I've experienced this, it is easy for the fear of failure to whisper in my ear "This is harder than those other problems you barely remember... there may be no solution this time... look at the calendar... how much time are you willing to waste on this approach..."

Even as a write this, there is a little nagging voice telling me that stepping away from a problem is for quitters.

Doing the Math

Many years ago, I put in a backyard playground for my kids.  It's not large, but large enough.  With slop, I calculated that I'd need something like 14 cubic yards of mulch to create the recommended depth, etc..  (I'll admit I could be remembering the numbers wrong but the rest of the story will put it in perspective if the numbers are off.)  So I found a place that would deliver playground mulch and sometime later a dump truck dropped a full load of mulch at the foot of my driveway.

A dump truck can hold a lot of mulch.  I mean a lot.  This pile was very big.  Seeing my little wheel barrow sitting next to it was depressing.

To get the mulch back to where it needed to be was lots of 'fun'.  If you imagine my lot as a big squarish area, the pile of mulch was on the opposite corner from where it needed to be... and obviously there is a big house in the way.  We'd load a wheel barrow full with snow shovels and then wheel it back over uneven ground, through the gate, around the house (and pool) and to where it needed to go: plop... then repeat the process.

Then back to the front to see a pile that looked exactly as big as it had before.

It took all day and into the evening to move that pile.  By the end, we were working from flood lights just to get the job done.

For fully half of that time, it felt like we'd never finish.  We might not have been moving a mountain of sand with a spoon but it certainly felt like a job more similar to that than anything I'd done previously.  (People who do or have done this work for a living are welcome to laugh at me... but I doubt you'll tell me it's easy work.)

Concentrating on the math was what saved me.  The irrational part of my brain only saw more mulch than it could comprehend.  It wanted to quit... with every trip.  The rational part of my brain knew how much mulch I ordered and how much a single wheel barrow would contain.  It focused on a rough estimate of the number of trips and tried to keep a rough tally of how many trips were left.  As I recall, I pegged it within +5/6 trips or so... but by then the pile was down to a size where even my irrational side has to shut up.

Conclusion

I cannot tell you how often I kept that day in my mind these past few weeks.  When life was pulling at me from five different directions and the amount of time I could spend on Sim-Ethereal was pulled apart, I kept thinking of that relatively little wheel barrow moving mulch from a pile the size of a dump truck bed.  I kept thinking if I could even spend 20 minutes on it then at least I was still keeping the issues fresh in my brain.

But mostly it was a case where rational thought won.  I can point to that.  The feeling of the size of that pile is still viscerally real to me but we moved it just the same.

Fast forward to this weekend.

Despite another two major design redirections (both of which improved the library greatly, mind you), I was able to make enough progress that I'm seeing object state messages on the client... and they have the right values.  Given the layers that information has to travel through, you'll have to trust me that it's a pretty significant milestone.  The pile is now below the fear-of-failure threshold.  Rational thought can take over.  It is entirely plausible that all major design decisions for the library have been handled (never say never but I'm optimistic).

There is still plenty to do but now it's in the realm of just hooking it all up.  It may take another week before I have it fully integrated into the E-space test game... depending on day-job obligations.

It may be more than a month later than I wanted to be right now... but the fear-of-failure can suck it.  Finishing late is still finishing.

Procrastination aside, I think your 'subtle' fear has helped you create things that may one day rival will surpass AAA titles and content, if only in their quality. I remember one of the biggest hurdles for 1.8 Minecraft was the fact that they had to recode the entire engine because it was so... bad. The ended up having to change over 1,000,000 lines of code because they did a sloppy job at first. The way I see it, after you have your way with something, it's platinum. Even if you overlook something, your content still rivals puts to shame what the millionaire studios are dishing out.

Thinking about that pile of mulch made me want to have irrational thoughts about ever finishing, too..

I remember that Rayblon, and that makes me think that once I learn what I'm learning I'll end up rewriting my whole project (or bit by bit right now)..

I've never had this issue. I just mentally groan at how many months a project may take.

Funny thing is, most of the projects I've started will never be finished.

Yes, and that's what the 'fear of failure' will be constantly reminding you: "This is just another one of 'those' projects.  It will be among good company with all of the other cool stuff you never finished..."

I still have faith in Mythruna Paul, and I will continue to wait for it for years to come(Give or take a few periods of inactivity on the forums).

None of my Mythruna projects have been abandoned, though. :>

(http://th09.deviantart.net/fs71/PRE/f/2011/299/4/3/rage_face_by_rober_raik-d4e0fxk.png)

NOBODY CAN TAKE A BREAK FROM THE CULT OF EVIL LAUGHTER.

http://blog.roblox.com/2015/06/create-all-new-worlds-with-smooth-terrain/ (http://blog.roblox.com/2015/06/create-all-new-worlds-with-smooth-terrain/)

Normally I wouldn't give any praise to this company but... Oh man. Bravo. They did the impossible and it looks freaking gorgeous.

Kind of like what you will do.

Last night I had a little time after finishing "day job" stuff (more like giving up on a 'day job' problem for the day) and so looked at hooking  up the messages needed to get the delta compression working.

The old function/method that updated the object state baseline was a couple pages long.  So I was preparing myself for the conversion to take some time... but because the new design is nicer and the lower level parts were already implemented well (and better organized), the new code was only 4-5 lines.  Basically: "loop over objects, call their updateBaseline method".

Took more time to make sure that the server was receiving the acknowledge messages from the client than it did to hook them up... and  hooking them up automatically hooked up the client, too.

It's nice to have an easy win and makes me feel better about the design at the same time.  Much better than the previous night where I spent an hour tracking down an issue that was a type in deeply low level code... a cut-paste error where a 'y' was an 'x'.  Oops.  Those kinds of nights undermine confidence.  Nice to have a good one to balance it.

This week I've done my best to try to chip away at this a little a day.  Work has been pretty crazy but I've tried to keep to a rule of working on ethereal at least every night even if it's only for 15 minutes.  Usually it amounts to more... sometimes it's just a matter of getting started.

So tonight I finally have my space ship flying around over the network and I was able to log two clients in and see.  Yay.  There are lots of bugs, of course (boo) but that's to be somewhat expected at this stage.

From the client on the same machine as the server, everything is super smooth.  There are some glitches with a delay in object removal causing the lasers to pause at the end of their fly time... but I kind of expected those sorts of issues.

Also, on the client that is not local to the server, the movement is very jerky much of the time.  And it's not just that ship's movement but all movement so I have some things to look into there.  It could just be that my history window is not large enough.

The more trouble issue is that I occasionally see objects in the wrong zone.  So there is some issue with zone homing and/or overlapping zones or something.  In one case, it didn't fix itself until I flew away and back again.

On the plus side, these are the sorts of issues that are "real" and I knew that I'd be debugging as soon as I got the library to this point.  So it's still part of the progress... I was just supposed to be here a month and a half ago.

Oh well, progress is progress.

When do you think a playable release of Ethreal Space will be ready?  8)

I have no idea.  Depends on how long it takes for me to track the bugs and stuff.

I've been taking my time tracking down the zone bug because I feel like building out the infrastructure for tracking down problems is apropos at this point.  Networking bugs are notoriously hard to find so whatever tools I can give myself are gold nuggets.  In fact, just creating the debug layers has already pointed to one problem I thought I had (and turned out to be something else entirely).

Here is a picture of the layers I've added so far:
(http://i.imgur.com/rMtWVn4.png)

In the upper corner, we see this player's current location and zone. (The font's Z is kind of funny... but whatever.)  On screen, we see the local zones around the player.  These are fixed in space and there is always that 3x3 grid around the player.  As we cross one border, it adjusts to always keep a one zone border around us.  My zones are too small for the real game but good for debugging... but that's why I thought the lasers had a problem pausing in space... once an object leaves the watched zones it just freezes.  No more updates.

Over each mobile object with some 'name' text, I also get its name and its current zone.  This will be useful for tracking down the cases where I was seeing an object appear where it shouldn't be... because that indicates that the network layer is giving me the wrong zone.

Right now, this zone is based on the position is the client detects it... which on its own won't tell me anything, really.  Next step will be adding some additional debug info that includes the zone as the server sees it.  Then we'll be getting somewhere.  Not only can I then display both, but I can indicate errors when they are different and stuff.  I can even add information from each stage of the process: zone as the server sees it, zone as the network client received it, actual displayed zone based on client-side world position.

The other plus side is as I've been thinking about this, I think I might have some clues as to the issue.  So letting is simmer hasn't been bad either.

Whatever I can do to find a way to test this without logging two clients in would be best.  And that's what I'm trying to get to... narrow the problem down to the point where I can cause it with a moving object that isn't controller from another computer... because testing with two clients takes way longer. 

I've even considered adding some simple AI ships just to fly some patterns to test various theories... and I might still do it.  That has double-benefits.  But if I can solve this zone problem without it then I will.

So, after adding all of this cool debug info... turns out that the zone 'bug' I saw before might not have really been a bug at all but my misinterpreting something else I'd left out.

...basically the same problem as the lasers 'pausing' except in this case the objects were still live.  They'd gone outside of my normal zone view but didn't disappear.  So I was just seeing the other ship hanging out in space somewhere but it wasn't the real ship.  The real ship had flown to a zone I couldn't see. 

Once I'd started adding the zone information as the server saw it, I could easily see that the object was still moving around, just somewhere else.  So the debug layers really did end up saving me a lot of trouble because it turns out I'd completely misunderstood what I was seeing.

I just need to make objects disappear once they are 'out of view'... ie: once we stop getting updates about them.

On the plus side, I flew around following an asteroid for quite a while without any major issues.  There were a couple places where it jumped because of network hiccups and I noticed that it's bounding radius is offset strangely but the rest is fairly encouraging.

Maybe by this weekend I'll be ready to see how this runs over the real internet.  (bites nails nervously)

Going from running your server on localhost onto a separate machine is like your kid learning to drive a car...
You'll fear fear itself.

Heheh... yeah, well, I've already tested it from multiple machines on my LAN.  Opening it up to the world is like sending your kid out on his/her own for the first time in a car.  :)

Well, I think I've gotten the absolute minimum done to allow a test release to be posted for espace.  I'll probably do it Sunday afternoon to see what additional polish and stuff I can shove in before then.  Maybe at least count hits against other players or something.

There is some wonkiness with the object creation/removal timing that appears every time you fire a plasma cannon so I want to see if I can deal with that.  That's officially in my "bug" list.  If that goes well, maybe I'll get hit counts in too.

I finally setup the second monitor again and cleaned up the espace code to the point I was ready to release something.

I've got the server running on the same server that Mythruna does now... and the second monitor means I can also keep the corner of my eye on it occasionally.

The issue is that for some reason the client is getting some timing issues.  I haven't had much time to track it down yet, mostly been adding more debug information.  I fixed one issue but now a different one has cropped up.  re: the first issue, it's actually a good thing because I'd seen that once or twice before even in local testing but not consistently enough to find and fix.  It was nice that it consistently happens against a "real" server.  Same with this second problem: it consistently happens.

Hopefully I can fix it soon and then post a client download.  Sorry for the delay.

There is no need to apologize for releasing free stuff late. :P

Heheh... sure... but if I'd tried it with a real server on Saturday instead of late last night then maybe we wouldn't be having this problem and logging would just be messier (what I worked on on Saturday).

On the other hand, logging is great to have now. :)

These issues have been taking me longer to track down than I wanted.  First because we lost power here a few days after I had to stop working on it and the computers all went down before I could get the generator up.  So all of the logs I had open to the "exact right spot", files open, etc... were all closed and I had to start over.

Add to that: my day job was grinding me down plus "last week of school" stuff for the kids and I only this past weekend got back to it in earnest... and I've been dreading it.

Fortunately, I've been able to identify at least one major glitch causing jerkiness and fixed it.  The nice thing was I was able to see it happening when the ship was supposed to be sitting still right after the game started.  Seeing the ship jerk around there when the position was supposed to be 100% knowable made tracking the issue soooo much easier.  As with most things like this, it would have been nearly impossible to find otherwise.

Some jerkiness was caused by the camera "spring" acting funny... I knew this before and so I've turned it off.  (I'd even get some little skips and strangeness in single player.)  Some more time related tweaks were made so that the camera 100% follows the ship in single player as there were still some hiccups.

All in all, I've fixed about half a dozen bugs.  The networking is much smoother now, though still not perfect... and it's a little laggy now.  (That may be because of all the logging I currently have running and it may be because I set the lag delay high to make sure I wasn't trying to interpolate data that was too fresh... never a good sign when one resorts to just 'randomly trying crap' until something sticks so I still have a bunch to undo there.)

Every now and then I still see the ship jump to an odd location.  I will have to come up with a way to detect that in code such that I can dump some data to figure out why it happens.

Getting closer, though.

I may have to take a break and work on some unrelated Mythruna stuff because frankly I'm jonesing to work on that code again... got some stuff I want to try.

Once again I spent several days tracking down an issue that turned out not to be networking related.

Once I'd gotten rid of the jumping ship issues I was noticing that the ship appeared to speed up and slow down even when velocity should be constant.  It was quite jarring.

Ultimately, I even put in some tracking at the object level just to show me the calculated velocity of each object and how much it varies... it varied _widely_.  No surprise then that it showed up visually.

After tracking back, it turned out to be a time source issue on the server.  Physics frames are supposed to come consistently every 16 milliseconds (60 frames per second).  As an architectural/design decision, I've set the physics engine up as a fixed step 16 ms no matter what.  (It improves physics accuracy overall to not vary and given the relatively low requirements of this game it should have been no problem.)  However, the thread that runs the game loop was varying its frames widely.  Sometimes as much as 48 or 50 ms between frames.  There was really no reason for this other than that the thread is relatively low priority and running on the same system as the Mythruna server.

I tightened up the timing loop to not vary so much and the problem went away on the client.  So good news again that another problem turns out to not be networking... it was sending along exactly the crappy information I was giving it on the server.



Yesterday was kind of a milestone because it's been exactly three years since a public Mythruna release (20120627).  Not really a happy day for me... I'd planned to go into this weekend full bore getting this networking thing finalized so I could move on.  Unfortunately, life had other plans.  As if to remind why I'm here and how foolish it is for me to make plans in advance, my wife fell on Friday and tore some ligaments in her foot.  We spent 4-5 hours doing the emergency room + prescription pickup thing with two kids in tow... good times.  This week she will go do see a bone-doctor to get a real cast and stuff.  She can already barely walk as it is so this essentially puts her in a wheelchair for the foreseeable future.  So this weekend was co-opted for other purposes: mainly putting in a wheel chair ramp for our front door.

:( Good god. I have nothing to say besides how much I want to help. Or be there.
Family isn't family without effort, and as saddening as it is to see all of this, I hope you all stay safe and pull through... And I hope your wife recovers faster than it would take.
You and your family, stay awesome. And make sure to place the rest of us behind family ;)

And sorry I've been gone for so long. Life has other plans for everyone, and it's daunting and horrible!

I thought about E-space and got all excited and thought I'd tell you guys. It'gonna be so cool when it's finished! :D

:) So many things are going to be so cool when finished...

Just gonna put it out there... I wasn't hyped for MGS V and, while impressed, wasn't terribly interested after seeing it.
... And I'm not hyped about FO4, particularly.
HL3 and L4D3 has medium hype.

But your games are... I just can't help but to be excited because I know you'll take the idea and make the best damn thing possible out of it. Not just something that works well and looks pretty, but something that's innovative in one way or another.

Yeah, I'm excited for my games, too... but I am also really excited about FO4... I plan to take a few days off work maybe.  (Note: planning and doing are not the same thing but it's fun to fantasize. :))

I just played some of the MGS5 one level pre-thing that was relesed some months back.  It's definitely my kind of game as I really loved MGS4... not quite the same vibe there.  Hard to say if that's because it's essentially a tech demo for the real game or not.  I'm definitely experiencing a bit of 'stealth game' fatigue though as a month ago I finished the game Styx over a many week period.  There is much that is similar about these games except now I can kill people from a distance.  MGS tends to put a lot of nice detail in.  Saw a video of a guy who accidentally snipped a power line and it actually broke and stuff.

For the record, there is a no-mans land in my day where my brain is too dampened to do any real work but I'm also not sleepy enough for a nap... right about dinner time.  It's a nice time to drop into a game for an hour or so.  Lately I've been digging a giant pit in Minecraft as a monument to a place I died.  Instead of rage quitting, I'm turning the whole cave system into a giant crater.  I should post a pic sometime.  It's the perfect level of mindless activity to round out my day and give me solid thinking time. ;)