Elf Clan Social Network

    One of claims I hear from time to time... both from users and server companies... is that prims cause "lag".   The more prims you have, the more lag they create.  There is also the concept of prims taking more server memory.

    While prims do take up a bit of RAM*, the claim that prim count causes lag is a myth.    We know this because we tested this claim years ago, on Second Life and put an end to that myth forever-- or so we had hoped.  Some monsters just won't die, even when you cut off their heads.

 

CORRELATION is not CAUSATION

    Now to be fair, there is a correlation between prim count and lag.   It would appear that sims with low prim count have little lag, whereas sims with high prim count have a lot of lag. 

    But as statisticians and math majors know, correlation does not equal causation.   Quite often correlation is just a symptom, whereas the disease is something else entirely.  This is the case with prim count.

   

THE ULTIMATE TEST

    About a decade ago when the "Lag Monster" hit SL seemingly out of nowhere, we techs started running lag tests.  What was the source of the lag?   Was it "user content" as Linden Lab claimed?  Was it "too many prims"?   Was it "too many textures"?  Too many scripts?  If it was any of those things, then why had sims with identical setup worked just fine a week before and now they weren't?     Users wanted answers.  (The answer was that LL had made some major server changes without informing their customers... but that's another story entirely.)

    A friend had just bought a new region and before building anything on it, invited me to spend a day running whatever lag tests I wanted to run.   I accepted his invitation.

    The tests were simple:  we would rez prims in various quantities and configurations and run metered "lag tests" checked both by scripts and avatar experience (mainly, turning in place, walking around and flying and seeing if we noticed any difference at all).   We had half a dozen people present serving as observers to verify experience.  

    First we walked and flew around an empty region.   We checked the sim stats, recorded sim stats, got a baseline to check against.

    Second, we rezzed 1,000 cubes and placed them all around us.   Ran the same tests.   No change in baselines.

    So we duplicated that 1,000 cubes until they numbered 5,000.   Ran the tests again.   No change in baselines.

    Linden Lab allows 15k prims on a sim so we couldn't go past that.  To be totally fair and not push the server to max, we rezzed 12,000 cubes and spread them around.   Ran the tests again.   All observers reported the same thing:   Absolutely no change in baseline stats or avatar experience. 

    Twelve thousand prims-- zero server or client impact.

    Okay, there was the "prims cause lag" myth blown out the window.

    But to be fair, let's start from scratch and do it differently.   So we rezzed cubes, spheres, cones and cylinders to see if prim shape made any difference.   Ran the tests.   No change in baselines.

    Several years later an Opensim team decided to test prim lag on their system.  This was a good test to run since they were trying to stabilize their platform by eliminating as many causes of server lag as possible.   They rezzed 140,000 prims at ground level.   Result:   Zero discernible lag.   This validated all the tests we'd run years earlier, but on a factor almost 12x greater than what we'd run.

SCRIPTS.  IT MUST BE SCRIPTS

    Okay so if it's not prims, what would cause lag?  Scripts, right?  Linden Lab told us:   the more scripts, the greater the lag.   That made sense.   But there are two kinds of scripts:  active and inactive.   Active scripts are constantly changing things, doing something.   Inactive scripts just sit, waiting to do something (sit / touch scripts) or have already done something (texture animation scripts).   Some inactive scripts can be run once and then removed-- with their effect still on the prim.   For example, you can animate a texture and then remove the script.  The texture will continue to animate.  A non-animated sit-scripted prim, once the script is run and removed, will still continue to seat avatars properly.

    But Linden Lab claimed that all scripts cause lag, because the sim server had to check every script every cycle to see what was going on.   Okay, let's test that out.   We removed 5,000 of the prims and created a prim that contained a touch-based sit script (inactive).   Then we replicated that prim to a count of 5,000.    Now we had 5,000 prims with trigger-happy touch-scripts in them, just waiting for someone to click them... an amount the company claimed was "far too many scripts on a sim".

    You have probably already guessed the outcome:   zero change in sim baselines.  The entire region still ran as if it were totally empty.   How about that.  Inactive scripts have zero perceptible server or viewer impact.  No lag.

 

OKAY, SO WHAT WAS CAUSING THE LAG?

    If prim count, prim type and simple script count wasn't causing lag, what was?    It turned out the primary lag issue was due to internal SL server issues-- which we discovered by a very extensive Elf Clan experiment that revealed LL was stacking full regions on single cores "by accident".    We asked a dozen people to check all 800 regions on SL (the sim count at the time) and found out that "accident" happened a LOT... and was the primary source of region lag at the time.  It has been so ever since.

    However in case of modern day lag issues, with companies that do not pull such shenanigans (at least, not without properly informing the customer, such as with Inworldz legit 2x2 regions)... lag is caused by several identifiable issues, depending on the grid involved.

    Active scripts can indeed cause lag to an extreme amount.  In fact on SL it was possible for one badly-designed script to completely crash a sim.   Inworldz took care of this by allowing scripts only so much "server room".  While script response might lag if the scripts are poorly written-- they would be unlikely to  impact the region server itself.  That was a smart move for Inworldz.

    On all grids a major problem is texture handling.   The more textures there are, the more textures the client has to load when entering a region.   Add to that poor texture processing where many grids load the same textures over and over again.   This is the result of a badly-designed cache system (the software that stores textures on your hard drive and in RAM for instant access) and buggy loaded-texture tracking.   That is a major issue, because this is in part server but mostly viewer based-- and grids tend to use the same viewers.  (I am not sure at this time how the official Inworldz viewer handles textures; I haven't tested it.  I have tested Firestorm and the problem is still present in that viewer.)

    Avatars cause incredible lag.  Take every factor in the book (textures, prims, active scripts in AO devices) and put them all on a moving, multi-jointed avatar, and you are going to have server and client impact.   The more avatars there are, the more lag.  Pretty much everyone is aware of this. No huge surprise there.

    Arguably (depending on the surroundings), Avatars are the single most laggy factor on any grid.  You can take a nice, peaceful, content-rich, fast-performing region and drag it down significantly by adding 20 avatars.  Add 50 to 60 avatars and the region can go to borderline crash-status.  Yup, avatars lag.  We all know this.

    Server issues.    From time to time there are server-related issues that cause lag; the best thing to do about that is to reset your region daily, or at least once a week.  Server software can get confused, start dragging its heels and needs refreshed from time to time.   This is simply nature of the beast.

    But some server issues could be fixed-- teleporting for example.   On OpenSim grids, an avatar teleporting into a region can bring the entire region to a standstill until the avatar has finished porting.  You can imagine the impact this has on a busy nexus.

    Flexys cause considerable lag.   A classic test was when we examined three types of avatars:

    1. A huge armored avatar consisting of a solid 1,500 prims.

    2. A standard female avatar with flexi hair and dress.

    3. A dwagon, 401 prims.

    Checks were done to see the impact each of these avatars had on a standard sim.  Surprisingly the dwagon caused least impact, with an avatar "lag factor" of 1.0 (by our scale, starting on a scale of 0, a plain avatar with no attachments).   Second place was the armored avatar, which caused a lag factor of 2.5.   (No surprise there... direct prims-on-a-moving-avatar correlation).    But the big surprise was the standard human avatar with flexi hair and dress.  Nothing special, just standard attire.   A whopping lag factor of 6.5.  That one avatar took double the system resources of the two other avatars combined. 

    Try telling a paying customer they can't have flexi hair or clothing and see how far you get.   ; )

    In another test we went to a region that had several "content tests" already set up (very interesting region, that).   One of the most interesting tests was their "flexi ring".   They had twelve flexi "blankets" all hung out in a ring.  When any blanket was clicked they all either turned not-flexi (standard prims) or flexi (standard prims with flex), blowing in the wind.  The result was astounding.  When the flexi on those twelve items was turned off, the region ran fine.  When the flexi was turned on the region lagged significantly.  Walking became difficult.  Turning in spot became more difficult.  Turn off the flexi blankets, stability restored, no lag.

    So if you have a region that seems to lag and there are lots of flexi tree branches and flora and flags and other items... turn off the flexis.  They are seldom worth the incredible impact they have on a region.

 

LIMITATION OF PRIMS ON A REGION

    If you ever set up an Opensim server you will discover something interesting:   in the instructions they recommend setting maximum prim count to 200,000 per region.  What?  Are they insane?  

    Well no.  As we've seen simple prim count has little if any impact on the server or client.   You can set up 45,000 prims on a region (as on Inworldz) and still have plenty of server RAM left for sim performance.  You can set the prim count to 200,000 and so long as you keep scripting and textures to a minimum, no problem.

    Bottom line:  the number of prims allowed on a server is pretty much irrelevant.  What is relevant is what you do with them.

    The reason smart grids like Inworldz limit prim count is because they know there is a correlation between prim count and server performance.   Why?  Because the more prims people use, the more active scripts they use, they more textures they use, the more flexis they use.   So limiting prim count is a sensible way to keep things within server stress limits.   45K seems a "happy medium"... allowing far more prims than the sim owner is likely to need while at the same time keeping away from the edge of the cliff.

 

BUILD SMART, BUILD WISE

    If you visit ElvenSong region on Inworldz, you will find a prim-rich environment.   You will also notice (once the sim loads), there is very little "lag".   This is especially the case if you go into high sky and visit Replicant City.   This prim-heavy, script-heavy, texture-heavy area manifests hardly any  lag at all, if any (once you give it time to load).  Why is this?   How is it such a creation can have almost no lag?

    1.  Smart scripting.   The scripts are written to be inactive where possible.   The fewer active scripts you have, the lower the sim impact.   Replicant City is highly interactive and almost everything is scripted, but the majority of scripts have to be triggered either by touch or contact.   Sensor scripts are very limited.   Active scripts are very limited.  There are no scripted bots.  (Scripted bots can be a major source of lag if not used sensibly.)

    2. Smart texturing.   There are thousands of textures at Replicant City... but they're all hidden from simultaneous view.   They're down corridors and around corners and inside closed buildings.  To "see" them you have to enter the area where the textures are.   So for the most part you're never loading or seeing more than about 200 textures at any one time.   This greatly limits texture impact on visitor experience.

    3. Almost no flexis.  Flexis were limited to an absolute minimum.  Almost no flexis = almost no flexi lag.  Simple fix.

 

    That's the skinny on lag and prim count.   Prim count does not cause region lag.  Zero prims or 140,000 prims...  they're not a source of "lag".

     Watch your scripting (use active scripts as little as possible-- especially scripted bots), how you display textures, and above all watch how many flexis you use.   Keep these down and you'll significantly reduce or even eliminate lag on your region.

--o--

* SERVER RAM.  While prim count does require server memory, most server systems are set up so that they have more than enough RAM to run a server, regardless of prim count.   A 2-gig server (standard size) can handle an entire region full of prims and still have lots of room left over.   In fact it can handle multiple regions-- to greater or less effect depending on the number of regions.    So any grid company claiming that they can only offer 15K prims because of server limitations is full of baloney-- or cutting their server costs so low they're really not giving their customers what they're paying for.  

    Inworldz offers 45k prims-- and could easily provide more except for the prim-count / lag correlation (along with prim count tends to come scripts and textures and flexis etc-- items that can cause severe lag).   45K prims is a reasonable and customer-friendly prim allowance.

Views: 255

Comment by Yichard Muni on July 20, 2016 at 6:25am

What people call lag is a complex thing which can manifest in several way and have different causes and remedies:

-Low frame rate. Since frame rate depends on the computer, we need to compare with for instance an empty sim, or a scene at high altitude. Things known to bring lower frame rate is of course a lot of content (high land impact). But there are specific items which are very bad, such as a lot of alpha textures (trees, rain, hair, etc.) This is one of the reasons why creaors are more and more creating mesh plants which do not use alphas.

-latency when clicking on a key, moving, etc. This latency appears automatically when the frame rate is very low. But the most usual cause is the internet connexion. In other worlds than inworldz, it may be a bas derver or bad hosting company. Some specific cause are objects which send a lot of updates on the network, for instance objects containing hundreds of scripts, or carelessly designed schripts.

-long rezzing time (things remaining grey) is caused by a bad connexion, which slows down the download of data. Exceesively large or numerous textures also cause this problem. Since sculpties are defined by textures, they start appearing as huge blobs. In severe cases, meshes start to appear as lower LOD.

Comment by Yichard Muni on July 20, 2016 at 6:26am

-chat lag is caused by a bad server, or sometimes bad connexion

Comment by Wayfinder Wishbringer on July 20, 2016 at 8:54am

Siwan:  the prim count of the dwagon was stated:  401 prims.

Your question about how the avatar with hair and dress was configured is valid.  But I don't have the answers.  The test was years ago and I've slept since then.   The only reason I remember the dwagon and armored avatar is because they were mine and I used them quite often (still do).  However I remember the volunteer avatar wasn't dressed to the nines; she just wore one of her "standard avatars" with hair and a basic flexi dress.   Remember I stated in the examples that twelve flexi blankets lagged all who were nearby.  That's a pretty low number.

Regarding "lag factor" there are many ways to measure such.  In this case we used the standard ARC method (which has been questioned as to accuracy, but its measurements matched our "perception" values).   To establish the factor we took the lowest denominator of four:   an avatar with no attachments.   Arc rating:  0 (the standard of which LL apparently set for a basic avatar).   The dwagon rated about 1500, the armored avatar about 2500, the dress and hair about 6500. 

 

It's not a matter of whether the avatar was female or male; anyone focusing on non-existant pixel gender issues is missing the point.   The point is flexis... as well as the point being standard avatar.   Flexi hair and flexi dresses are common among female avatars.   Not so common among male unless we're wearing robes and Elven-type hair (but even then, for some reason males tend to go with just basic hair rather than flexi).   Flexi dresses are almost essential for appearance.  Non-flexi hair may look a bit static (especially if it's long hair)... but the impact of flexis on a region are considerable.   So imagine 20 avatars at a party... all of which are wearing flexi hair in addition to their flexi dresses.  That's why at big events they ask people to strip down to basic prim hair and non-flexi clothing-- so they can get more people on the region.

It's a love-hate relationship.   On the one hand flexis look great and in some situations (like dresses or robes or capes) they're almost essential.   But on the other hand, very little lags worse than flexis.

 

And you're right... I try to keep my textures at 512x512 or below.  However... if a 1024x1024 texture crashes someone's computer, something is set wrong in their viewer.  You load textures far larger on a regular basis when you visit standard websites.   If you view a video in 720 or 1080p, you're loading hi-res textures constantly... at 24 frames per second or higher.  So if a simple 1024 texture is crashing your system in VR... need to go over viewer and computer settings with a fine tooth comb.

Mind you, such issues can be very hard to find.  Thus the bone I keep picking with these viewers.   All bells and whistles but failure to fix the core issues for more than a decade.

Comment by Wayfinder Wishbringer on July 20, 2016 at 9:09am

Yichard:  in our findings-- despite LL and tech claims to the contrary-- bad connections (while possible) are seldom the issue with region lag.   If someone has connections that bad, all of their Internet would be lagging (with the exception of a single bad connection between them and the VR server).    At time yes, bad connections can seriously effect performance.  But one can go to a site (Zauber has the address I don't) which shows general Internet health and trouble spots and double-check.   Each person can also check their ping times to examine issues.

 

Even with a relatively slow connection, VR should still be able to provide all the information necessary to prevent lag.   Aside from textures all VR is sending out is data (well, yes, textures are data... but a lot of data).  It was discovered long ago that one of LL's major problems with textures was 1) They were using lossy UDP transfer methods where lossy UDP was unacceptable procedure) and 2) They were (and still are) loading the same textures over and over again.  

The truth is:  if VR server, asset and viewer code was written as it should have been... there would have been very little lag at all in these worlds.   That fact was proved a long time ago.   Tranq further verified this when he climbed into OpenSim and SL Viewer code and wound up having to totally re-write major chunks of code because they were simple very badly written.  

 

Evidently when SL was founded they didn't count on growth, so didn't consider or write in the ability to grow.  The asset servers got bogged down, sim servers get bogged down the more avatars are present, and in the old days hard drives got bogged down as several sim servers accessed them constantly (causing one region to interfere with another.   Heaven help you if you were on the same server as a club). 

 

Bottom line:  neither Philip nor Cory Linden's vision foresaw the phenomenal growth spurt of SL... and so they didn't write such expansion into the code.   Bad coding.  Unfortunately when OpenSim was written they used the open-source SL viewer as their guide and repeated the same mistakes all over again. 

 

So while net performance can, in some instances, affect user experience, such is usually not the case.  If there is "lag", it is almost always due to asset server performance, region server performance, or viewer performance.   The percentage of "Net-related" issues is very low compared to in-house, just-plain-bad-code issues.    Those issues are what Inworldz tackled when they first started-- and is part of what sets Inworldz apart from OpenSim.   They had to re-write major chunks of bad server code to function even halfway properly.

Comment by Balpien Hammerer on July 29, 2016 at 11:24am

One can get into a lot of tech to get exact details on what lowers performance but it boils down to three basic pain points: things that burn time in the simulators, things that burn time in the viewer, and things that burn bandwidth.

This is important to know because each of those pain points are very different parts and they will be felt differently if at all depending on what people have. Someone with a medium power relatively modern computer will not feel flexi lag at all unless they are in a faerie forest of thousands of wiggling faerie leaves :) However, anyone with an older computer will experience lag from flexi prims since their graphics capabilities are limited. The viewers do not fare well under flexi and large particle loads (more on this later).

Also things have changed in SL and IW.

Scripts no longer bog down the simulators. They used to but once SL lowered their priority relative to other things, problem gone. InWorldz used to suck raw eggs previous to phlox. It used the OpenSim script engine, but in 2011 phlox created a far smarter engine, one in which inactive scripts were utterly inactive. Scripts run in their own thread. The thread is lower priority than other stuff, so like SL no more problems, but better, because servers are multi-core nowadays, it means the scripts run in parallel with the other stuff, lead to very efficient, very high performance.

Scripts used to burn a little time in SL even inactive. They fixed that when they introduced mono. They had a nasty problem when scripts were created (when an object rezzed). I could with one special object bring an entire region to a stop, literally. I sent that toy to LL devs back in 2009 and they hurriedly fixed that bug. Such a thing has no issues in InWorldz - the phlox design is immune to that kind of attack.

Viewer related issues are primarily the simulation of moving things. Every flexi prim, every particle, every spinning, sliding, hopping, twisting, jiggling, dancing object or avatar takes a teensy amount of time to simulate in the viewer (remember this is a viewer simulation, not the region simulator). So, the more of those things that are happening nearby the more work the viewer has to do. And, if that work exceeds the capabilities of your computer then yes, a lot of lag will happen. SL does a superb job of occluding things not in view. Land hills, nontransparent objects all will occlude things (not require viewer attention). Not so in InWorldz. InWorldz lacks an important tech feature to do that kind of occlusion. Only draw distance occlusion works. And worse, because of the lacking feature, if you visit a place with a lot of particles and then go elsewhere in that region, even is far far away, the viewer will end up simulating those totally unseen particles. By far, this is the weakest point of InWorldz (and yes, OpenSim too has that deficiency).

Bandwidth is by far the worst for most people because for many people, their network bandwidth is still somewhat low. Every time a script changes a color, a texture, moves something non-physically, the region simulator sends a message to the viewer to do that action. It is easy to write scripts in which these actions are done inefficiently, and indeed that is what happens most of the time. For people on 50-100Mb/sec Comcast Infinity networks (with your child not doing bit torrents :) ), this is rarely an issue. But, if you are on a 2Mb/sec DSL connection, your viewer will be in a world of hurt because busy regions often means a lot of activity is going on. Coupled with the InWorld deficiency of strong object occlusion, the situation gets worse. It is easy to determine if that is your problem. Ctrl-shift-1 will show you the bandwidth burned.

And finally a lot of stalls where objects take forever to rez or texture take forever to load is a somewhat recent bug in InWorldz (it started in 2013 and has gotten much worse). The combination of slow cloud based asset storage and some message delivery bugs make this a hit and miss problem. Nothing you can do about it.

Prims and mesh models do not cause simulator slowdown. They do take up memory and the simulators have fixed amount of memory, which is why there is maximum number set. Too many in a scene will cause other issues.

 

Comment by Balpien Hammerer on July 31, 2016 at 12:04pm

Oh, I forgot to add that the missing textures or textures taking a long tie to rez has always been blamed on network problems, but I demonstrated this was a grid code bug long ago. I reproduced it running everything locally on the same computer including the simulator and asset stores.

Comment by Wayfinder Wishbringer on July 31, 2016 at 2:32pm

Good post Balpien.  It's always good to shoot the target in the middle of the bullseye. :D

Reminds me when years ago you and I were having what became a fairly heated discussion with the Lindens, providing absolute proof that there was a texture bottleneck in SL and them denying everything we presented and telling us no, it's user-created problems (and being backed up by sycophant "tech" pro-SL customers), as if LL could do no wrong.  Ugh.

Sometimes, people just don't want to hear the truth-- because then they have to do something about it. 

Comment by Balpien Hammerer on August 3, 2016 at 3:51pm

Gosh I remember that time. They were 'no, no, heck no' until I demonstrated their own viewers were DDOSing their servers. That did prod them to look into the problem. They were making great strides fixing the texture bugs in v1, but then v2 happened, code reset, bugs back. *sigh*

And yes, I got into some intense and heated discussions with InWorldz dev(s) about letting these old bugs linger. I even offered to (and did) fix some of those kinds of bugs, only to end up in territory battles. It wasn't worth the constant frustration. I agree with the  'don't want to hear the truth' sentiment.

And to be fair, it's not just InWorldz. OpenSim has its own share of 'my way or the highway' development craziness.

Comment by Wayfinder Wishbringer on August 3, 2016 at 9:52pm

Yeah, Inworldz is downright friendly compared to most other grids, despite the imperfections (all grids have imperfections).   I do think with Inworldz it's likely a matter of constant stress getting to them.  They used to be a lot more open and willing to try things.   But I suspect after you have a whole buncha people demanding different things... after a while ya get to a point of just stop listening and do whatever we wanna do.   Human nature.

Inworldz can be happy that OpenSim is so disorganized and non-professional.  One thing I'll say for Inworldz:  they do a better job than the other grids, despite the problems. 

But I remember when you and I tackled LL Balpien, just like you said they flat out ignored the proof provided.  Then in 2010 they suddenly "discovered" the very things we'd been telling them before-- and took all the credit.  LOL.  That is typical LL method.  They did the same thing with the Greeters concept:  we created it, they swiped it, outright.  Then of course, they ruined it.  Because they weren't Elf Clan and although they had the concept, they didn't have the heart behind it.

"Development craziness"... I think you're right.  I can see no other reason this type of software would be this old and still in "buggy beta" stage.   You'd think it was Windows or something.

Comment

You need to be a member of Elf Clan Social Network to add comments!

Join Elf Clan Social Network

© 2017   Created by Wayfinder Wishbringer.   Powered by

Report an Issue  |  Terms of Service