2005 the Palm OS Year in Review

We'll just have to wait for Skippy to finally find out and blab.

May You Live in Interesting Times

;-O

Do you really not know what's happened, or are you just trying to be nice? Nudge, nudge. Wink, wink.

TVoR

------------------------
Sony CLIE UX100: 128 MB real RAM, OLED screen. All the PDA anyone really ever wanted.
------------------------

The Palm eCONomy = Communism™

The Great Palm Swindle: http://www.palminfocenter.com/comment_view.asp?ID=7864#108038

NetFrontLinux - the next major cellphone OS?: http://www.palminfocenter.com/comment_view.asp?ID=8060#111823

My PDAs: Visor --> Visor Neo (blue) --> Zire 71 --> Tungsten T3 (with 4 of 6 screws still remaining) ~?~> zodiac 2?

http://palmaddict.typepad.com/palmaddicts/2006/01/cobalt_10.html

-------------------------------
Editor, http://Pocketfactory.com
Contributing Editor, http://digitalmediathoughts.com

Palm: Things That Annoy the Developers (Like Me)

Sun, 11 Dec 2005

Well, its that time of year again where things must be purged, the soul laid bare in preparation for a new year of technology silliness. (Okay,I had pleasant time at a Christmas party last night and a nice little lunch out today but I'm still in a mood to let fly the flames of war ;) While I really enjoy developing for mobile devices (and development in general) and have very much enjoyed being on the front edge of mobile technology
the last half decade or so, I've become a little weary of dealing with PalmOne's (now Palm again) and PalmSource's sillyness to tell you the truth - one can only have so much time wasted through their sheer stupidity before you start to wonder - when will it end? When will they stop being idiots? Their "historical issues" (mistakes, guffaws, unfortunate changes in the timeline) are forever in the developers faces. After writing this huge rant, I realize - it is a wonder people develop for this platform at all. I've been assisting other developers in forums for years, and I know how tough it is for them to get past all this stuff. This is why after a major
Shadow release theres usually a few weeks of quiet .. time to decompress and come to terms with the sheer amount of crap that had to be shovelled. This is why theres so much time between releases sometimes -- its getting harder to get the nerve up to "dive into the mountain of crap" :P

A comprehensive list of things that annoy me would be enormous as I routinely work around dozens of different Palm OS system bugs every day. But off the top of my head right now, heres what comes to mind .. call it the abridged and shortened summary of the greater litany :)

Ben, and my many friends at Palm and PalmSource -
this isn't personal or directed, instead just the frustration as I work arond more and more bugs that make me pull my hair out. Gotta vent and get it off my chest :) I enjoy the
platform, and I admire the work and dedication you guys have put in, but we still suffer :)

Edit, Jan 2006: Remember, I love this platform and work on it every day. I wrote this after working around some really aggravating OS bugs and answering hundreds of support emails about it. Don't be going and thinking I'm anti-Palm OS or anything ;)

Note: I could equally flame Microsft Pocket PC, WinCE, etc. I will,
but after Christmas :)

Backwards and Forwards Compatibility

Microsoft has been doing operating systems for awhile now so I don't
give them much slack when they've blown the game of increasing device
resolution (requiring new developer APIs to support higher resolution
units, instead of one neatly designed API - we all knew that devices
would get better over time, right?) but at least they followed
Palm's example of making low res apps just work on higher res devices.
Well and good. Now, with all Microsoft OS PDAs working the same,
compatibility is pretty high - but not so with Palm OS.
It literally drives me and most other developers absolutely batty dealing
with the volume of OS bugs that infect every Palm OS device - and
do so differently with each device incarnation.

• Bugs - the
  developer doing something that works for years and then suddenly on new
  devices it makes the unit crash or not work as expected, due to bad
  ("hurried"?) coding in Palm. Example: On most halfway recent OS 5
  devices where you can copy a file out to SD card, and not back into
  RAM afterwards. Thast not a feature, folks .. and its been around a year
  or two at least now.. unfixed. In some cases it can be worked around but
  it often involves risking making problems for syncing later, and such.
  Not good. Please.. fix it?
• Bad decisions due to 'history' - Great, the T|T1 introduced the
  5-way d-pad like controller for better one handed control and an API
  to support it more or less, and it could even be used in games. Alas,
  the developer API for the 5-way was due to change itself several times
  as new devices came out and with the usual lack of PalmSource controlling
  licensees developing on its OS, it was only a matter of time when Handspring
  came out with its entirely different API for a 5-way controller. Of course,
  sure enough when Palm bought Handspring and returned them to the fold, they
  adopted the Handspring model so new Tungsten units have the Handspring
  control methods.. a little weird. Still, its so typical of Palm's bad
  management as to allow dozens of incompatible control types - the jog
  dials of Sony versus Handspring etc etc. We developers just love having
  spaghetti code - dozens of exceptions in every piece of the code to handle
  each device type, or requiring us to make massive compatibility layers
  for our applications to operate halfway right. Naturally, Palm encourages
  the users to blame the developers for problems or when an application doesn't
  feel native to their device... when its Palm OS being changed on every once :/
  This is getting better, due to the attrition among OS licensees, but I
  know most of us still dont' well support some of the new APIs as we're
  too mired down the the dozens of older APIs. ie: I need to better support
  the Handspring-style 5-way with its floating blue box, but I still need
  to support the 5-way on the older devices.
  
• Lack of standards - Even when you do manage to fully support an API,
  that API will itself change from device to device in how it is implemented
  within the OS, and more often than not it was never designed in such a way
  as to be flexible this way; consider the virtual grafitti, where there is such
  a mess of changes that not even Palm can keep its own applications working
  correctly with it. (The prefs panel system is notorious for having the
  virtual grafitti come up and cover half its options as they forget to resize
  properly.) - a "DIA" compliant application will work fine on one device,
  and then flicker on another device with too many redraws, or fail to rotate
  properly or receive system events in different orders or .. or .. thankfully
  it "more or less" works, but it is in fact impossible at this time to make
  a bug free virtual grafitti application. So many developers are overjoyed
  that the Treo square-screens are taking over so they needn't worry about
  virtual graffitti anymore.
• Lack of forsight - The PIMs, the lack of more than a few categories.
  The PIM (todo, datebook, etc) databases were never really designed to be
  extensible. Sure, back in the OS 1 days they had no idea if they would
  be successful at all, had limited time and budget and constrained devices
  but I would argue this is no reason for shoddy development. For all the
  years we've all be saddled with this lack of forsight, so Palm could not
  extend their applications to suit users needs (grumbling users the
  result, as they watched Pocket PC applications get more funky), and
  developers having no way to extend those applications (leading to annoyed
  developers making hacks.) Consider - Agendus and DateBk5 (and dozens of
  other applications) adding garbled text to the Notes of todo and datebook
  items so that icons can be assigned, or other special features added.
  It works, but its not really pretty with that garbled text visible in other
  applications and the desktop applications. Worse still was when PalmOne
  decided they'd had enough and actually upgraded the PIMs - great, the users
  will be happy .... - but they again forgot to supply a useful API and
  instead just extended with databases without actually making them
  extensible per se. So still the garbled text sits in the notes, and
  now developers have to support two formats of data instead of one.. more
  code duplication.

I always championed PalmSource by
suggesting this is due to the difference in mindset and perhaps a lack
of developers - that Palm pushed for
diversity of devices and would happily try and warp their OS to fit a
new device, while Microsoft clamped down specifications very rigidly. Hence
to this day there are quite a variety of Palm OS units from watches to
game machines to phones all on "one OS" (riddled with bugs), while in
the Microsoft PDA world you'll find a fair variety of devices all
working more or less alike (not as many bugs, but surely some duzzies!)
Good and bad - Pocket PC is fairly easy to develop for, since all devices
have pretty much the same feature set, but Palm OS is fragmented to all
heck. Still, I like the variety of devices.... well, I did, until I had
to support them.

In the end, to summarize - it is a chore to develop for Palm OS
these days; most disregard all the older devices, but I try to maintain
compatibility with and not forgot those trusty old friends... but the time
is coming when I just start ripping out all that code -- it is difficult
enough keeping up with all the new bugs in the OS, let alone old ones.
Developers wish to be productive as time is expensive and tight, yet 90%
of time I find is being consumed finding OS bugs and investigating work
arounds. How much more intuitive would an interface be if we could
focus on improving it, rather than dealing with bugs? How many more
features would there be? How much lower would prices be? We have no
time for 'business logic' when we're wasting it on fluff. This my friends
is one of the biggest challenged in Palm OS development.. they're taking
the fun out of it, and making it hell.

Palm Competing with Developers

So great, we were all happy PalmOne decided to start growing their PIMs.
Typically, they just altered the database format without actually
improving it, but worse still - they didn't document their changes for
years - I can only assume they resisted this because they feared
competition from the very developers they depended upon. What other
explanation is there? Certainly there is no patenting or licensing
issue at large.. though it could just have been lack of manpower to
make it happen. Regardless, it made our lives _miserable_ and to this
day we're all still dealing with the fallout.

Remember, when a tire won't fit on your car, you don't blame GM.. you
blame the maker of the tire. When users found crashes, slowdowns or weird
data loss, they blame the developers. Not PalmOne for breaking their own
specs and then not informing anyone how it worked..

What they did was introduce new databases, undocumented; when developers
applications opened the old databases, the OS would kick in and ferret data
between the new and old. Sounds good on paper, but evidently PalmOne didn't
debug or test this process much, since it was flawed in the design and
buggy in its implementation - frequent device crashes would occur due
to bugs in their 'PIM sync' (my name) code. Things would slow down
enormously (especially with the later NVRAM devices) as data was
forever being copied back and forth, something that never occurred
before. Worse yet is data loss as the PIM sync layer tried to guess
the intent of things going on and often got it wrong. So the developers got
flack again, for PalmOne's mistakes.

They finally did release some documentation, years late. I don't
get it. Months of suffering for no good reasons.

The real tragedy is the lack of learning - this wasn't the first nor
last time the documentation didn't come until devices were on the market
(sometimes for years) and our apps late to be stable. I'm not even going
to talk about the dozens of serious NVRAM flaws - NVRAM being a great
little system that users need, but when a large %age of apps become
unstable it makes you wonder how much testing went on..

Development Tools - or Lack There-Of

Microsoft, aside from its immoral behaviour, knows
how to make some decent tools for developers to work with; their OSes tend
to be backwards compatible and they do a pretty good (if pricey) job of
ensuring developers can get things done. Their Visual Studio suite is
quite popular and its available for C/C++ and Visual Basic development for
Pocket PC - a big comfort to many. With their new .net platform you also
get C# tossed in, and halfway useful drag and drop GUI building tools.

With Palm OS 5 and earlier, the official development suite was always
Metrowerks Codewarrior - a pretty good product. For OS5 "ARM" code
development it was useless for awhile (seriously buggy), but they worked
hard to move CW 9.0 to become CW 9.3 - flaky, but overall pretty darned
decent for doing OS 5 68k+ARM development. A lot of us have been using
CW for years, since it was the best overall system - and it was
discontinued a year or so back or more. New developers have a hard
time locating a copy of the package to use, so they resort to the PRC-Tools
(freeware gcc, a thing close to my heart) which is much harder to get
into.. discouraging away new developers. CW went away likely because it
is hard to make money with development tools, but also because PalmSource
was building their own development environment - PODS - for their OS 6
product. So PODS is out and about, but as you all know OS 6 is not.
So PalmSource spent enormous resources and time working on OS 6 upgrades
and development tools for us, that we couldnt' really use.. as PODS isn't
so good for OS 5 and earlier. Great - Palm depending on out of print
tools, and PalmSource working away for years on things no one is using.
(gcc is great stuff that I use every day, but due to the 68k+ARM
system in OS 5 devices its never been all that convenient.)

So in the Palm OS world, developers have been cobbling things together ..
a sorry state of affairs .. something Palm should be ashamed of.

OKay, theres some other tools so lets flame them too - Java is a popular
thing these days, correct? Well, a shame that Palm only licensed Websphere
Micro for a few of their devices, and isn't really committed to it. Certainly
it is unusable since how can you afford to base an application (months or
years of work) on something which is only for a subset of devices and may
be dropped at any time? Certainly, Java isn't available on Pocket PC at
all since Microsoft isn't all that nice...

The reverse is also true; MS wants everyone to move to C#, but of course
they will never provide a C# environment for Palm OS, and Palm just can't
afford to kick off yet another development tool project.

If you want some cross platform code to work, your best bet is C/C++,
or maybe somethign like NSBasic or SuperWaba. Still, a painful situation.

One other sub-annoyance; people all want java and C# and python these days
as they're sick of building from the metal; sure, productivity is up with
these languages as they've all borrowed some of the best inventions of the
past and mixed in recent patterns and experience; further, they've got
enormous manpower poured ito making new libraries to save time. (C++ and
such never quite got so organized as to have these giant libraries bult
for them in standard ways.) Anyway, people are forever asking why all our
Palm OS apps aren't in java -- they seem to forget the fact that java
isn't really available or fast for mobiles (hell, most phones don't
even do JITting or garbage collecting these days) - that C and C++ are
the really truly only portable codebases out there, to this day. But
try explaining this to the newbies fresh from school, without sounding
like an old codger. They do likes their java ;) Sadly, me too -- but its
not an option so stop bugging me :)

The OS 6 Fiasco

This is much publicized so I needn't go into it much, but in summary -
PalmSource (the OS developer) spent years building and pushing Palm OS 6
as the follow-up to Palm OS 5. Numerous developers (including myself)
started working on OS 6 versions of our applications - either changes to
remain compatibile or re-developed to actually use some of OS 6 features.
Alas, OS 6 was not to come .. despite PalmSource pushing it and announcing
devices and partners, we all quickly realized it was doomed from the
get go, so all that time was pretty much shot. To this day, PalmOne
has been knocking out OS 5 devices with their modified (read, buggy) version
of OS 5 .. a great overall product, but with so many extensions and PalmOne
seemingly out of control with regards to quality and standards, its started
to be referred to as "Frankengarnet" by the developers and users.

Mac OS

Mobile developers aren't big companies with enormous budgets to burn, so
we like to be cross platform (using one codebase for multiple operating
systems) but its nutty over in OSX-land. We have a stable and dependable
conduit for Windows, so that Shadow Plan can sync with Shadow Desktop and
work very well. It would be nice if that same codebase could work under
OSX, but we quickly found out that not only would that not happen, but
that Hotsync Manager for Mac OS X is basicly a butchered up version of the
Mac OS 9 system - that modern OS X style code cannot work with it.
Wonderful how our Mac OS X application is OS X based and due to various
dependancies can't tie into this OS 9 Hotsync Manager system. What silliness!
Its possible to rebuild the needed dependancies, but I am rather sick of
returning to the dark ages, of building up from scratch for every
application somewhere along the line. At least Mark/Space is on top of
the ball and has built their own Hotsync Manager replacement that offers
modern OS X facilities.. but I do not feel it right to saddle our
customers with an extra product (and none to cheap!) just to cover up for
Palm's mistakes. So we .. as usual, work around the system limits and
bugs. Another shameful situation.

Globals, 68k/ARM, and Other 1970s Problems

Again, being spoiled by the wonderful development environments of the
1980s, am sick of painful development situations requiring great devious
minds to actually produce great things. Moving from 68k to ARM was a
difficult yet amazing thing for Palm - its incredible they pulled it off
as smoothly as they have; almost all Palm OS applications to this day are
68k processor applications, running inside of PACE - Palm OS 5's compatibility
system for its ARM processors. Amazing stuff, so that our old apps can run
on the new architecture without changes. Great!

The problem is when you wish to actually write ARM code, to get some
speed. PACE exacts a large performance hit - something you wouldn't notice
for GUI applications but when putting together a high performance
multimedia application such as a game or video player or the like, you need
the real hardware - the ARM system. Ben and the boys worked hard to make
Codewarrior 9.3 stable and its a great product overall.. but due to the
nastyness of the OS and interfaces, we're all forced to jump through hoops.
You see, the 68k suport is quite good of course, but for ARM code you have
almost no access to the rest of the application APIs unless you 'stub them'
over. So to begin with, we all have to make stubs like mad, so we can
make ARM code halfway as easily as 68k code. Ben really helped out here
and the community worked together to produce a lot of useful code, but
a shame PalmSource didn't provide it - they assumed we'd all be on
OS 6 by now, and it just didn't pan out. The whole ARM-68k division is
really quite painful - a lot of us have it pretty well mastered, but we're
sick of the pain; newbies - forget it. Theres just such a learning curve
and hump to get over that few ARM-based applications have come out.
Certainly, to this day its still difficult to port applications from
standard operating systems that use certain features, and you'd better
have CW 9.3 (costware and discontinued) if you want your life to be sane..
using the freeware tools should be enjoyable and powerful, but its taken
years for that community to get to this point. Terrible.

At least I did my part.. I got XCade out pretty fast (one of the first
all ARM applications).. but its not pretty :)

Another thing that annoys me is the whole launch code and globals
situation - its bizarre to be in the year 2005 here and have to deal with
things like an app being launched without globals being available, due
to how Palm OS works. It was innovative and a must-do-to-survive situation
back in OS 1 and 2 and whatnot, but for OS 5 .. we're just sick of having to
warp our code in these terrible fashions to make things run. Lets not even
talk about 64k segmenting and whatnot .. I hate nothing more than copying
functions around, into small little files, so I can re-order them so that
functions can reach each other. Good god, that was the 286 days :)

OS Behind the Times

We all like compatibility (see all that ranting above?), but we're also
cool with well controlled changes. If your OS is broken - fix it. We'll be there
if you manage that process, publishing specs and documentation so that we
can be ready when the newer versions roll out. Bring it on.

It seems alien again to be in 2005 with simple bleep-bleep alarms in
Palm OS; the user can't easily assign mp3s or wav files to alarms or
ringtones, without third party hacks. I've even written and given away
a simple hack for years, to let small wav files be used as alarm tones.
We need an OS change here, so that all applications can have a modern
alarm managing system. Theres some cool stuff in the OS now, so that
alarms can be made to vibrate or be silent from a user configuruation
level withotu changin the apps.. but wheres the modern stuff?

Missing APIs

This ties back to the question of whethor Palm treats developers as
competitors. Where are the APIs that the built in applications can use?
How can we, on a Treo, control the radio to the degree the built in
applications can? Perhaps its documented in some buried Handspring
reference, but when I take a quick look through the usual places (dozens
of out dated documents :) I find it hard to find some of these things.. if
they're to be found at all. Lets cut to the chase - we want a good solid
knowledge base and search, like Microsoft provides. (Well, better than
they provide would be nice ;)

Licensee Madness

This is mostly old news, but it drove us all mental over the years.
I still support many of these older devices so its still relevent to me
but not for much longer I tell you. Especially, Sony as they were King
of Bastards.

Sony pioneered high-resolution because PalmSource couldn't get something
together fast enough I suppose, but they failed to think ahead and made
a pretty dimwitted API. At least it worked more or less... but it was
prone to crash. If the user or developer did certain unknown things,
the device would just hang up - again, the developers took flack and
it took ages to work out the myriad causes. To this day I keep an old
device around for testing, since the problem randomly manifests. Sony
never fixed this bug.. years would pass and the issue only went away
when OS 5 came along and Sony pseudo-adopted the OS 5 high res
system.

Course, Sony was evil for refusing to document how their joystick
add-on operated. Their documentation to the developer consisted of saying
"it just works", but alas it did not - it depended on some undocumented
series of events occuring in order to work, so for all my gaming applications
it worked ramdomly. (That was fascinating.. it should work, or not work.
But working sometimes and not others.. bizarre.) It'd have been nice of
they'd taken 20 mins to document the requirements.

They also refused to document the sound system; as usual, developers
took flack for not supporting special audio on Sony devices, but such
is life. Years passed before Cliepet, a whiz in the community, reverse
engineered it all. Bastards.

I'll try not to go on, but the problem was PalmSource's lack in taking
control of the reigns - why did Sony and Palm and Dana Alphasmart and
Handera all work out different ways of doing things? Shadow Plan
supports some 4 or 5 different incompatible techniques for managing high
resolution. Again, think how much higher application quality, and how much
lower costs would be, if we didn't have to both develop dozens of duplicate
ways of doing things, and then support them over years!

Really, its amazing anyone developed for the platform at all when
you look at this mess :)

At least Tapwave got most of it right.. a shame they tanked :/

Installations

A standard installer process would have been nice; heck, thinking ahead
would've been nice, too, so that Sony and Palm installs could sit nicely
side by side without blowing each other up. We rely on Beik's Pilot
Catapult installer builder, since its always been simple and inexpensive
and fairly stable, but even they can't quite keep it solid due to all
the varieties of weird in the registry that the various licensees put
in. (Microsoft doesn't help with their eternal OS monkeying on the desktop,
making Hotsync Manager sometimes run in Administratior mode while other
apps do not, etc.) A shame when PalmSource laid off the fine fellow working
on a promising new standardized installer.

Quality Assurance

Last but not least is just the lack of QA here; how does it slip through
the testing cracks that sometimes a file can be copied out to SD from RAM
and then not back into RAM? Or that devices crash/reset more frequently
now than ever? Since the new PIM databases weren't documented we
were required ("supposed to") use the old PIM database interface (thus
being denied the new features, plus gaining all the slowdowns and bugs)
and yet the magic PIM sync would frequently crash. Crazy stuff, and
all the blame comes back to developers.. with nothing we can do.

Whew!

When I began this opus I had not intended to write all this out .. I
merely wanted to rant a bit, to get it off my chest how miserable the situation
has been for years.. but out it poured.

Now that I'm all healed, you're suffering. Merry Christmas! :)