Gestures as a language not a technology
by Jeffrey
John Gruber, commenting on a review of the Droid, made a comment about multi-touch which got me thinking:
The big multi-touch gesture that Apple uses in the iPhone is pinching — in particular, pinch-to-zoom. You pinch to zoom in MobileSafari. You pinch to zoom photos. You pinch to zoom maps.
There is no pinching on a Droid running Android 2.0. As for why, my somewhat-informed best guess is that it is related to Apple’s patent applications for the pinch-to-zoom gesture. If so, this stinks. — John Gruber, Daring Fireball
Personally, I’m all about innovation. Allowing an innovator to benefit from his or her innovation through patents is what makes businesses innovate in the first place. At the same time however, gestures are not innovative, nor should they be patentable. Pinch-to-zoom is just common sense. How else would you do it with one motion? If Apple wants to patent something, patent the technology that allows the device to recognize multi touch and react to it, don’t patent the gestures themselves.
Gestures should be treated as a language, like sign language for touch devices. We need a common set of gestures to interact with all touch enabled devices. I shouldn’t have to lean a different language just to use a different device.
This situation reminds me of the hyperlink fiasco from many years ago when BT contacted Prodigy and 16 other ISPs, including AOL asking them to buy a hyperlink licence. If Apple goes down this path and tries to enforce pinch-to-zoom as “theirs” I would only hope it, as well as any other gesture parents, would lose.
Comments
If it’s so obvious, why didn’t others implement it first? I hope the patent does come through, deciding that something is obvious (common sense) in retrospect only underlines how wonderful the invention is, not that anyone could have thought of it.
Although standardization gestures would be an awesome idea, awesome ideas are always shot down by a little thing called the patent.
The world could possibly be 10 times more advanced than it is today if patents did not exist.
The gesture is obvious, the technology to implement it is not. The only thing holding others back was their inability to create the necessary hardware.
Ask a lay person “If you had an image and wanted to make it bigger or smaller using body actions, what would you do?” Most people would probably instinctively do this: Stretch or Shrink.
With is the same as pinch if you only use one hand.
Interesting that the first two commenters state the canonical extremist opposing viewpoints on patents—and, like extermists in most cases, I believe both to be wrong.
Patents are good. Without them, there would be little or no incentive in the minds of most people to invent things. Certainly I wouldn’t have any desire to create without the protection of patents, copyrights, or trademarks.
On the other hand, the law always has acknowledged that such protections have their limits. Without such limits, potential inventors or creators would be paralyzed by the fear of infringement—something that indeed happens routinely in the real world.
Language, in the sense of organized communication, is one realm that probably should remain beyond those limits. Consider that the most heated debates over whether something should be patentable all deal with variations on this theme: patenting genes or algorithms, to name a couple of examples.
In short, I think the argument presented is correct. The technology interpreting gestures is the proper venue for patent, not the language of gestures itself.
Oh, but they did. That’s the problem. Multi-touch research has claimed a bunch of these; Apple was merely very early to the multi-touch marketplace.
What other aspects of a UI should be patentable or not patentable? When does an aspect of the UI become part of a language?
For instance:
– slide to unlock
– tap the banner to scroll to top
– shake to shuffle / undo
– double-tap to zoom in, two finger double-tap to zoom out
– flick right/left to view next/prev picture
– iPhone’s select/cut/copy/paste mechanisms
– holding down a key on a virtual keyboard to get a set of accented characters
I don’t think you can have patent for a gesture alone. There must be some “method or a device” involved in order to push it through patent system.
That only works for the first innovator. All others are then prevented from innovating. Usually progress gets made by taking what we know, and improving it and patent system hinders this, especially in IT where patents are in force too long compared to commercial viability (i.e. technology may remain patented even after it’s dead obvious and obsolete).
Myron Krueger implemented pinch-to-zoom in 1983. Apple acquired its initial multitouch technology from FingerWorks, which was co-founded by Wayne Westerman, who cited Krueger in his doctoral dissertation.
Strictly speaking, John Smith only asserted that an absence of patents would be better than the status quo, not that it would be optimal.
The comment box says “using markdown is ok”, but apparently blockquotes are not.
Myron Krueger implemented pinch-to-zoom in 1983. Apple acquired its initial multitouch technology from FingerWorks, which was co-founded by Wayne Westerman, who cited Krueger in his doctoral dissertation.
“Interesting that the first two commenters state the canonical extremist opposing viewpoints on patents”
Strictly speaking, John Smith only asserted that an absence of patents would be better than the status quo, not that it would be optimal.
” I shouldn’t have to lean a different language just to use a different device.”
Why not? You have to learn a different skill set to operate a plane compared to operating a car. Or, more to the point, every iphone app requires a little learning. So far, and in the foreseeable future, this “language” consists of maybe a dozen words, and no grammar. Child’s play.
Smart phones have been around for a decade, gestures for longer than that. Yet no one thought to combine them until Apple did it. They might not have bothered if they couldn’t have used patents to protect their investment, and then innovation would have been retarded, not advanced.
“Why not? You have to learn a different skill set to operate a plane compared to operating a car.”
Samuel Black, I think you’ve just undermined your own point. You’ll notice that most cars use identical mechanisms for steering, acceleration, etc.
@Lem Lemmon
“If it’s so obvious, why didn’t others implement it first?”
That’s a really, really illogical question.
Since everything that got implemented is once implemeted for the first time – regardless of being obvious or not.
Lots of innovation looks obvious with hindsight, but there were touch screens on phones for over 10 years before the iPhone, so why didn’t somebody else implement pinch-to-zoom if it’s so obvious? There were also photos on phones for years before iPhone, and zooming of photos. There were Web pages on phones for years before iPhone.
Yet the iPhone launched in 2007 into a world full of devices that were STILL mimicking the 1992 Newton and its stylus.
There are 1000 ways, which you don’t want to think about because you’re so used to pinch-to-zoom. On a Wacom tablet you zoom by sliding up or down on a vertical strip, which not only requires only one hand, it requires only one touch. You can use the scroller on any mouse to zoom also, although they’re typically set by default to scroll. You can drag a box around the area of the photo that you want to zoom in on, like Photoshop. There are many, many other ways. The one you prefer is the one Apple uses is all.
There is no universal sign language for hearing impaired, there are multiple sign languages, same as what you find in gesture-based computing. You can’t learn just one sign language and talk to all other signers, same way you can’t learn just one set of computing gestures and talk to all devices. False utopian dream.
Why? Why do you need to interact with ALL touch-enabled devices? Have you been doing this for the past 20 years? Do you know all the Newton gestures, all the Wacom gestures? If not, then why do you need all future devices to limit themselves to one common set of gestures?
Also, are all people, regardless of their profession and tasks, really going to need just one set of gestures?
Also, there is not even just 1 set of gestures that are common to both iPhone and Mac, both from Apple. For example, scrolling is 1 finger on iPhone, and 2 fingers on Mac. How could there be one set that would work on all devices of all kinds, from all manufacturers?
I think the world of gestures is much larger than we give it credit for right now.
Why not? You have to get a separate driver’s license to drive a motorcycle, car, or bus. Different planes have different control systems and instruments. Why should you be able to learn one computing system, e.g. an iPhone, and then magically you can work all others?
Keep in mind that every device in the world is either a computing device already or soon will be. The idea of making them all work exactly the same is attractive at first but not practical. Even right now, I’m doing gestures on iPhone, a Mac trackpad, a Wacom tablet, a Mackie mixing surface, and they all have very different gestures because they’re all very different devices.
“Allowing an innovator to benefit from his or her innovation through patents is what makes businesses innovate in the first place.”
Is this really true? It seems pretty ridiculous to me. I don’t think I know of anyone who does anything remarkable with the overriding goal of patenting it. Which isn’t to say it doesn’t happen (the keyword is “remarkable”). But should it? Maybe individuals or companies should be allowed a time-limited monopoly on a certain way of doing or making things, but I hardly think it should be taken for granted. After all, I think we can all agree that the patent system is stronger and more extensive than ever, but then why have all the great research labs (Apple, AT&T, IBM, Xerox, etc.) disappeared? These are the places that brought the world transistors and PCs and the mouse and GUIs and Ethernet and the laser printer and on and on (in other words, they were drowning in patentable research). And yet many of those companies have hit hard times, but no new research labs (with the bizarre exception of MS) have taken their place. Instead we’ve been left with nonsensical patent wars over very obvious “inventions” (one-click purchasing, a gesture).
This kind of nonsensical patent is what happens when someone “innovates” in order “to benefit from his or her innovation”.The mentality that the goal is to generate a patent, rather than solving people’s needs and generating patents as a byproduct, is the real problem.
“Lots of innovation looks obvious with hindsight, but there were touch screens on phones for over 10 years before the iPhone, so why didn’t somebody else implement pinch-to-zoom if it’s so obvious?”
They DID. They didn’t put it in a phone because the technology wasn’t small, cheap, or reliable enough.
“Even right now, I’m doing gestures on iPhone, a Mac trackpad, a Wacom tablet, a Mackie mixing surface, and they all have very different gestures because they’re all very different devices.”
A phone running iPhone OS and a phone running Android are very similar devices. That less similar devices exist has no bearing on whether Apple should be able to patent specific gestures, especially gestures Apple came nowhere close to inventing.
“Pinch-to-zoom is just common sense. How else would you do it with one motion?”
It’s obvious now that you’ve seen it and you’re used to it. I’d be willing to bet that Apple tried many, many different approaches to zooming, scrolling, etc. with gestures before they settled on the ones we use today.
That is excusifying. I don’t care why they didn’t ship. They didn’t ship! Apple did.
Portable? The gesture-based Newton is portable and it’s from 1992. Then there are 15 years and thousands of portable handheld devices until the iPhone ships in 2007, and its contemporaries were all still using the Newton stylus! What is your excuse for that?
The fact is, the “obvious” smartphone features are making it look like a little PC, with display, keyboard, and pointing device, like the Verizon Droid when it’s opened up. Obvious is a replaceable battery, with a skinny size you sell with the phone, and a fat size you sell later. Obvious is little menus you click on to choose items. Obvious is a dialer keypad. I know for a fact that all of these things are obvious because they were the common features of all smartphones before the iPhone.
That’s the problem: Android is way too similar to iPhone already. Now you’re saying that what hasn’t already been cloned is fair game, go ahead and take it, nobody invented any of these iPhone features, nobody designed them, nobody did the artwork, nobody inside Apple did any work on this at all. They just stole it from Xerox PARC, right? It’s fair game. Feeding frenzy. Apple devices are just prototypes for the “real” generic device that comes a few years later.
The thing that Google and Microsoft have most in common is no artists, no designers. Google hires Ph.D coders and Microsoft hires the non-Ph.D coders. So when either of these companies, more than any other, says that they don’t need to do artwork or design work because it’s obvious, you have to be suspicious. You mean it’s not because you have zero creative people on staff? How convenient for them.
@Jeffrey
Come on, Jeffrey. That’s a bunch of nonsense and you know it. That’s like saying that people that want total war or total peace are extremists and that the balance lies in the middle — that there should be some wars and some peace. I bet you really do think that there should always be a middle ground.
Well I’m sorry if you wouldn’t feel like inventing anything if a patent system wasn’t around to protect your precious, precious ideas. You live in a corporate world where everything should be secret because money is at stake.
But really, the human race should not limit itself because of it’s own created devil — money. Ideas should be free-flowing. Who cares who invented what as long as it pushes us forward.
[...] Original source : http://jeffreysambells.com/posts/2009/11/03/gestur…; [...]
[...] Reader: PhotoGestures as a Languagejen o’neill’s modern genealogy chartsCincinnati Footprint Print by VisuaLingualworld’s [...]
What if there’s disagreement between what is deemed to be the most appropriate method of standardisation? Why force gestures upon device makers if they don’t agree with the “standard”? We here may all think pinching is obvious, but that doesn’t mean it’s obvious to everybody.
I’ve found I’ve been “flicking” other touchscreen devices like GPS navigators because of my iPhone, but I agree where do you draw the line between “gestures” and the UI?
I take it you guys are new to multitouch? This was discussed several years ago on countless NUIGroup threads, and Apple even bought the company that holds the patent on what could possibly be called a standard set of multitouch gestures. There are even gesture engines in Javascript that use XML descriptions.
“That is excusifying. I don’t care why they didn’t ship. They didn’t ship! Apple did.”
Patent law doesn’t care who ships.
“The gesture-based Newton is portable and it’s from 1992.”
It didn’t have multitouch. The technology wasn’t small, cheap, or reliable enough, even for a device considerably larger and more expensive than the iPhone.
“the Newton stylus”
Apple neither invented nor popularized pen computing and had little to no influence on early smartphones.
“Now you’re saying that what hasn’t already been cloned is fair game, go ahead and take it, nobody invented any of these iPhone features, nobody designed them, nobody did the artwork, nobody inside Apple did any work on this at all.”
I’m saying that if anyone invented pinch-to-zoom, it was Myron Krueger.
There’s an old legal maxim: “If the facts are on your side, pound on the facts. If the law is on your side, pound on the law. If neither is on your side, pound on the table.”
You’re pounding on the table. I haven’t offered up an opinion on anything else that Apple might have patented or be trying to patent. I said that other people did this one thing first, and you’re histrionically accusing me of supporting copyright infringement.
“The thing that Google and Microsoft have most in common is no artists, no designers.”
Bullshit.
“What if there’s disagreement between what is deemed to be the most appropriate method of standardisation? Why force gestures upon device makers if they don’t agree with the “standard”?”
Jeffrey likened gestures to a language. Language standards develop naturally, without any need for force. Anyone can invent their own language, but most people prefer to use the language they know best or a language that allows them to communicate with another group.
Actually, deaf sign languages are not standardized, and I mean apart from dialect differences.
Language standards develop organically, not naturally; that I’m quibbling over semantics is part of the point. That gestures are language not technology is off too. Language is technology. However, it is a generic and continuously developing tool that only very specific “streams” can get a patent or copywrite. {“Final Four®” for example} Having said that I stress that language is, in this sort of example, an “item” and means of identifying the item at the same time.
I’ve just typed a whole paragraph that while being true is also bullshit sophistry.
For the record, patenting “pinch to zoom” is technology is valid (imho). It is not language since it is a unique implementation of an idea to control and manipulate a physical interface. And yes, Apple did acquire the rights to this patent from the original inventor and was the first to bring a viable device to market with such an elegant implementation that many (if not most) consider it obvious in hindsight.
[...] JeffreySambells.com: “Gestures as a language not a technology.” by Jeffrey Sambells: 03 November [...]
[...] UI, インターフェイス カテゴリーへ shiro が 2009年11月5日 に投稿 Gestures as a language not a technology. : JeffreySambells.com Gestures should be treated as a language, like sign language for touch [...]
[...] Gestures as a Language. Gestures are certainly a kind of language, and it would be nice to treat them as such. Except, [...]
[...] way or another, I agree with Jeffrey Sambells, who says that gestures are language and should be consistent on multiple platforms. Or to think of it a slightly different way: I know [...]
“I’ve just typed a whole paragraph that while being true is also bullshit sophistry.”
It’s certainly one of those.
“And yes, Apple did acquire the rights to this patent from the original inventor…”
Krueger never patented pinch-to-zoom, and the patent would have expired by now anyway. Gruber refers to pending applications.
[...] Gestures as a language not a technology : JeffreySambells.com “Personally, I’m all about innovation. Allowing an innovator to benefit from his or her innovation through patents is what makes businesses innovate in the first place. At the same time however, gestures are not innovative, nor should they be patentable. Pinch-to-zoom is just common sense. How else would you do it with one motion? If Apple wants to patent something, patent the technology that allows the device to recognize multi touch and react to it, don’t patent the gestures themselves. Gestures should be treated as a language, like sign language for touch devices. We need a common set of gestures to interact with all touch enabled devices. I shouldn’t have to lean a different language just to use a different device.” (tags: patent technology multitouch) [...]
“Pinch-to-zoom is just common sense”
Oh please for fuck sake, its only common sense because apple showed how it should be done, if they had never done that I can guarantee no one else would be using pinch to zoom now.
Thats like saying “well the mouse is common sense” well why did we waste time looking into light pens and other devices before we settled on it?
Claiming things are common sense is an idiotic comment by someone who doesn’t know the first thing about design.
@ Massive Dog…
WOW how much does Apple pay you??? How can you guarantee that pinching would not be in use if Apple didn’t do it on their iphone.
First off, people were using this in the 80’s!!!! Google for Myron Kreuger and “Video Place” and you will see him making pinching gestures and 2 fingers to control zooming and rotation (there are some very good ones from 1988).
The iPhone was first talked about in concept back in 2003 whereas Microsoft Surface was already under research and development back in 2001… a full 2 years before iPhone was even a twinkle of an idea in Steve Jobs’ mind.
A lot of people beat Apple to the punch.