?

Log in

No account? Create an account
Previous Entry Share Next Entry
Apps in Querki
querki
jducoeur
Today's Querki release has one major change: the first appearance of the "Apps" menu. It's hard to overstate how important that is. When I originally figured out what an "App" would be in Querki, three and a half years ago, I realized that I had a viable business here, not just a tinkerer's toy. It's taken all those months to get to the point where the rest of the system is solid enough to start implementing them. So pardon me a burble about what the word "App" means in Querki -- how it works, and where it's going.

The thing here is, Querki doesn't tick like most platforms. There's an almost universal assumption in the programming industry that "code" and "data" are fundamentally different. A program is built by a programmer, who writes code; he then gives that program to a user (often treated as a Dumb User), who puts in their data. That dichotomy underlies most of the history of the field, and it's central to the Priesthood of Programmers that you see even today.

And yes, I use the term "Priesthood" very deliberately. I mean, think about it. It's a body of specialists who understand the arcane rituals of intercession between those Mysterious Beings (computers) and ordinary mortals, to beg said Mysterious Beings to make their lives better. Many people want to join, in order to advance themselves. The only real difference is that you call a Priest when *you* are dying, but you call a Programmer when your computer is doing so.

Querki is, at heart, a rebuttal of the Priesthood model. The whole idea is to make it as easy as possible for everyone to manage their own data the way they want to do so, not constrained by some programmer's preconceived design. It's a Database as Wiki, trying to demystify things as much as possible, with an absolute minimum of Programming.

All that said, not everyone *wants* to come up with their own designs. If I build, say, a kick-ass Historical Recipes Space, and you like it, you don't want to go to all that work yourself -- you just want your own copy of how it works, so you can plug your own recipes in. You might want to tune it and tweak it a bit to your own needs, but basically I want to be able to share my work with you. That's where Apps come in.

So: A Querki App is a template, lifted out of an existing Space, which other Spaces can share. Simple enough, but there are a lot of implications there, so let's unpack it.

Code vs. Data

First, there's that code vs. data dichotomy that Querki just plain doesn't have.

Querki's all about the data, and tends to treat code as a necessary evil -- precisely the opposite of most programs. Whereas conventionally each program is its own little special snowflake, hand-crafted by some programmer, often reimplementing the same wheels as everyone else, Querki builds most of that directly into the system. Identity, Security, Sharing, Communication -- all of those programmer-y things are part of the standard package, so you don't have to think of them, and can just worry about what information you're trying to keep track of.

There *is* one exception, and it's what the QL language is designed for: displaying that data. There's no way around a *little* programming there, but it's intentionally minimalist -- a deliberately simplistic expression language that you embed in your pages, so that they come out looking a lot like ordinary wiki pages.

So if it's almost all data, what does an "App" mean? I mean, conventional Apps are all about the code, right?

The answer is that a Querki App is mainly about data structures. Specifically, the Models that you've developed to represent your data. What I want to share with you isn't a big blort of code -- it's the *idea* of how to represent this problem as data. (In reality, there are a *lot* of nuances -- I'm still figuring out the fine details of what usually constitutes an "app" -- but that's the high concept. Everything else is details.)

Extracting an App

So how do I go about doing this sharing? How do I write an App? The answer is, you really don't, at least not in the conventional sense.

The workflow of an ordinary program reflects the priesthood model: the programmer toils on the program, occasionally giving sermons in the form of new releases. The users receive this largesse periodically and (usually) uncritically.

Querki's workflow is *very* different. First of all, it's iterative beyond the dreams of most programming environments. You usually don't spend a huge amount of time on formal design, then coding, then finally testing. Rather, you think about about what you're trying to build, and *immediately* start fiddling with the data structures. You create a Model or three, and usually start entering data before writing any code at all. It's almost always easier to start out with some data, kick the tires, see what seems to be missing, and *then* start writing some Views for making the Things look nice. (If you bother -- Querki will work fine even if you never do anything other than enter data.) It's not unusual to have a first draft working decently within five minutes.

Finally, when you've entered a bunch of data, you're happy with how it's working, and you think it's useful enough to share with your friends (say, after a few days), that's when you turn it into an App. Like everything else in Querki, this is designed to be as quick and easy as possible. You say "Extract an App", and Querki shows you everything in the Space with checkboxes, so that you can choose which Things you want in this App. But it works pretty hard to pre-populate that sensibly, so that, in most cases, you just have to give it a name and *poof*, you have an App. This automatically contains all of your Models, your Pages, Stylesheets, Choices, and stuff like that -- all the data (and a little bit of code) that gave this Space its shape. And your Space is automatically rebuilt to use that App, so that everything still looks the same.

The programmers in the audience are asking (correctly), "So how do you maintain it?" That gets into the idea of Shadows, and the generally tweakable nature of Querki.

Shadows and Tweaking

I often say that Querki is the platform for the Maker Age. The fundamental assumption, unlike nearly every other system out there, is that One Size Does Not Fit All. Just because I've shared my Songbook App with you, and it works really well for me, doesn't mean it's ideal for you. You might want to add a Property to track the Time Period of each Song, or to spell out its Copyright Notice. You might want to add additional pages, that categorize the Songs by those Time Periods. You want to be able to *tweak* the App, with no more difficulty than I had in building the Space in the first place.

Hence, when a Space is derived from an App, it is populated by Shadow Things. Basically, you don't get copies of the Models and Properties of the App, nor do you use them directly; instead, you use local Shadows, that point to the Models and Properties of the App. Those Shadows are yours, part of your Space, and you can customize them as you see fit. (There will probably be limits to this, but we're going to work hard to get as close to this ideal as possible. Customizability is one of Querki's touchstones.)

So maintenance grows out of this. If I've created an App, and I later want to make some changes, I simply make them in my child Space. I fiddle and play as usual; when I like where it is, I lift my changes up into the App, as quick and easy a process as creating the App in the first place.

Moreover, if you're *using* my App, and you've come up with a great idea to share, you can suggest changes -- basically the same process I use for lifting my changes, but they become a suggestion that I can choose to incorporate into the App or not, as I see fit. And if I decide I don't want them, you can turn them into your *own* App, with your own improvements, which inherits from my App and is new parent of your Space.

(Many programmers are, at this point, saying, "So you've reinvented git?" Not exactly, but yeah, I'm using git for a lot of inspiration here -- you can see workspaces, pull requests, forks and suchlike in the above. If I can get most of the key functionality, without such an impenetrable process, I'll count that as a win. And I should note that most of this is *not* implemented yet -- this is stuff that will be coming over the next few months.)

App Sharing

While Querki is going to have an App Gallery, which will undoubtedly eventually have the usual comments and ratings and so on, that is *not* the primary model, because Querki is mainly social. So by and large, you don't *publish* an App, you *share* it.

Once Querki has social network integration (coming in the first half of next year), you'll be able to share not just pages, but Apps with your friends. This fits one of its sweet spots, which is building custom Spaces for communities. For example, I have a *pile* of Spaces that I've built, that are highly customized to the needs of the SCA (a club that I'm active in); I expect many of those to become Apps. I don't expect that to be unusual -- we've talked about communities ranging from book clubs to soccer moms, that need to keep track of bits of specialized information. So instead of focusing primarily on the "store" model, we're going to stay true to our social roots, and encourage members of those communities to share their useful Apps within those communities, so that others can use them.

In other words, Querki's App model is intensely community-focused -- much more like the open-source ecosystem than the conventional development world. We're trying to make it easy for communities to build the Apps that matter to them, collaboratively, and to share those Apps around. More than most features, Apps particularly reflect Querki's underlying ethos.

Conclusion

Ultimately, Apps are the reason why I believe Querki is going to be a mass-market system. While it's already great for anyone who knows how to "think in data" and enjoys fiddling, most people just want something that works, that they might want to tweak a little but mostly just want to use. In that respect, Querki Apps are a lot like the ordinary ones: people who just want to use them can just use them. The difference is mainly creation-side: an App that would take days, weeks or even months to build on a conventional platform can often be built and shared in an hour on Querki.

There is a *ton* more to do yet, before the above vision is fully realized. Some of the key steps in the coming year include:
  • Scalability -- moving Querki into the cloud, so that it will scale properly and quickly when needed.

  • History -- changing Querki's internal architecture to make it nearly impossible to lose data, so that you can Code With Courage.

  • Social Network Integration -- at least Facebook and Google+. (I'd love to have LJ integration; we'll see if that's practical.)

  • Fleshing out the above workflows, to make App maintenance easy.

  • And probably most important: improve the "user onboarding" experience. The first hour of using Querki is still *way* too mystifying -- fixing that, providing an easily-understood entry ramp, is probably the most important "feature" yet to come.
But even with all that on the horizon, I'm very excited: after all this time, Querki now looks more like the long-term vision than not. It's becoming ever-more-useful, and Apps will go a long ways towards introducing it to people.

Questions? Comments? I would *love* to burble more about this, if anyone's interested...
Tags:

  • 1
Unless it's changed substantially in form since the time I tried to use it, I think you may still be expecting more functionality out of your userbase than the average non-programmer is likely to have.

Well, that's why user onboarding is going to be the highest priority next year. At this point, it's certainly too hard for non-programmers to figure out what to do; even for programmers, it can be hard to figure out where to start. I believe that problem is at least somewhat tractable, but it's not going to be easy.

But that's one of the reasons why Apps are critical-path. I realized, in the wake of talking with you about your initial experience, that trying to describe the process of building a Querki Space from scratch is a lousy way to do it -- it's too big a wall of teaching before you get anything at all. I believe we're likely to be more successful by starting you off with creating a Space from an App (once that's quick and easy), and then teaching how to tweak *that*, step-by-step, and seeing what happens. That seems likely to be a much more satisfying tutorial process.

And mind, I expect most non-programmers to just want to use something with no tweaking -- that's why Apps are essential from a business POV. You're a good edge case -- some general technical savvy but not actually a programmer -- so I'm hoping to get things to the point where you're comfortable doing some building yourself. But I expect 80% of users will just want to enter data into something that someone else builds...

And probably most important: improve the "user onboarding" experience. The first hour of using Querki is still *way* too mystifying -- fixing that, providing an easily-understood entry ramp, is probably the most important "feature" yet to come.
I agree that this is the A#1 most important feature.

If you want non-programmer end-users to tweak things readily, I suspect you're going to need:
(a) to overcome the attitude-default: they won't even think of doing so because most other sites don't let them do that;
(b) to build out *substantial* assistance with QL editing; and
(c) to do actual focus studies to find out where the pain-points are with (b).

What I'd personally love to see is, when I'm editing a QL expression, to have some inline way to look for the QL that does the thing I want - like autocomplete in an IDE, but ideally searchable/better. Flipping over to the help pages and trying to scan through dozens of methods to find the half-dozen that might do what I want then looking at each of them to evaluate them is a PITA.

Non-programmers might actually want to go up a level, though, and be able to avoid text QL entirely via wizards, assistants, graphical query builders, etc.

Generally agreed, and building a QL "IDE" has been on the cards pretty much from the beginning.

It's actually one of the main reasons I took most of a week out last month to introduce the idea of Signatures to the built-in QL functions, and will be fleshing those out in the coming months. I've realized that we can only get useful autocomplete prompting if the system is at least moderately strongly-typed, with situational type inference for user code. (That's why Signatures for user Functions are low priority: ideally, those types should be inferred anyway.) Basically, I'm coming to understand that QL *is* actually strongly typed, albeit duck-typed when you're talking about Things.

So basically -- yeah, I've been wanting what you're describing for a couple of years now, but had to get the underpinnings working well enough first. (Among other things, this wasn't even plausible until the UI was rewritten in Scala.js a year ago.) I hope to get this started this year -- frankly, even I agree with you that having to look up functions in the documentation is a pain.

Non-programmers might actually want to go up a level, though, and be able to avoid text QL entirely via wizards, assistants, graphical query builders, etc.

Yep. Indeed, if you look *way* back in the project's history, you'll find that this was one of the original design motivations for the QL language. There's always been a plan to add a Querki Explorer Wizard, for which the existing Explore page is the first, very primitive step. The theory is a page which shows the steps of the QL expression on the left-hand side, and the output on the right. The steps are shown spelled-out, wizard-style; at the bottom is an "add a step" wizard; and the whole thing is intended to make it easier (if, likely, much slower) to play around with this stuff without thinking in syntax. The actual QL expression is probably displayed at the bottom for those who are interested, but it's basically de-emphasized.

The notion is that, if you're a non-Advanced user, this opens when you type "[[" in the Text editing box. When you finish building your expression, it goes back to the box, with a placeholder (possibly named) showing inside the [[]]. Advanced users can get into this wizard somehow, maybe with a ctrl sequence.

This is a *huge* project to get right, and I suspect has to wait until I get funding and can hire more people -- I'd bet we're talking several months for at least one person. In a perfect world, this is the sort of reason I'd like to be able to hire someone like metahacker -- a serious UX expert with good programming chops -- because this is mainly a really interesting UX challenge. I *might* try prototyping it myself, but not until more critical-path stuff like History is done.

Anyway, QL was *specifically* written to facilitate this UI: that's why it has the semi-strict left-to-right syntax, designed to translate relatively well to a top-to-bottom listing of steps. Most programming languages turn into a graphical soup when you try to wrap them in graphical wizards, because folks start with existing languages and try to turn them into UI. I started with the UI, and built the language to support that UI. I've needed to make some compromises that boil down to being able to nest an expression within a stage, but I think the UI is still pretty conceptually straightforward...

  • 1