Previous Entry Share Next Entry
Right, this is why "mostly functional" is always a dangerous phrase
[For the programmers.]

As I have burbled many times before, Scala is a lovely language: powerful, concise, and flexible. Sometimes flexible like rope, and occasionally flexible like a noose. (Somewhere, I have a button that says, "There is not now, and never will be, a language in which it is the slightest bit difficult to write bad code." This is a reminder of that.) Scala is a functional language, but specifically and intentionally *not* a pure-functional one. You can have state and side-effects, and you are responsible for using them in a judicious way.

Gradually over the past two years, I've been moving more and more towards functional techniques, and have gotten to the point where "var" always causes me to twitch slightly, but I haven't entirely foresworn them. The problem is, vars can interact with functional programming techniques *really* badly.

Today's example came from my rewrite of the system to use Autowire. This is a nice little library for marshaling commands across the wire -- now that Querki's client is itself written in Scala, this means that I can simply call the server from the client, and *poof*, it mostly just works. No farting around with hand-coded JSON or intermediate languages to describe the API: I just give the API in ordinary Scala, provide an implementation on the server, and call it from the client. It's quite lovely.

The thing is, though, my server-side code generally requires *context* that the client doesn't know about -- the calling User, the current state of this Space, that sort of thing. The main structure is called the RequestContext, and it is laced all over Querki. But of course, the client knows nothing about that, so how do I work it into my calls?

That is, my server-side code looks somewhat like this:
... req is the request; local rc param is the RequestContext
route[ThingFunctions](this)(req).foreach { result =>
  senderSaved ! ClientResponse(result)                  
The "route" macro takes the given request, interprets it as a request to the ThingFunctions trait, unmarshalls it, calls that method on "this" (which implements ThingFunctions), remarshalls the result, and returns that as a Future that my code then sends back to the client once the call has resolved. Basically, it does most of the scutwork boilerplate. But there is no way to pass that "rc" into the called function. What should I do?

My old-fashioned reflexes led me right down the primrose path to hell here:
var _theRc:Option[RequestContext] = None
... req is the request; local rc param is the RequestContext
_theRc = Some(rc)
route[ThingFunctions](this)(req).foreach { result =>
  senderSaved ! ClientResponse(result)                  
_theRc = None
That is, stuff rc into a variable that the implementation can get to. (Note that this is all inside an Actor, so it's effectively single-threaded; I wasn't *that* dumb.) Yes, the implementing code would sometimes have to copy this into a closure if it needs to do something asynchronous, but that's *kind* of obvious, so I could roll with it. And I did have a twitch from the "var", but it works just fine, right?

Yeah, not so much. Remember the above paragraph, which says that route returns a *Future* of the result? That is, it returns asynchronously, so that the implementation can be async if necessary. Which means that the actual re-serialization can happen well after the above code resolves.

And the thing is, functional code does *not* always work the way you expect. For example, deep inside one of the implementing methods, I had this function:
      def filterProps(props:Seq[AnyProp]):Seq[PropInfo] = {
        val filtered = for {
          prop <- props
          if (!prop.ifSet(Core.InternalProp))
          if (!prop.ifSet(Basic.SystemOnlyProp))
          yield prop
          => ClientApi.propInfo(prop, _theRc.get))
Seems innocuous -- it filters out inappropriate Properties to return to the client, and formats them. And this function is itself synchronous, so it ought to be fine.

Only -- the input is the values from a Map. And Map.values() returns a *lazy* sequence -- that is, in good functional style, it doesn't evaluate until it needs to do so. If you describe standard maps and transforms, it just adds them to the to-do list, but doesn't actually evaluate the sequence yet.

Until it is time to re-serialize the results. Which happens in a Future, remember? And notice that reference to _theRc above? Yeah -- while that is properly filled while the above function is called, it is totally empty by the time we finally resolve that Future and finally get around to resolving the sequence -- and finally call _theRc. D'oh! Took me an hour to track that one down and figure out what was going on.

So today's project was all about fixing this mess:
val params = AutowireParams(user, state, rc + state, spaceRouter, this)
val handler = new ThingFunctionsImpl(params)
route[ThingFunctions](handler)(req).foreach { result =>
  senderSaved ! ClientResponse(result)                  
That is, rather than trying to be clever with var's, do something more correctly functional: encapsulate the state and the functionality into an instance of ThingFunctionsImpl, and create one of *those* to handle each request. Works just fine (indeed, required very little change to the actual handler code), and doesn't commit the evil of mutable state.

Moral of the story: functional programming is a Very Good Thing indeed. But the further you go down that road, the further you will find you *need* to go. Querki is increasingly getting to the point where "var" is by definition a bad smell. (Indeed, on the client I am starting to eliminate it in favor of the reactive-expression Scala.Rx library. I may chat about that more later...)

  • 1
I'm always amazed at the compactness of the functional programming gun with which you can shoot yourself in the foot. Or rather, promise to later shoot yourself in the ERROR 'foot' NO LONGER ALLOCATED.

Heh. Yeah -- towers of crystalline elegance. Which are great until they are hit with that impure rock. I am beginning to sympathize with the Haskellers who get pissy about the fact that Scala lets you commit this sort of evil...

Hmm. I've been really intrigued by the trend toward fuzzing-based testing and its ability to uncover code weaknesses (or find intrinsic structure). I wonder if a continuously-fuzzed crystalline structure might end up usefully strong while retaining maintainability.

(side rant: as I type, I want my IDE to warn me about code fragility as well as warning me that my tests are failing and or that my syntax is wrong...)

I wonder if a continuously-fuzzed crystalline structure might end up usefully strong while retaining maintainability.

I don't think I know what that means, but it sounds very intriguing.

(And yes -- fuzz-testing has proven very useful. Although on the Scala side, the *really* popular one is ScalaCheck, which allows you to describe the characteristics of a Type, and generates a comprehensive set of unit tests for that Type. Very cool.)

as I type, I want my IDE to warn me about code fragility as well as warning me that my tests are failing and or that my syntax is wrong

Seems plausible for at least a limited definition of "code fragility", and I expect it'll happen sooner or later. But it requires either a *ferocious* amount of CPU, or a truly remarkable incremental compiler, to be able to do that level of semantic analysis in realtime. Still, sounds like a great Master's thesis for somebody...

It was an attempt to make an analogy to real-world crystalline structures, which can be cleaved easily if perfect; hence the use of doping agents to break up the regularity and increase strength. You can also do it by just shaking the darn thing, e.g., using ultrasound.

As for real-time...well, there's this cloud thing. I wonder if it can be harnessed without breaking the bank. (Or, I say it near a Googler and in a few months they do it for their internal tools.)

Fair point. My suspicion is that whether it's plausible or not depends on the language, and whether it is clean enough that you can reasonably efficiently derive "this is a bad idea".

(Curiously, QL *might* be that clean, and a Web-based IDE *is* already in the design. Hmm. Many steps to go first, but we'll see. First step is getting to the point of exposing the Type system strongly enough to be able to offer the legal functions as you edit...)

  • 1

Log in

No account? Create an account