Moya 0.6.0 Released

I'm happy to announce the release of Moya version 0.6.0.

This version contains a number of new features and enhancements, as well as improvements in performance. See here for the full change list. I've also blogged about this release.

read more…

Creating a Wiki with Moya - Screencast

I'm starting a series of screencasts on how do build a web application in Moya. In part 1, I show how to build a functional Wiki.

read more…

Moya 0.5.14 Released

I'm pleased to announce the release of Moya 0.5.14. Moya is a web application platform written in Python.

This release is an accumulation of bug-fixes and modest features. Here is a highlight:

  • Cross Site Request Forgery (CSRF) protection was added to Moya Forms. CSRF is a class of exploits that involve making a request on behalf of a user, for malicious purposes. Moya protects against this using established techniques that check that the submitted form was generated by the site itself. This will be done automatically and doesn't require any changes to form code.

  • Added thread support to Moya Code. If you want code to run in a thread, wrap it in a <thread> tag. With no parameters, the code inside the thread will run in parallel until completion. If you add the dst parameter, and reference that value in your code, it will block until the thread has completed.

  • Added <system-call> tag to invoke external applications.

  • Added {% lorem %} template tag to easily generate lorem ipsum text in templates.

  • Added db:decimal tag for decimal fields in db models.

  • Added <db:property> tag which adds properties based on expressions to models. It is used in the auth lib, so you can do user.full_name which combines the first_name and last_name fields.

  • Added forupdate to <db:query>, which issues a FOR UPDATE in the generated SQL.

  • Added a number of parameters to the table library to dynamically hide columns / cells and set cell alignment. The Tables library can now pack a lot of punch.

  • Added linkify parameter to the text markup type, which turns URLs in to links in addition to adding line breaks. For example: {% markup content as 'text' set linkify=yes %}.

  • Added token: modifier to generate a random token which generates a random alphanumeric token. For example; token:16, creates a random token of 16 characters.

  • Added hidelabel parameter to form fields, which hides the label, but renders it in the markup for screen readers.

Moya can be installed via PIP, or by installing the code from GitHub.

Other than the tags, which are auto-generated, the documentation hasn't yet caught up with some of the new features. I will be working on the docs in the next few weeks.

For more details on recent changes, see CHANGES.txt.

For a complete working Moya project see Bean Counter or Notes.

read more…

Moya Context Python Interface

The Context class is an interesting data-structure Moya uses to implement much of the features of its high level language and template system. It is flexible enough that it could have applications in other Python projects.

Template systems, such as Django's, also have a context object, which is a container for template data. A Moya context is an encapsulation of a container plus the functionality for extracting values from it. But unlike Django where a context is created for the purpose of rendering a template, Moya creates a single context for each request were all data is stored.

In essence, the context is a namespace for nested data where every item may be referenced by a data index. A data index is much like a path in a filesystem; where a path contains directory names, a data index contains the keys or attributes of the intermediate Python objects.

Let's consider the following nested data:

middle_earth = {
    "species": {
        "hobbits": {
            "names": ["Sam", "Bilbo", "Frodo"],
        },
         "dwarves" : {
            "names": ["Durin", "Thorin", "Fili", "Kili"]
        }     
    }
}

If we want the second name in the "hobbits" dictionary, we could write the expression middle_earth["species"]["hobbits"]["names"][1]. With the data stored in a Context object, we could refer to the same value using the equivalent data index, with middle_earth["species.hobbits.names.1"]. We can see that the data index is more or less a series of chained getitem calls, where each key is delimited with a period in the index. The final '1' in the data index is notable in that it is treated as an integer even though it is part of a string.

In addition to saving a few characters, the extra layer of abstraction provided by the data index allows the Context to implement some more advance features such as scopes, links, and lazy evaluation. I'll cover those here, but first an introduction to working with Context objects.

Creating a Context

Creating a Context object is very simple. Here's how we could turn the above data in to a context:

from moya.context import Context
middle_earth = Context(middle_earth)

Constructing a Context instance like this doesn't copy or modify the original data in anyway. In fact, the Context keeps a reference to it; middle_earth.root is a reference to the original dictionary

Frames

In keeping with the filesystem analogy, a Context object has a 'current working directory' of sorts. By default, a data index starts from the root of the context, but the starting point for the index may be set by pushing a frame. Here's an example:

with middle_earth.frame('species.hobbits'):
    print(middle_earth['names.1'])

The first line pushes the frame 'species.hobbits'. All further indexes are considered to be relative to the this index. In filesystem terms, this would be the equivalent of doing cd species/hobbit.

Absolute Indexes

Like filesystem paths, a data index can be relative or absolute. A relative index will start indexing from the current frame, whereas an absolute index will start indexing from the root. Absolute data indexes begin with a single period. Here's an example of using an absolute index:

with middle_earth.frame('species.hobbit'):
    print(middle_earth['.species.hobbit.names.1'])
    print(middle_earth['names.1'])

The above code print "Bilbo" twice. The first index, .species.hobbit.names.1, is absolute due to the initial period and will start indexing from the root regardless of the current frame. The second index, names.1, starts indexing from the current frame, species.hobbits, and ultimately references the same value as the first index.

Moya makes use of absolute indexes by placing frequently needed data in the root of the context. For instance, the request object is always available as .request.

Scopes

A scope is a similar concept to a frame. Like a frame, a scope is a data index which sets a new initial starting point for index operations. But unlike a frame, multiple scopes my apply. The Context object will try each scope in turn until it finds a value. Probably the best way to illustrate this is with an example of how it is used in Moya's template language. The following is a simple for loop that generates a paragraph for every item in the dwarves list:

{% for name in .species.dwarves %}
    <p>${name} is a dwarf</p>
{% endfor %}

In each pass through the above code, Moya creates a new scope containing the value 'name'. Inside the loop, references to 'name' will use the value from the recently created scope. When the loop completes, 'name' will have its original value restored (assuming 'name' was assigned to anything prior to the loop).

Here's some abbreviated code that explains what the template system is doing behind the scenes:

for name in middle_earth[".species.dwarves"]:
    middle_earth[".for"] = {"name": name}
    with middle_earth.scope(".for"):
        yield iter(children)  # defer to the code inside the loop

Magic

Within the template system or Moya Code, a data index is essentially a variable, much like a reference in Python. But because it's the Context instance that looks up the value of an index, there is greater opportunity for magic where the context runs some code when a value is requested, rather than pulling it out of a dictionary.

The Python language can and does do magic like this (weakref , asyncio), but with a Context interface, the abstraction is less leaky.

Dynamic Values

One of the simplest ways of implementing such magic is with dynamic variables. Rather than set a value on the context, you can set a callable which returns a value. The following is an example of setting a dynamic variable:

>>> from moya.context import Context
>>> from datetime import datetime
>>> c = Context()
>>> c.set_dynamic('now', lambda context: datetime.now())
>>> c['now']
datetime.datetime(2015, 4, 5, 15, 41, 9, 713115)

The data index 'now' returns the current time every time it is looked up. The value returned is a simple Python object, and not a proxy of some kind.

Lazy Evaluation

Another use for magic is lazy evaluation. This involves setting a value on the context to the result of a function call, but only invoking the function if the value is referenced; a concept known as a future or promise in some languages. Let's look at an example.

The following Python code sets a lazy value 'mol' to the return value of the function 'meaning_of_life'.

>>> import time
>>> def meaning_of_life():
...     time.sleep(5)
...     return 42
>>> c.set_lazy('mol', meaning_of_life)

The call to set_lazy sets the value of 'mol' to the result of calling 'meaning_of_life', but returns instantly. It's only when we attempt to use that return value that the 'meaning_of_life' function is invoked. So the first time the value of 'mol' is looked up it will take 5 seconds:

>>> c['mol']
42

Subsequent calls to c['mol'] will return the value instantly. As before, the value returned is a simple Python object. Any code referencing the value of 'mol' in the context, would be unaware that it wasn't just an integer stored in a container.

Moya uses lazy evaluation for things like current user object. The value .user is always available, but it is only retrieved from the database when it is referenced. This avoids the requirement to call a function everywhere the user is required, without the risk of making a query that is never used.

Expressions

The Context is also the interface to Moya's expressions system. Here's an example of evaluating an expression from a Context:

>>> from moya.context import Context
>>> c = Context()
>>> c['bitstamp_api'] = 'https://www.bitstamp.net/api/ticker/'
>>> c.eval("float:(fromjson:get:bitstamp_api)['ask']")
257.91

The above code gets the current asking price of Bitcoin from an online API. Moya's expressions are designed to pack a lot of punch and can be used in templates and in Moya Code (the high level language). Data indexes in expressions are treated as variables, for instance c.eval('a+b') is the same as c['a'] + c['b'].

Summary

The Context is a powerful Python collection, with a remarkably simple interface. This post touches on some of the uses of the Context from the Python side. In a Moya project the context is entirely transparent, in that you use the context every time you assign or read data. See the Moya Documentation if you would be interested in learning more.

read more…

Moya 0.5.12 released!

Moya 0.5.12 was recently released. Moya is a language and web application server built in Python. You can use Moya to create dynamic sites, such as this one.

You can install Moya with the following command:

pip install moya

Add -U if you already have Moya installed and want to upgrade.

Run the following command to test Moya works on your system:

moya serve .

What's new?

This release includes a number of fixes and new features.

  • Python 3 compatibility fixes.

  • Windows fixes.

  • Multiple media URLs. The media url setting now accepts a list of media URLs, which Moya will cycle through when rendering a page. This is useful if you want to serve your media files over several subdomains, to trick the browser in to making more simultaneous requests.

  • An experimental generic key field was added to models.

  • Added new --slow switch to the runserver command. This switch inserts a small delay before responding to requests, to simulate network latency.

  • moya-pm command. This command can download and install packages from packages.moyaproject.com

  • Added a priority attribute to <mount> and a urlpriority attribute to <install> tags. Previously URLs were checked in the order they were defined. Now you can explicitly define the order.

  • Added ability to set the default project from an environment variable. If the MOYA_PROJECT environment variable is set, Moya will use this path in preference to the current working directory.

  • Added moya-srv command. This command allows you manage multiple Moya projects for a single server. Not yet documented, but more info can be found here.

  • Added RSS feeds. You can now use the <feed> and <add-feed-item> tags to serve an RSS feed, as used in the built in Blog application, and the official RSS feed for Moya news.

Discussion Group

If you are interested in Moya, please join the discussion group.

read more…

Moya discussion group

I've just create a Google discussion group for Moya. Sign up if you are interested in this project. This is probably the best place for technical support.

read more…

Moya Package Index

Just live is Moya Package Index, a site where you can find and download packages for Moya.

If you have installed Moya recently, you will have the moya-pm (Moya Package Manager) command, which you can use to download and install a package from the command line. It's as simple as this:

moya-pm install moya.sociallinks

You can use the same command to create and upload packages of your own. There's no documentation on creating packages yet, but you can get usuage information with the following:

moya-pm -h

Let me know if you run in to any problems!

Will

read more…

Encrypted Notes with Moya

Encrypted Notes is an application written in Moya, and only the second Moya website online (the first being this one).

http://notes.moyaproject.com/

Although mainly a Javascript application, the back-end is served by the Moya JSON-RPC library, which is built in to Moya. You can see the auto-generated API docs here:

http://notes.moyaproject.com/jsonrpc/

read more…

A BrainF*** interpreter written in Moya

Have you heard of Brainf***? It's an esoteric programming language, with only 8 commands. It's virtually impossible to build anything useful with this language (hence the name), it's more of thing to amuse software engineers.

It may be a nightmare to write Brainf*** code, but the interpreter is remarkable simple. There are interpreters available for most languages, so I couldn't resist implementing one in Moya Code. It was actually a useful test of a few control structures. Here it is, in all it's dubious glory...

<moya xmlns:let="http://moyaproject.com/let">

    <macro docname="bf">
        <let memory="[0]*30000" cp="0" ip="0" in="0"/>
        <list dst="output"/>

        <repeat>

            <let c="program[ip]"/>
            <break if="missing:c"/>

            <switch on="c">
                <case>&gt;</case>
                <let cp="(cp + 1) % len:memory"/>

                <case>&lt;</case>
                <let cp="(cp - 1) % len:memory"/>

                <case>+</case>
                <set-item src="memory" index="cp" value="memory[cp] + 1" />

                <case>-</case>
                <set-item src="memory" index="cp" value="memory[cp] - 1" />

                <case>.</case>
                <append src="output" value="chr:memory[cp]" />

                <case>,</case>
                <let v="input[in] or 0" in="in+1"/>
                <set-item src="memory" index="cp" value="v"/>

                <case>[</case>
                <if test="memory[cp] == 0">
                    <let braces="1"/>
                    <while test="braces">
                        <let ip="ip + 1" c="program[ip]"/>
                        <switch on="c">
                            <case>[</case><let braces="braces+1"/>
                            <case>]</case><let braces="braces-1"/>
                        </switch>
                    </while>
                </if>

                <case>]</case>
                <let braces="1"/>
                <while test="braces">
                    <let ip="ip - 1" c="program[ip]"/>
                    <switch on="c">
                        <case>[</case><let braces="braces-1"/>
                        <case>]</case><let braces="braces+1"/>
                    </switch>
                </while>
                <dec dst="ip"/>
            </switch>

            <inc dst="ip"/>

        </repeat>
        <return value="join:output"/>
    </macro>

    <macro docname="main">
        <!-- BF programs are tricky to quote in XML due to their use of > and < -->
        <str dst="example"><![CDATA[++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.]]></str>
        <if test="not program">
            <echo>A B*******k interpreter (http://en.wikipedia.org/wiki/Brainfuck)</echo>
            <echo>Run with --let program="BF program"</echo>
            <echo>e.g.</echo>
            <echo>  moya run bf.xml --let program="${example}"</echo>
            <echo>Set input with --let input="my input"</echo>
            <exit/>
        </if>

        <call macro="bf" let:program="program" let:input="input" dst="output"/>
        <echo>${output}</echo>
    </macro>

</moya>
read more…

Moya is a new web development platform written in Python

In this post I'm going to try and pitch Moya to you, Dragon's Den style. Moya has been my hobby project since 2011, and it's about time to share it with other developers. If my pitch is successful, I hope you will invest a little time to try it out.

Moya is a Python project which you can use to build a web application, but it would not be technically correct (the best kind of correct) to call it a Python web framework. This is because although although Moya is built with Python, it's not Python you use to create content. Moya has a language of its own for that, know as Moya Code.

Show Me the Code!

For the show me the code developers, here is an example I wrote to solve the challenging Fizz Buzz problem:

<moya>

    <macro docname="main">
        <for src="1..100" dst="number">
            <switch>
                <case if="number % 15 == 0"/>
                <echo>FizzBuzz</echo>

                <case if="number % 3 == 0"/>
                <echo>Fizz</echo>

                <case if="number % 5 == 0"/>
                <echo>Buzz</echo>

                <default-case/>
                <echo>${number}</echo>
            </switch>
        </for>
    </macro>

</moya>

If you're not a fan of XML, you might just be reaching for the close tab button about now, but hear me out. The code above is possibly verbose due to the XML syntax, but quite readable. You will recognize it at has at least statements, expressions, loops. It is actually a complete language, with the data types and constructs you would expect from a modern language.

Moya Code can be used in a standalone fashion (you can run that script from the command line with moya run fizzbuzz.xml) but it isn't going to replace Python or your go-to scripting language. Moya's sweet spot is high-level glue code between other systems. The kind of code you write day to day when building web applications.

This next snippet is a view (called to generate a response from a URL), which creates a new post in this very blog:

<view libname="view.new-post" content="#content.edit-post" requires=".permissions.admin">
    <forms:get form="#form.newpost" dst="form"/>
    <forms:validate src="form">
        <db:create model="#Post" obj="form.data" dst="post"
            let:user=".user"
            let:slug="form.data.slug or slug:form.data.title"
            let:published_date=".now"
            let:published="form.data.action == 'publish'"/>
        <redirect name="list"/>
    </forms:validate>
</view>

There is a lot of functionality packed in to this snippet. I'm going to gloss over the details (there is documentation for that), but in essence the above code performs the following steps:

  1. Check if the user is an administrator. If not respond with a forbidden error
  2. Get a form object (used to render, validate and process HTML forms)
  3. Validate the form if the current request is a POST request
  4. If it validates, create a post object in the database from the form data, generate a slug, set the published date and the current user
  5. Redirect to the recent posts page

Virtually one tag per step, and still quite readable.

Why Use Moya Code?

Hopeful I've made the case that you can build a web app with nothing but Moya Code, but why would you want to? Moya Code does eliminate a few of the minor annoyances with Python web frameworks. You don't have to import things in Moya (quick, what is the import statement for Django's reverse method?); there are no issues with circular imports; and very little in the way of boilerplate code; but these are somewhat incidental benefits.

Moya can detect many errors in your code before the server even runs. It can do this because Moya knows more about your project up-front than a Python web application would be able to. So you don't always need to execute the code to find bugs.

For errors that do occur at runtime, Moya's debug library displays nicely syntax highlighted stack traces, with plain English error messages. And the errors pinpoint the bug, rather than leaving you to decipher a traceback that may have occurred as a side effect of the real problem. Error detection and reporting is a focus of Moya, because it doesn't matter if the fix takes seconds, when finding the bug takes minutes. The minutes can add up to hours in a day, and days or weeks in a project.

Then there are Moya's expressions, which are superficially similar to Python and other high level languages, but have built in syntax for web related things. Here's an example that gets the current price of Bitcoin from an online service and displays it in the terminal:

<let btc="(fromjson:get:'https://www.bitstamp.net/api/ticker/')['ask']"/>
<echo>1 BTC is currently worth ${btc} dollars:</echo>

Here's another example, this time in Moya's template language. The following snippet generates a link to the next page, where the current page number is stored in the query string:

<a href="${urlupdate:(page=page+1)}">Next Page</a>

The link generated by this expression preserves the other values in the query string. So if you are rendering a page with a query string of ?q=tuna&sort=price&page=2, the above expression will return ?q=tuna&sort=price&page=3. Implementing this is not particularly difficult in any language, but in Moya, it just works and it is always available (in Moya code or templates)

Data types in Moya are designed to be as expressive as possible, in that you can glean a lot of information from them with attribute access. Here's an example that uses an expression to render an unordered list with the dates in the current month:

<ul>
{% for day in (.now.month_start)...(.now.next_month) %}
	<li>${localize:day.date}</li>
{% endfor %}
</ul>

The expression in the {% for %} tag creates an exclusive range between the date of the start of the month and the end of the month. Again, this is not something that would challenge most programmers, but in Moya it's effortless, and leaves you to focus on the more interesting problems.

But I Really Want to Use Python!

I don't blame you -- I love Python as a language, but in Moya, Python is generally only used to add new features. Most of Moya's built in libraries don't contain any Python code. For instance, the blog and comments system for this site is entirely Moya Code.

For situations where you do need to implement something in Python, Moya offers a number of ways to interface your Python code with Moya Code. The following example was taken from a URL shortening application, which encodes a primary key in base64 to save a few characters:

import moya

alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-"

@moya.expose.filter('encode_id')
def pack(i):
    letters = ''
    while i:
        letters += alphabet[i % 64]
        i //= 64
    return letters


@moya.expose.filter('decode_id')
def unpack(s):
    i = 0
    for c in reversed(s):
        i = i * 64 + alphabet.index(c)
    return i

This Python code creates two filters, one to encode an ID and one to decode. It could have been done in Moya Code, but the Python was a better fit for this case.

Here's the view that handles a short URL and redirects to the original URL (note the use of the 'decode_id' filter):

<view libname="view.redirect">
    <db:get-required model="#ShortURL" let:id="url.urlkey | 'decode_id'" dst="shorturl"/>
    <inc dst="shorturl.visitcount" />
    <redirect-to url="${shorturl.url}" />
</view>

If you are interested, the rest of the code is on GitHub. Other than the filters, the entire URL shortener application is implemented in a single file.

It is also possible to create entirely new tags or language features with Python extensions. But I'll save that for another post.

Content is King

Another of Moya's systems that could make a big impact on your work-flow is content. This is Moya's answer to the tag soup that is often the result of even a well designed system that uses only templates.

Content is a high level description of a page. It is an XML definition that specifies what will be rendered in a given page, but not how it should be rendered. Separating concerns in this way helps to keep your code manageable.

Let's have a look at some content. The following was taken from Moya's built in FAQ application:

<content libname="content.faqlist" template="faqs.html">
    <title>Frequently Asked Questions</title>
    <section name="body">
        <for src="faqs" dst="qa">
            <faq:faq qa="qa" />
        </for>
    </section>
    <section name="admin_links" if=".permissions.admin">
        <w:link url="${.appurls.createfaq}" icon="star">New question</w:link>
    </section>
</content>

Without going in to the details we can see this content definition divides the page in to a number of sections. A section typically corresponds to an area on the page, which could be a column, a footer or a non-visible area such as CSS in the head tag. The tags inside sections are responsible for generating HTML in the page.

Note the use of a tag called <faq:faq>. This is an example of a custom widget. Widgets ultimately render a template with the data you pass via attributes, but they also have the capacity to process data and pull in CSS, Javascript, and other external files entirely automatically.

Because widgets are self contained, you can easily move them from one part of the page to another or even to a different page entirely, and they will work as expected. No cutting and pasting markup, fixing URLs, or copying media required.

Let's look at another widget. Moya has a rich text editor widget which uses WYSIHTML5. This is how you would replace a textarea in a form with this editor:

<field name="text" label="Text" src="text">
    <wysihtml5:editor/>
</field>

The field tag is part of Moya's form system (I'll leave that for another post). The presence of the <wysihtml5:editor> tag pulls in two external CSS files, two Javascript files, and generates an embedded script tag to set things up. The result is that the form contains an editor that supports bold, italic, links etc.

What Next?

There is plenty more I could cover, but this is a good point to end my pitch. I'll leave the form system, database expressions, JSONRPC support, templated email, and the automatically generated admin site for another post. In the mean time, you might want to check out the tutorial or go straight to the documentation.

read more…