About me
RSS Links
Join 6,651 other subscribers
Prof. Jayanth R. Varma’s Financial Markets Blog
A blog on financial markets and their regulation (currently suspended)
The SEC and the Python
Posted by on April 16, 2010
Last week, the SEC put out a 667 page proposal regarding disclosures for asset backed securities. What I found exciting was this:
We are proposing to require that most ABS issuers file a computer program that gives effect to the flow of funds, or “waterfall,” provisions of the transaction. We are proposing that the computer program be filed on EDGAR in the form of downloadable source code in Python. … (page 205)
Under the proposed requirement, the filed source code, when downloaded and run by an investor, must provide the user with the ability to programmatically input the user’s own assumptions regarding the future performance and cash flows from the pool assets, including but not limited to assumptions about future interest rates, default rates, prepayment speeds, loss-given-default rates, and any other necessary assumptions … (page 210)
The waterfall computer program must also allow the use of the proposed asset-level data file that will be filed at the time of the offering and on a periodic basis thereafter. (page 211)
This is absolutely the right way to go particularly when coupled with the other proposal that detailed asset level data be also provided in machine readable (XML) format. For a securitization of residential mortgages for example, the proposal requires disclosure of as many as 137 fields (page 135) on each of the possibly thousands of mortgages in the pool.
Waterfall provisions in modern securitizations and CDOs are horrendously complicated and even the trustees who are supposed to implement these provisions are known to make mistakes. A year ago, Expect[ed] Loss gave an example where approximately $4 million was paid to equity when that amount should have been used to pay down senior notes (hat tip Deus Ex Macchiato).
Even when the trustees do not make a mistake, the result is not always what investors had expected. A few months ago, FT Alphaville reported on two Abacus deals where the documentation allowed the issuer (Goldman Sachs) to use its “sole discretion” to redeem the notes without regard to seniority. People realized that this was possible only when Goldman Sachs actually paid off (at face value) some junior tranches of these CDOs at the expense of senior tranches.
When provisions become complex beyond a point, computer code is actually the simplest way to describe them and requiring the entire waterfall to be implemented in open source software is a very good idea. The SEC does not say so, but it would be useful to add that if there is a conflict between the software and textual description, the software should prevail.
Now to the inevitable question — Why Python? The SEC actually asks for comments on whether they should mandate Perl, Java or something else instead. I use Perl quite extensively, but the idea that Perl is a suitable language for implementing a transparency requirement is laughable. Perl is a model of powerful but unreadable and cryptic code. As for Java and C-Sharp, there is little point in having open source code if the interpreter is not also open source. I do not use Python myself, but it appears to be a good choice for the task at hand.
It is gratifying that the SEC continues the one good thing that Cox initiated when he was Chairman – the use of technology as a key regulatory tool.
Prof. Jayanth R. Varma's Financial Markets Blog 
So Python and XML? Hate to see what the average investor’s site-packages directory will look like when they finally download all the different Python XML parsing libraries that will inevitably be used.
Red herring. Installation of python libraries is easy.
Also, there aren’t that many Python XML parsing libraries…
Also, based on the size of these proposals, the only one they’ll use is cElementTree. >_>
“technology as a regulatory tool”, what a concept! Pretty impressive
change we can believe in. Here’s hoping open-source human-readable
computer programs become the standard for verifiable elections and all
sorts of financial uses. Otherwise, computers will end up being used
to defraud the populace rather than to serve it.
I can already read the first line:
from random import *
Python has a background in scientific computing, aka very high performance libraries (in C) designed for number crunching. Check out Scipy and Numpy. Plus Python is really easy to learn so there is less of an excuse of “programming language foo is hard, waaa!” and easy to validate/examine (thanks to enforced white space/etc.).
“Perl is a model of powerful but unreadable and cryptic code”
That, sir, is a bunch of malarky. That, sir, is your fault and not the Perl languages fault. Any language, including Python, when programmed in a half-ass way becomes “unreadable and cryptic”. Don’t fault the language. Fault the programmer.
Any language, including Python, when programmed in a half-ass way becomes “unreadable and cryptic”.
===
Yes, any language does. SOME languages though, cough Perl cough, are particularly prone to that, due to having a bad syntax to begin with.
Do you seriously insist (a) or (b):
(a) All programming languages have equally clear syntax?
(b) All programming languages do not have equally clear syntax, but Perl’s is as good as Python?
You can write modern Perl code which is as clear or clearer than Python. But you can also write ancient-style Perl code which looks like crap.
It’s like programming in C++ but sticking to the old C ways – the language allows it, but it’s definitely not recommended.
The important point is: Perl has evolved in the past few years, you should take a look in case you’re interested.
To be honest, stating that modern Perl is unreadable makes people look a bit silly to me.
However, I do agree that Python is a better choice in this case. Of course, for a very different reason: Perl is much more dynamic and evolving these days, while Python seems more “static” and thus more suitable for what will be a legal requirement.
Although it’s true that “Any language, including Python, when programmed in a half-ass way becomes ‘unreadable and cryptic'”, Perl is unique as a “write-only language”. When Larry Wall designed it, he seems to have taken every language construct he’d ever encountered in Unix and tried to throw them all into a single language. Google the phrase, “There’s more than one way to do it”. That’s Perl’s motto. And that’s what’s wrong with Perl.
In 1980 I learned C by reading the K&R book. It took me about 3 days, and by the end of the week I had a high level of confidence that if someone handed me C source code, I’d be able to recognize all but the most obscure syntactic constructs without having to refer back to the manual. I’ve written Perl on and off for over 15 years, and yet I’m still constantly encountering constructs in other people’s code that I have to look up in the manual.
Every Perl programmer codes in his own unique subset of the language, so it’s virtually guaranteed that you’ll encounter syntax you’re unfamiliar with when reading someone else’s code. The Perl “There’s more than one way to do it” mantra is a great philosophy for creative writing. It’s a lousy philosophy for designing a programming language.
Any decent, up-to-date Perl programmer can understand any of the currently used idioms and also the previous ones. Most of these up-to-date Perl programmers currently write code using the same idioms.
Perl is not a language which you learn once and you’re good to go. You need to constantly relearn Perl as it’s an evolving language.
Good Perl code written today is very different than good Perl code written 5 years ago. And some Perl code which was considered good 15 years ago might even be considered a steaming pile of antiquated junk by today’s standards.
As I’ve said, it probably isn’t suited for SEC purposes, as it’s a moving target.
However, saying that Perl has a “lousy philosophy” clearly shows a lack of desire to understand different philosophies.
If you try to read some piece of recent Perl code once every couple of years you’re sure to find new idioms every time. That’s a feature of the language.
I’d argue that the problem isn’t with the language but with you not being able to cope with changes.
R5RS Scheme (let’s forget R6RS ever happened) or maybe even islisp strike me as rather more suitable languages than Python for such things.
Python is simply not standardised in a manner that makes sense for a legal requirement to refer to – basically it’s “whatever CPython does, try and make other implementations (Jython, Iron Python, PyPy etc.) do something vaguely similar”.
You could, however, formally limit the structures that are allowed in the submitted “financial Python”. That wouldn’t absolutely prevent someone from writing in obfuscated code, but it might make it far more difficult to bury a loophole in the details.
It’s very astute of you to point out this gem, and I absolutely agree with the usefulness this approach.
I also like the choice of Python as an expressive, easy to learn language.
I will point out, however, that this: “…there is little point in having open source code if the interpreter is not also open source…” is not really true in two senses.
In the first sense it’s not correct in the sense of being unnecessary to the idea, because as long as the correct version of the compiler is specified along with the source code, and the tools are verified as the same as the issuer used, it’s the consistent output of the program that is important. In other words, as long as the Python or PERL or C# or Java program produces the same output (and therefore same expectation) for the investor as for the issuer, then the purpose of the idea fulfilled.
In the second sense, both C# and Java have open source compilers available which could be used to implement the idea.
However, I don’t think a statically typed language is necessary for this idea to work, I think Python (CPython in this case) is still a very good choice.
Richard
-1 for Perl. My preference would be for Ruby, but I don’t know if it would be acceptable in this case or not. Couldn’t imagine why not, just don’t know. And yes: I do know Ruby borrows some stuff from Perl, which is equally unreadable. Too many syntax uses in Perl that are not obvious.
Why not Ruby?
Why not LOLCode?
Why not Whirl?
Why not Brainfuck?
Why not VBA?
– sorry, took it a little too far there.
If they are to specify such a requirement there must be a mandate – it appears they plumped for a (set of) coding language(s) over a (set of) modelling language(s) and their choice then may have come down to error likelihood and how common the language is amongst practitioners.
Why mandate a particular language? If anything, use C. It’s (fairly) platform-agnostic.
Because C would make the logic harder to grasp. Python is a very readable language. Someone compared it to executable pseudocode.
Plus one for Ruby. It reads extremely naturally and yields itself well to embedded languages, which make code more descriptive. It’s not as declarative as say, Haskell, but it’s friendlier to a broader variety of programs while still having a more declarative feel than Python.
I meant to say: friendlier to a broader variety of programmers
Why is a more declarative feel an advantage?
(After your first post, I was wondering if you might be Guy Steele, noted language guru and author of the Java standard. He used to use the login and nom de plume Quux back in the 1970’s… But he wouldn’t need to ask this question).
Imperative vs Declarative is NOT a matter of “feel”. It is a fundamental concept in computer science. You can think of it as being “program” vs “data”.
Imperative means — it’s a program, and you have to run it to find out what it does.
Declarative, on the other hand, means that, rather than being a program you execute, it is information about something. A program can take that information, and tell you the answer to a question, much like with an imperative program — but another program could take that same information, and do quite a different analysis.
Further, there is a major difference in what can be said. Imperative programs cannot, in general, be fully analyzed — you can’t even tell if they will ever complete. That’s because they are “Turing equivalent” — fully powerful.
Declarative information, on the other hand, is more constrained. While you can describe a non-halting program declaratively, since you’re not executing it, it does not present the same kinds of problems.
This is a MUCH deeper difference than just C++ vs Java vs Python. Basically, what I (and a few others here) are saying is that it should not even BE a program.
Given a declarative description of the behavior, and a suitable program, you can still achieve the same results. But you can also do many, many other things. Search for certain characteristics, for example. Apply compliance checking rules to see if there are any conditions under which this contract would fail to meet regulatory requirements — or which a pool of them might push an institution out of regulatory compliance.
That’s simply not possible with imperative programs.
This strikes me as very similar to the work that was actually done here: http://research.microsoft.com/en-us/um/people/simonpj/papers/financial-contracts/contracts-icfp.htm
Of course, the language there is Haskell, which made sense because Haskell has some pretty strong advantages as a base for EDSLs, which was the goal of that project. They wanted to build a language embedded in Haskell that could actually be used by financial brokers to describe their own contracts, so there couldn’t be breakdowns where the contract actually sold doesn’t agree with the description. Not sure how the goals here are similar or different.
I posted this
http://twitter.com/sclopit/status/12501736501
just minutes before reading your comment. What I find promising about the Haskell combinator approach is that it may help reducing contracts to a canonical form. and the canonical form may itself be easier for humans to read.
While I think Python is a satisfactory choice from a purely language standpoint, I share the concern of others around standardization. The lack of a standard could result in a rather gnarly legal mess, should an investor’s choice of interpreter produce a different result than the one the trustee uses.
If you want a purely interpreted language, Scheme would seem to be a good choice. Widely known, with many free implementations, and an actual de jure standard.
If even wider familiarity with the language is desired, Java would be the obvious choice. Sun has made it available under GPLv2, so the argument that it is not open source is obsolete, if it were ever relevant.
Frankly, though, the attraction in this case of a de jure standard surely outweighs whether interpreters are open source or even how widely familiar the language is. Scheme has used as a language for computer science instruction, so decades of CS graduates will be familiar with it from their coursework, even if they’ve never used it professionally.
Let me differ with Mark in his suggestion of C — C is extremely platform GNOSTIC, though it can, with care, be used in a platform-agnostic way. C, and by extension, C++, would be extremely poor choices, because of the potential for memory allocation errors and corruption, which could go undetected except under specific circumstances. (Otherwise, the presence of a de jure standard would be attractive).
Java has a standard, but it is administered by a commercial entity (Oracle) rather than a formal standards body. I don’t see this as a major shortcoming, though others might.
Hi Bob,
just like all the other languages – standardised or not, You would state the version of Python used and compile options etc – everything from the ‘python -V’ output.
Standards change and implementations are knowingly/unknowingly different to the standard. You would end up doing exactly the same thing if you used Algol or ADA, or .
By choosing Python, they buy into a culture where obfuscation is frowned on, The Zen of Python, and code readabilty are guiding principles of the developers.
I think Python is a good thing for the financial world – if only the Zen were adopted for their financial transactions.
Paddy.
PEP8 compliance could be mandated
Totally agree guys. Keep it up.
Actually, I just had another thought.
I’d like to argue against using a procedural language for this.
I think using a language is a wonderful idea. I’m not being critical. However, I think a suitably-designed, purely declarative model may be substantially superior.
One aspect is that it allows us to control just what kind of stuff can be specified with this language. That is, it more tightly CONSTRAINS the range of possible behaviors. It doesn’t constrain them in any particular way — but we have to decide to allow them by introducing declarative vocabulary, rather than just anything you can code up.
A big advantage of this is that it will facilitate formal verification and analysis. In particular, it will allow auditing for regulatory compliance, as well as allowing users of this information to determine in an automated way whether the behavior meets their criteria. It will allow automated tools (such as search engines) to categorize these offerings, or produce automated comparisons not solely based on a specific assumptions, but rather on more abstract and general criteria (such as not having a particular trait at all, rather than whether it happens within a particular date range in a particular set of assumptions).
For example — does the face value ever drop below what I paid?
This would also allow various non-numeric assertions to be captured, such as the type of underlying investments, fund asset allocation rules, etc.
There would still be a computer program that allows the investor to do the kind of explorations envisioned by the original proposal here. The difference is, that the program would not be specific to the investment company, but rather common to all, and operating to a standard.
The published data would be in XML format to allow a variety of tools to access it.
I think this information would be FAR more valuable, when viewed as knowledge to be represented, rather than just a program you can download and run.
Clearly this question does not require a declarative model.
It couples the investigator / investor too closely to the issuer. A procedural programming language relaxes this coupling to a useful extent, and a procedural language like python still allows enough expressibility to answer interesting questions in a reasonable amount of time.
The difference between a sufficiently high level language like python and a sufficiently complete domain-specific language is more and more a matter of hair-splitting.
Pingback: Fighting the Man With Python « The Committee of Public Safety
As someone who had the responsibility in the past to audit CDO “waterfalls”, I can tell you that the tool we used most was Excel. The intermediate calculation layers that occur in Excel give more drill-down capabilities and visibility of the logic for non-IT users of the information. A standardized disclosure in the form of a program would be a blessing for auditors but care has to be taken that some structure in these programs is also standardized.
http://www.python.org/dev/peps/pep-0008/ should help
In no way should a spreadsheet be used for this task. They are opaque, and very hard to audit!
If the intent is that investors will be running the program and need to interact with it to enter parameters, then perhaps JavaScript would be a good choice. An HTML file with embedded JavaScript can be run locally in most browsers. Languages like Python will require the investor to have their runtime installed. Also, an HTML form is a pretty natural way to do the input.
Of all the languages people have suggested, JavaScript is the only one that is ready out of the box on nearly every desktop computer, and even many mobile devices. Python, Perl, Ruby, and so on are add-ons for many.
http://www.hunlock.com/blogs/The_Complete_Javascript_Number_Reference
I don’t think JavaScript is good for finance.
Numeric data types, and it’s various issues with NaN’s.
This should definitely be in a declarative ( but executable) form of some kind.
I vote that the SEC acquires and open sources Lexifi.
http://www.lexifi.com/ . It is the evolution of SPJs contract work
nearly a decade on in OCaml.
A great piece of work, but due to the structure of banks IT departments and quant teams, politically impossible to sell.
Although, the python thing might lead to a lot of interest in PyPy.. I certainly wouldn’t like to have the horrors of CPython in the middle of my inner loops.
Thanks for mentioning us!
Indeed, our approach seems to be really in line with the SEC proposal: we propose to describe complex (or simple) derivatives in a domain specific language, made of a few building blocks and combinators. That way, we have formal description of terms and conditions of contracts, which could, if needed, be understood by humans. More importantly, the description of contracts are just data, which can be exchanged between institutions (e.g. in XML format) and analyzed in a precise way by various and independent tools.
Based on our DSL, we have implemented various “universal” treatments (that can be applied to any contract represented in the DSL): pricing, life-cycle management, all kinds of reporting and risk analysis tools. Of course, we expose that to the end users in regular GUI applications, but the same descriptions of contracts can also be used as executable “electronic term sheets”.
And this is not impossible to sell to IT departments and quant teams! Just a little bit difficult 🙂
I’d be concerned that a) now there are two things to learn – OCaml and the DSL; b) the DSL couples the investigator / investor too closely to the issuer.
A more general-purpose programming language relaxes this coupling to a useful extent, and a language like python still allows enough expressibility to answer interesting questions in a reasonable amount of time.
The difference between a sufficiently high level language like python and a sufficiently complete domain-specific language is more and more a matter of hair-splitting.
Martin: you don’t need to learn OCaml. The contracts expressed in the DSL are just pieces of data, which you can manipulate in the language of your choice.
The problem with expressing contracts as programs in a general-purpose language (be it Python, OCaml, or anything) is that the only operation you can mechanically apply to a program is to run it. Inspecting the program for other purposes requires heavy artillery. The purpose of the DSL is to make these operations easier.
As a concrete example, if we have a contract described in our DSL, it is straightforward to answer questions like “What are the possible currencies for future cash flows?”. If the contract were described in a general purpose language, we would need at least to use a parser for this language, an then perform non-trivial data-flow and control-flow analysis to get some rough approximations of the call the the “cash flow” function.
And this assumes we have a well-specified interface for things like the “cash flow” function. If each program uses its own convention to report cash flows, or produces this info in the form of, say, an XML document, then any hope of doing any non-trivial analysis is lost.
You highlight a number of problems in the space, but I’m not sure how you’ve clashed with the assertion that the difference between python and a DSL is a matter of degree; for example, you state:
and:
and then contrast this scary 🙂 state of affairs with a contract expressed in a DSL, where presumably without “parsing” it, it is:
How, precisely, does one achieve this answer without parsing the contract definition?
I’m sure we both can think of clear python that makes it just as “straightforward” given a few reasonable assumptions, and can also think of a confused DSL that makes it quite obfuscated without getting too pathalogical.
Thus I see a good case for saying that a good DSL is sufficient to aid processing, but no case for saying it’s necessary. And when we consider why it’s not necessary, I find it hard to miss concluding that “the difference between any sufficiently high level language and sufficiently broad DSL is a matter of degree”.
I would love to understand how far LexFi’s come since your paper ( http://www.lexifi.com/downloads/MLFiPaper.pdf ), but I have yet to see an existence proof for a useful financial contracts DSL that obviates the need for understanding the underlying programming language (or descends into hairbrained complexity, negating any benefit of the DSL). For example, while I admire the expressiveness of the simple, quite minimal set of combinators discussed in that paper, try explaining “give (c1 ‘or’ c2) != (give c1) ‘or’ (give c2)” with descending to the underlying “max” definition. Trying to get away without, say “min” (sure it’s roughly -max(-a, …)), is going to really make one annoyed after a while. And once you’ve got min, or defined “max” as “or” to appease some boolean-logic afficionado, exactly when do you decide you’ve got a half-decent general purpose programming language on your hands? Again, I don’t see how you’re going to be able to claim many of the advantage of the non-general-purpose quality trumpeted by the term “DSL”.
The how what advantages does your DSL offer above, say, XBRL? Is just the same idea, but perhaps with a better ontology and tools?
…sorry, the bit “with descending to” should of course read “without descending to”
Of course it is. A badly designed DSL can become as difficult to analyze as a general purpose language, and a sufficiently controlled restriction of a general-purpose language (with a limited set of allowed constructions, no call to external libraries, and well-specified APIs for the domain of interest) is in effect a DSL.
That’s quite simple. These two contracts are different because “giving the option to choose between receiving c1 or receiving c2” (i.e. the short party decides) is different from “receiving the option to choose between giving c1 and giving c2” (i.e. the long party decides).
Thanks for putting it succinctly. I find it interesting that a) your exposition is not as simple as “think about why -max(a, b) != max(-a, -b)” (though your explanation requires less handwaving); and b) if you try to replace “option to choose” with “choose” in your exposition – it’s simpler and seems more intuitive given the paper’s example where “c1 ‘or’ c2” is explained as “you must immediately acquire c1 or c2 but not both” – I find the exposition even more succinct but now, IMO, a bit less clear:
“giving the choice between receiving c1 or receiving c2″
is different from “receiving the choice between giving c1 and giving c2″
…perhaps making a bit my point elsewhere about the Illusion of Comprehensibility (it’s what you know that ain’t so, and all that).
But if I go further in succinctness, clarity comes back, I think:
“having the choice of receiving c1 or receiving c2″
is different from “having the choice between giving c1 and giving c2″
Finally, even more succinct is your parenthetical: the long party chooses vs. the short party chooses.
But now I think I’m really making my point that in many cases, to avoid applying natural language ambiguity resolution to a precisely-specified DSL that is designed at, least partially, to look like a natural language, one has to effectively compile the DSL to a general purpose programming language. Or, I suppose, reduce it to another, more familiar DSL like -max(a, b) != max(-a, -b) (lest you point out that this is not the precise justification for the inequality: sure, but then we’re back to that you have to understand the nuances of the DSL to avoid the pitfalls of trying to understand it like natural language).
I didn’t pick up on this earlier, but: a) I agree that saying “the cash-flows shall be described in XML” is far from sufficient to make any meaningful improvement, what’s roughly the minimum you think that would be needed? Do we need a non-trivial DSL to describe the cash flows?
I agree that this is a change in the right direction. However I suppose something like Prolog along with constraints would have been much more closer to requirements of the financial transactions and safer to use than python. Also even if it is opensource, I hope that the libraries that are used have the behaviors specified in a watertight way or it introduces subtle means to manipulate the meaning of the code.
In as much that I am a Python advocate (when appropriate for the application requirements) why on earth does the SEC meddle with such things and push Python?
The SEC mandate does not include technology architecture. By getting into the technology domain they are dabbling with tightly coupling a technology with a business application requirement, so to speak.
It gets even more messy when they advocate XML (which, in my opinion, has reached it usefulness). There are indeed other technologies that may be better suited for such things such as JSON and YAML.
The SEC should stick with their mandate of regulation. Nothing else.
I agree, but feel that the complexity of these Asset Based Securities have done immense damage.
They should be banned.
Colin W.
This is very much the right sort of thing to do.
However, I’m slightly surprised as I would have thought they would have chosen a language that has better sandboxing support (javascript?), because people will surely want to be able to do automated fetching & execution.
Don’t get me wrong – I love python.
I agree with Bob; wondering why one would pick an imperative language for this kind of task when the problem space begs to be represented by a language which is declarative and/or functional.
That said, it’s better than nothing; now if only the same was being considered for the US Code.
Pingback: real tangible » Blog Archive » Rules? Where we’re going…
Pingback: Top Posts — WordPress.com
i think this article so help full for me…i will back soon to read more..ono your blog..
http://tamped.info -> fastest news, showbiz news, healt news and headline
http://corhot.info -> fastest web proxy unblock your access
http:_//dragulaz.info -> fastest web proxy unblock your access
I think moof makes a fairly good point with respect to python; potential variance in various CPython versions and possible alternative implementations of Python is a potential cause for concern here.
However, although I love Scheme, I think the same concern applies in slightly different ways. Scheme’s cross-implementation portability is not good, and even within the standard it is easy to write programs that evaluate to two different values on different implementations.
On the other hand, it should not too difficult to avoid these variances by mandating a particular subset of Scheme without side effects and without “eq?”.
Honestly, I think the winner here would be Standard ML: it has a battle-tested formal definition, and no other language comes close to SML’s consistency across a multitude of implementations.
The only technical downside would be SML’s fairly impoverished default numerics; R5RS’s numerical tower (especially exact integers and rationals) would be a win.
But in any case, I think that Python, Scheme, or SML would all be good choices.
Pingback: SEC Proposes Wall Street Transparency Via Python | JetLib News
Pingback: links for 2010-04-19 « Object neo = neo Object
Pingback: Formalising Bonds with the Informal – Untyping
Will it have RSS?
Ratings agencies and most banks have sophisticated (usu. web based) tools to run both credit and cashflow models on securitizations. These are usually scripting based, and with the right permissions users can see/vary all historical data, assumptions, inputs and cashflow scripts.
These would provide the user with a full suite of sophisticated tools “to programmatically input the user’s own assumptions regarding the future performance and cash flows from the pool assets, including but not limited to assumptions about future interest rates, default rates, prepayment speeds, loss-given-default rates, and any other necessary assumptions … (page 210)” as mentioned above.
I think the SEC should mandate that these should be freely available to all investors, rather than forcing everybody down a limited functionality Python / XML route that may not be the best solution.
Pingback: 271208 CHANNEL R FINANCE NEWS | financworld.com
very informative..thanks a lot..james
javajobs.net
Pingback: crankycoder.com » Links for April 21st
I am overwhelmed by the number and quality of comments on this post. Clearly, many of the readers commenting here know more about Python and other programming languages than I do or ever will.
I will respond to only one broad issue raised in several comments: should the SEC mandate a specific computer language? I think it should just as it mandates a specific natural language (English). Imagine how difficult life would be if some SEC filings were in English, some in Esparanto and some in Korean. Mandating a specific computer language has the same benefits.
In fact, the SEC mandates not only that filings be in English; it has for more than a decade now mandated that large parts of the disclosures be in “Plain English”. They even have a handbook on the subject: http://www.sec.gov/pdf/handbook.pdf
I think of a specific programming language and even a specific coding style within that language as consistent with the overall philosophy of disclosures that the SEC has adopted.
Over a period of time, we might even need to move towards a highly configurable and extensible “universal” waterfall program that each issuer would only configure and extend to model its specific waterfall so that investors can understand it very quickly.
The analogy that comes to my mind is that of a program for sorting (for example, quicksort). Most of these programs are universal sorters in the sense that they accept a “comparison function” as one of the inputs and then the same sorting program can sort many different kinds of objects without knowing anything about them.
Come to think of it, a waterfall program is a bit like a sorting program – you sort all the investors on some basis and then pay off the first guy/guys in the list! If there is any cash left, you continue doing this recursively until the cash is all gone.
I think a rigorously defined (in the Comp Sci. definition of “rigorous”) Domain Specific Language would be a better choice than any off-the-shelf, general-purpose programming language.
A representative Python implementation of the DSL interpreter could be a handy tool, but it could not be “the definition”.
The DSL could use some/all of the terms of “Plain English”, but in a rigorously defined, machine readable form.
I think a DSL could certainly work. A DSL also has the advantage of being something that could be implemented in a variety of lower level programs and on any number of platforms. What I don’t like is the idea of a government agency dictating a particular technology of software tool to accomplish a job. It would be like saying that from now on only Philips screws can be used for government construction jobs. There are lots of different kinds of screws, and (I imagine – I’m not a carpenter) lots of reasons to use different screws. All the government should care about is that when used, the screws should hold things together and that generally carpenters should know which screws to use when. So too with software developers.
I blogged more about my thoughts on this here:
http://stevesmithblog.com/blog/government-mandates-and-programming-languages/
Pingback: Another episode of “It’s Charlie’s Stross’s fiction, we are just living in it” : Will Wall Street require Python? | False Positives
Great discussion. Bob Kerns’ idea for a “suitably-designed, purely declarative model” is particularly interesting. There’s been discussion in the XBRL community about whether XBRL is necessary to achieve the validation sought, or whether plain XML is better. This group might be interested in the SEC’s three use cases for XBRL to date, described here:
http://www.sec.gov/spotlight/xbrl.shtml. ABS, with only about 130 concepts (compared to U.S. GAAP’s more than 10,000) would require an XBRL taxonomy the fraction of the size of GAAP. (Not counting the “waterfall” part of the disclosure — don’t know how many elements might be required for that.) But since XBRL is already a global standard for business reporting and the SEC has used it for comparable sized projects, like mutual funds, and smaller projects, like credit rating agencies, the question arises whether the benefits of consistency outweigh the benefits of innovation. Also, there’s been concerned raised about whether investors would be on a level playing field with respect to accessing and processing the information. Charlie Hoffman’s blog has lots more on XBRL if you’re interested: http://xbrl.squarespace.com/
Pingback: » links for 2010-04-22 (Dhananjay Nene)
But Java interpreter IS open source!! Called OpenJDK if you have not noticed.
Pingback: » Tweets for 2010-04-22 (Dhananjay Nene)
For such things, they should use a high programming language like Eiffel which is also open source and that will bring them Design By Contract features for a better accountability.
Most of the comments raise points that are too specific to be helpful or too general to be actionable.
What’s needed from the SEC is:
1) a set of simple principles (e.g., “The programs must meaningfully increase ABS contracts’ accessibility, transparency, verifiability, and processability”)
2) a set of tests thereof (e.g., “The program must be executable by any person with reasonable computing resources e.g., at home on their PC using the standard CPython distribution or at a library via the SEC-hosted testing website”)
2) a standing committee of buy- and sell-side Computer Scientists, Quants, and investors to recommend to the SEC improvements to and compliance with these rules.
Why Python? Isn’t it OBVIOUS? G O O G L E
Pingback: Government Mandates and Programming Languages : Steve Smith's Blog
I am embarrassed by the discussion thread but must post a reply, howsoever brief. 90% of the commentary exemplifies our industry (computer professionals) at our most narrow-minded, namely, our cliche’d bias that any problem can be solved by software, and furthermore, by a particular language.
The “business process” is broken. Even the most elegant Python code cannot compensate for an absence of critical thinking, and an excess amount of bad human behavior. But I can easily imagine how this happened: an SEC analyst asks the IT department if they can fix the problem, and the IT department says yes without gathering full requirements, and when the code is delivered for testing, the analyst says, Does It Work? Of course, the reply is, Of Course, but They must submit the data in the language that we wrote our module, nothing else will do. It is as humorous (sad) as if our Dept. of Education said that we were going to fix all of the education problems in the Afghanistan mountains, then mandate that teachers must use English in the schools we build or the buildings will not work.
There is either a published and documented API for a true Web Service, or there is not. Done properly, any client should be able to interact with the API across the Web.
The issue of large data sets is not lingual, but a performance issue at several layers of the stack. I cannot imagine financial reporting is more complicated than sequencing human DNA, and even with that amount of data, I don’t recall anyone proposing that all institutions working on the Human Genome must use C++, or even better, let’s require Cobol, I think I know someone who still knows how.
What about freedom of speech? I want to write my software in the language of my choice. Python? You have to be kidding. The government shouldn’t be able to dictate this. I want to use the language and skillset that is correct for my business, not have the government dictate it. What’s next, we all have to run Linux? Another step toward socialism (Open Source – FREE) from a government that knows nothing about technology.
In theory, this would be a great idea, but it could never happen. The amount of work to accurately model the cashflows would prohibit the securitization in the first place. Even if the waterfall was simple, modeling the collateral is very non-trivial: imagine writing the code to accurately model the principal and interest payments of a pool of ARM loans given a ramping index rate, a non-linear prepayment curve, defaults, severities, delinquencies, etc. Then, once all of that is modeled, now you can push it through the bond waterfall to see how the bonds pay. What if the deal has triggers? What about modified loans? Wraps and financial guarantees?
Plus, every month this perfect loan by loan information has to be accurately disclosed in a format that will allow it to be plugged back into the model above so the new analysis can be done? Is the issuer going to continue to provide that service long after the deal has been issued?
Look, I’m all for transparency, but that kind of analysis is up to the buyer/seller. Either model it up yourself or pay a 3rd party to do so. But if this is required for every deal issued, I’d expect issuance to grind to a halt.
In as much as I’m currently a Haskell advocate, I’m not convinced that it would be the best choice from a readability standpoint. However, giving people the illusion that they understand the code just because it appears readable isn’t necessarily the right goal (see COBOL).
Something I haven’t seen mentioned is MATLAB, in particular Simulink. Yes, I know, it’s OUTRAGEOUSLY expensive from a consumer’s standpoint. That said, I’m under the impression that the target market for this would be institutions, at which point MATLAB costs would amount to noise (assuming they didn’t already have licenses).
Certainly it doesn’t have all the touchy-feely-good-vibes of FOSS (dude!), but I see it as a much better solution to the problem.
@D: So you’re saying these deals are too complicated to model, so let the buyer model it?
#FAIL
If issuance of deals too complicated for buyers to understand grinds to a halt, I shall not shed a tear.
However I believe the core issue you are pointing to is that the model has to be properly modularized. A single monolithic model that embodies both the waterflow and the collateral would be a problem.
But I think what’s needed is a model for the waterfall, including all the contractual aspects, that the user can plug in his own modified collateral model.
I don’t pretend to really understand the structure of these deals. But I believe part of the value here lies precisely in limiting the amount of complexity and obscurity that can be introduced. That’s a part of why I favor a declarative approach.
Using a general programing language, you can specify a program which is so complicated that you cannot prove it will ever give you an answer.
What’s needed is a way that there can be a meeting of the minds between buyer and seller, so that they’re on an equal footing.
Let’s say each ARM in the pool had its own model, in some standardized form that allows for composition. This could be a matter of declarative descriptions, or a program with a standardized interface.
Let’s say the collateral pool’s structure also has a model. Compose this model with the ones for the underlying ARMs
Compose this with the waterfall.
If you want to select half the ARMs with some characteristic, and subject them to an alternate model of your choosing, you can can do that.
If the servicers of the ARMs provide performance data, and the packager of the deal aggregates it, this is something you only have to set up once. The ongoing burden is negligible — assuming everyone follows through.
In the real world, some banks handling the ARMs would likely fall out of compliance. Maybe get bought up by some giant, or (gasp) be allowed to fail, and parceled out.
I don’t see that as a major problem. Well, it’s a problem, it’s just not a problem with this system. You either extrapolate from stale data (“still current”, “still in default”, etc.) or substitute a better model (perhaps a transition probability matrix) of the projected course of the loan, that captures how you’d handle the missing information in your analysis anyway.
What I’m getting at here, is that if you can’t model the contractual portions, and you can’t obtain and plugin in ongoing performance data, then maybe you’re buying a pig in a poke, and it’s just too complicated to be allowed to exist.
“it’s just too complicated to be allowed to exist” seems wrong to me. I agree that in some cases these deals are too complicated….so don’t buy them! These are major institutions buying these bonds, not individuals- institutions with time, money, and very bright people working for them who should do whatever research they feel necessary before purchasing.
If modeling these deals is as simple as you seem to think, why would institutions pay 3rd parties like Bloomberg millions of dollars to do it for them?
I am certain that any of the major banks that purchased any mortgage backed securities that subsequently failed had all of the necessary research tools available to them.
Regulating the sell side seems un-American to me. Let the open market decide what’s a good purchase and what’s not. Why did these deals get issued in the first place? Because people bought them up. Not the other way around.
Yes, I fully realize these are institutions buying these. It is institutions which have been the major victims in the news, is it not?
Caveat Emptor is a nice idea. But it presumes the buyer is actually buying what he thinks he is.
I never said I thought modelling these is easy. But tell me, does it make sense for 100 institutions to pay 3rd parties for the same model?
The goal here isn’t to regulate the sell side. The goal here is simply the traditional role of government in enforcing contracts, by demanding that contracts be clear and unambiguous enough that both parties can agree on the operation of the contract itself.
It’s fine if different potential buyers bring different economic assumptions to the table, and I agree that that shouldn’t be the seller’s responsibility.
That’s why my point was about modularity — construct the models in such a way as to separate modelling of the contract being sold, from modelling the assumptions under which the buyer chooses to operate.
I think that’s pretty essential to the idea. I suspect we agree on that point, and probably only disagree about whether it’s feasible to do so.
My point about “too complex to exist” is simply that if the contract itself is too complex to be modeled by the seller (not the economic assumptions), then the seller will be operating blind as well as the buyer. The seller should have a model, and done right, it should be one that can be usefully shared with the buyer, and be basically an executable form of the contract.
After all, there has to be SOME way, at a given point in time, to determine what the contract says should happen. If it’s that complicated, that computers can’t do it, just lawyers reading the contract, where do you think things will end up? The courts!
@D: I think the crux of the problem you’re pointing out (and I would hope that the SEC is aware & planning for this), is that any specification system would have to have an SEC provided and managed library of objects that the contracts would rely upon. There’s no point in reinventing the wheel with each piece of code.
@Bob Kerns: I may be mistaken, but I think your point is allied with my response to D. Also, the halting problem doesn’t require a very complex program at all. Damned Gödel!
I forgot to add (what does wordpress have against “preview” anyway?), the linked paper earlier to the Simon Peyton-Jones research at Microsoft essentially provides for a library from which to draw (the combinator library).
What the DSL proponents fail to consider / adequately weight is that DSLs’s readability benefits come with at least three costs:
1) The Illusion of Comprehensibility: a DSL that is simple is useless, and a DSL that is useful isn’t simple. People may expect something that looks a bit like a natural language to be easily comprehensible, but the precision that we gain from a rigourous definition trip up people that don’t have a detailed understanding of the DSL and how it contributes to the value. If you think you’re immune, read that Simon Peyton-Jones paper that’s been mentioned ( http://research.microsoft.com/en-us/um/people/simonpj/papers/financial-contracts/pj-eber.ps ) and do the “routine” proof that “give (c1 or c2) != give c1 or give c2”. How many people are going to say that using “give” instead of “negation” and “or” instead of “max” is a net win? Check out SPJ’s justifications on page 2: “it becomes much easier to describe … unforeseen contracts” and “we can [where we couldn’t before] systematically analyse, manipulate, and perform computations over”. Such are advantages that a DSL or Haskell combinator library has over C. These are not significant advantages over python, especially when balanced against the…
2) Loss of Imperative: show me one person who checks a computation declaritively and I’ll show you 50 who check it imperatively. While is is easier to describe contracts declaratively, it’s easier to evaluate them imperatively. To make up for this loss, DSLs have to supply a lot in semantic compression. But see point #1. A bad DSL is as bad as a bad imperative implementation; and don’t forget the…
3) Loss of Generality: sure, DSLs are great because of semantic compression. But what if the DSL doesn’t allow you to express what you need? You’re back to needing a general purpose programming language. A well-designed API or library in a general-purpose programming language is better than a badly-designed DSL. A complete API or library in a general-purpose programming language is better than an incomplete DSL. And a complete DSL is a pipe dream for this problem space.
Reactions?
Dear all,
Since, I am a true person, I put some undisputable points.
1) We can write clean code in perl
2) With perl Moose, we can write superior OOP in perl compared to python
3) Perl is faster than python on any given day (for 90% of works)
4) Python is also a good language but till such time, it has faster implementation without using a C extension, it will not be used by industry
5) 99% of all programming projects that get outsourced to India don’t mention use of python. So, we may assume (may or maynot be true) that a vast majority of projects in the world don’t involve python.
6) Stick to one programming language for life and learn perl or python depending on ur convinience. Says perl is bad is an ugly joke.
Cheers
sorry! I point 6, I meant “Saying perl is bad is an ugly joke”
It should definitely be in Flash. ActionScript. No “reverse engineering” allowed. It’s not your business how the software works, you silly investors. Only Adobe and “software developers” need to know that information.
Alternatively, it could be in Javascript. But it will only work correctly if you use ____ browser.
Why not just give investors the machine readable language and let them decide what to do with it? If the investor wants to buy a program to display the data a certain way, then that’s her choice. If she wants to use someone else’s open source interpreter (e.g. Python), then that too is her choice. Or maybe she’ll just use UNIX tools and a shell.
Is the SEC is trying to create jobs for Python programmers? Will every company need to employ or outsource Python coders to write these apps?
Very strange idea, singling out an interpreter. Why not use Tcl? Why not learn to use a UNIX shell?
You see, the problem is personal preference. Making the choice for investors is pushing someone else’s preference on investors.
Investors just need data. Maybe they don’t want to use Python. (Though it wouldn’t hurt because Google makes such heavy use of it.)
But I admire the SEC for their continuing attention to giving investors machine readable data. Good show.
correction: change “machine readable language” to “machine readable data”
Very cool news; now I’m trying wrap my head around MBAs using list comprehension constructs and generators.
more on http://financemarket.iblogger.org free back links and finance markets news
Pingback: links for 2010-05-29 « Wild Webmink
The amount of rent escalation is written in the agreement, lease must meet or exceed the expected increase in the level of expenditure in the coming years, the landlord is otherwise going to lose money
My core issue with using something like Python for SEC contracts is that it’s not restrictive enough. I’ve had a moderate amount (~20 years) of experience with scripting languages used to test software systems. In every case, testers will flex the language to its full extent in order to save themselves time. Additionally, they will flex the language in order to make their dreary work more enjoyable. I have never seen an exception to this observation (although, more likely than not, I personally would have written a tool to generate the script — only because that was even more fun than flexing the language).
To me, the bottom line is that, in order to create a system that is both understandable and correct, one has to create a very limited language that is targeted to the problem at hand.
Now, I realize this is an extreme example, but here’s a Python script that sets variable = 1. It’s not that I think anyone will go quite this far, but it exemplifies the issue I have with full-featured languages that target a narrow problem.This example was stolen from http://www.p-nand-q.com/python/obfuscated_python.html
import thread, time, whrandom, mutex
alias for better code readability
x = whrandom.randint
def func(o):
if o[-1]:
o[1][0] -= 1
o[1].extend(o[2])
o[0].unlock()
else:
o[-2].extend([0]*50000)
for j in range(0,len(o[-2])):
o[-2][j] = x(0,100)
while not o[-2][j]:
o[-2][j] = x(0,100)
o[-2][j] = x(0,100) / o[-2][j]
o[0][1].lock(func,[o[0][1],o[0],o[1],1])
this is just meant as a threadsafe way of finding out
how many threads are alive (if I understand the elaborate
manual that is the python documentation on this)
def getcount():
result = [0]
def i(result):
global queue
result[0] = queue[0]
queue[1].unlock()
queue[1].lock(i,result)
return result[0]
testprogram: can be either single-or multithreaded.
singlethreaded = 0
start = time.clock()
if singlethreaded:
queue = [-1,mutex.mutex()]
func([queue,[],0])
else:
queue = [10,mutex.mutex()]
for i in range(queue[0]):
thread.start_new_thread(func,([queue,[],0],))
time.sleep(1)
while getcount() > 1:
time.sleep(1)
print getcount()
variable = reduce(lambda x,y:x+y,map(int,queue[4:]))/len(queue[4:])/2
Python?i don’t think is so popular.
There are plenty of visual programming languages, but unfortunately most of them are not free
What do visual programming languages have to do with this discussion?
Pingback: aleatory
An simplified example of what a waterfall program in Python might look like is at http://www.pylaw.org/demonstration.txt .
I cleaned up the python a little bit:
patch: http://www.martindengler.com/proj/sec_python/demonstration.patch
New version:
http://www.martindengler.com/proj/sec_python/demonstration.py
Thank you, Martin. I especially appreciate the replacement of the clunky dict definitions. Your response and improvements illustrate one of the great strengths of open source, the large community of “second pair of eyes,” that seems to organize itself spontaneously to improve whatever code it sees.
Hello,
I am new in blogs world. If someone want to now about blog history:
The term “weblog” was coined by Jorn Barger] on 17 December 1997. The short form, “blog,” was coined by Peter Merholz, who jokingly broke the word weblog into the phrase we blog in the sidebar of his blog Peterme.com in April or May 1999.]]] Shortly thereafter, Evan Williams at Pyra Labs used “blog” as both a noun and verb (“to blog,” meaning “to edit one’s weblog or to post to
one’s weblog”) and devise the term “blogger” in connection with Pyra Labs’ Blogger product, leading to the popularization of the terms.
Pingback: Yves Hilpisch's Blog » Blog Archive » Python for Financial Engineering
Pingback: Python for Financial Engineering « DEXISION
are there enough Python XML parsing libraries???
In perl there are many libraries to parse XML. If it is a big file, Python is going to be slooooooooooooow…….. U can make it fast by pypy, Psycho. Little bit of “C” knowledge is needed. There is also some book on python and XML. I don’t remember the title.
In Indian industry, no body (except google) uses python. PHP, Perl and Java Rock!! here.
Final conclusion, use Perl! for any job !!! Perl 6 “will” “surely” come by Jul 30th 2012 and “will” sweep all.
Till then wait.
Sorry for being off-topic!
Cheers,
Truman
My Quote: ” Englishmen put lot of terminilogies and abbreviations to classify 🙂 but then they catch themselves 😦 “
I am little confused what you said above”It is gratifying that the SEC continues the one good thing that Cox initiated when he was Chairman – the use of technology as a key regulatory tool.”
Can you explain a bit more about it? There is nothing related to technology in that matter. So it is better you explain a bit more.
ECRi7U http://gdjI3b7VaWpU1m0dGpvjRrcu9Fk.com
Otherwise, computers will end up being used to defraud the populace rather than to serve it.
Hi, trying to find the finest about insightful penny stock news letter? .
I JUST NOW have one you need to take check out I make an effort to have the best and fresh information , thus if however this is of involvement to you personally delight check out my penny stock newsletter.
Hello, I want to know more information about this event, thought I would join my job, if you allow it: strange: the number of reported, there may be another report? Start – how much from where. Links to the apartment a place to stay or need a photo. Wrote to me cool, someone who you go …
I’d like to tell you about a change of address young pthc
uayyou
hello!,I really like your writing very so much! proportion we keep in touch more about your article on AOL? I require a specialist on this space to solve my problem. May be that is you! Looking forward to peer you.
Pingback: freelance alliance
Pingback: everything about java
Pingback: Hyip Analyzer - The besthyip monitoring service|HYIP analyzer - The monitor you can trust|Hyip listing made simple with HYIP analyzer|HYIP ANALYZER - the investors paradise|HYIP investment never wa so easy|high yield investment programs honest monitor|hyi
Pingback: Exercises To Strengthen Eyes
International directory enquiries http://idiohufyba.de.tl european model nn mmmm fuck i love to handled and told what to do. mmmm, a d d me i love dirty phone sex.
What’s up to every body, it’s my first visit of this blog; this
website contains amazing and actually excellent data in favor
of visitors.
I visit day-to-day some sites and blogs to read articles, except
this web site provides feature based content.
Pretty section of content. I just stumbled upon your weblog and
in accession capital to assert that I get in fact enjoyed account your blog posts.
Anyway I will be subscribing to your augment and even I achievement you access
consistently rapidly.
Thanks a bunch for sharing this with all people you actually understand what you are speaking approximately!
Bookmarked. Please additionally consult with my website =).
We may have a link trade arrangement among
us
Just desire to say your article is as astonishing.
The clarity for your post is just spectacular and that i
can suppose you’re knowledgeable in this subject. Fine with your permission allow me to take hold of your RSS feed to stay up to date with imminent post. Thank you 1,000,000 and please continue the gratifying work.
Good day! I just would like to give a huge thumbs up for
the nice information you¡¦ve gotten right here on this post.
I will likely be coming again to your weblog for extra soon.
I love to disseminate understanding that will I have built up with the 12 months to help enhance group efficiency.
No doubt diet pills are one of the most effective and easiest means of losing weight. When sugar enters the blood, pancreas produces insulin. Here are some tips for effective weight loss:- Eat break fast daily.
Hello! This post couldn’t be written any better! Reading this post reminds me of my good old room mate! He always kept talking about this. I will forward this write-up to him. Fairly certain he will have a good read. Thank you for sharing!
Thank you a lot for sharing this with all of us you really know what you are speaking about! Bookmarked. Kindly additionally visit my web site =). We will have a hyperlink alternate contract between us
There’s definately a lot to learn about this topic. I like all the points you made.
It’s awesome to go to see this website and reading the views of all mates regarding this article, while I am also zealous of getting know-how.
My brother recommended I might like this blog. He was totally right.
This post actually made my day. You cann’t imagine just how much time I had spent for this information! Thanks!
I’m not that much of a internet reader to
be honest but your sites really nice, keep it up!
I’ll go ahead and bookmark your website to
come back down the road. All the best
Howdy just wanted to give you a quick heads up and let you know a few
of the images aren’t loading properly. I’m not sure why but I think
its a linking issue. I’ve tried it in two different internet
browsers and both show the same results.
Heya i’m for the first time here. I came across this board and
I find It really useful & it helped me out much. I hope to give something back
and aid others like you helped me.
” Vegetarians should eat at least 6 servings per day of vegetables and fruits. To make your experience more worthwhile and fun filled. For times when there’s nothing like home cooked food, but you are bored with your usual repertoire, Cooking Capsules is an app that can help.
Pingback: Ardalis - Government Mandates and Programming Languages | ardalis.com