Xenomorph Blog - Analytics Management

Posts categorized "Analytics Management"

Listening and Context for Big Data

There are (occasionally!) some good questions and conversations going on within some of the LinkedIn groups. One recently was around what use cases there are for unstructured data within banking and finance, and I found this comment from Tom Deutsch of IBM to be quite insightful and elegant (at least better than I could I have written it...) on what the main types of unstructured data analysis there are:

Listening for the first time
Listening better
Adding context

Listening for the first time is really just making use of what you already probably capture to hear what is being said (or navigated)

Listening better is making sure you are actually both hearing and understanding what is being said. This is sometimes non-trivial as it involves accuracy issues and true (not marketing hype) NLP technologies and integrating multiple sources of information

Adding context is when you either add structured data to the above or add the above to structured data, usually to round out or more fully inform models (or sometimes just build new ones).

Posted by Brian Sentance | 10 May 2013 | 2:17 pm

Big Data Finance at NYU Poly

I went over to NYU Poly in Brooklyn on Friday of last week for their Big Data Finance Conference. To get a slightly negative point out of the way early, I guess I would have to pose the question "When is a big data conference, not a big data Conference?". Answer: "When it is a time series analysis conference" (sorry if you were expecting a funny answer...but as you can see, then what I occupy my time with professionally doesn't naturally lend itself to too much comedy). As I like time series analysis, then this was ok, but certainly wasn't fully "as advertised" in my view, but I guess other people are experiencing this problem too.

Maybe this slightly skewed agenda was due to the relative newness of the topic, the newness of the event and the temptation for time series database vendors to jump on the "Big Data" marketing bandwagon (what? I hear you say, we vendors jumping on a buzzword marketing bandwagon, never!...). Many of the talks were about statistical time series analysis of market behaviour and less about what I was hoping for, which was new ways in which empirical or data-based approaches to financial problems might be addressed through big data technologies (as an aside, here is a post on a previous PRMIA event on big data in risk management as some additional background). There were some good attempts at getting a cross-discipline fertilization of ideas going at the conference, but given the topic then representatives from the mobile and social media industries were very obviously missing in my view.

So as a complete counterexample to the two paragraphs above, the first speaker (Kevin Atteson of Morgan Stanley) at the event was on very much on theme with the application of big data technologies to the mortgage market. Apparently Morgan Stanley had started their "big data" analysis of the mortgage market in 2008 as part of a project to assess and understand more about the potential losses than Fannie Mae and Freddie Mac faced due to the financial crisis.

Echoing some earlier background I had heard on mortgages, one of the biggest problems in trying to understand the market according to Kevin was data, or rather the lack of it. He compared mortgage data analysis to "peeling an onion" and that going back to the time of the crisis, mortgage data at an individual loan level was either not available or of such poor quality as to be virtually useless (e.g. hard to get accurate ZIP code data for each loan). Kevin described the mortgage data set as "wide" (lots of loans with lots of fields for each loan) rather than "deep" (lots of history), with one of the main data problems was trying to match nearest-neighbour loans. He mentioned that only post crisis have Fannie and Freddie been ordered to make individual loan data available, and that there is still no readily available linkage data between individual loans and mortgage pools (some presentations from a recent PRMIA event on mortgage analytics are at the bottom of the page here for interested readers).

Kevin said that Morgan Stanley had rejected the use of Hadoop, primarily due write through-put capabilities, which Kevin indicated was a limiting factor in many big data technologies. He indicated that for his problem type that he still believed their infrastructure to be superior to even the latest incarnations of Hadoop. He also mentioned the technique of having 2x redundancy or more on the data/jobs being processed, aimed not just at failover but also at using the whichever instance of a job that finished first. Interestingly, he also added that Morgan Stanley's infrastructure engineers have a policy of rebooting servers in the grid even during the day/use, so fault tolerance was needed for both unexpected and entirely deliberate hardware node unavailability.

Other highlights from the day:

Dennis Shasha had some interesting ideas on using matrix algebra for reducing down the data analysis workload needed in some problems - basically he was all for "cleverness" over simply throwing compute power at some data problems. On a humourous note (if you are not a trader?), he also suggested that some traders had "the memory of a fruit-fly".
Robert Almgren of QuantitativeBrokers was an interesting speaker, talking about how his firm had done a lot of analytical work in trying to characterise possible market responses to information announcements (such as Friday's non-farm payroll announcement). I think Robert was not so much trying to predict the information itself, but rather trying to predict likely market behaviour once the information is announced.
Scott O'Malia of the CFTC was an interesting speaker during the morning panel. He again acknowledged some of the recent problems the CFTC had experienced in terms of aggregating/analysing the data they are now receiving from the market. I thought his comment on the twitter crash was both funny and brutally pragmatic with him saying "if you want to rely solely upon a single twitter feed to trade then go ahead, knock yourself out."
Eric Vanden Eijnden gave an interesting talk on "detecting Black Swans in Big Data". Most of the examples were from current detection/movement in oceanography, but seemed quite analogous to "regime shifts" in the statistical behaviour of markets. Main point seemed to be that these seemingly unpredictable and infrequent events were predictable to some degree if you looked deep enough in the data, and in particular that you could detect when the system was on a possible likely "path" to a Black Swan event.

One of the most interesting talks was by Johan Walden of the Haas Business School, on the subject of "Investor Networks in the Stock Market". Johan explained how they had used big data to construct a network model of all of the participants in the Turkish stock exchange (both institutional and retail) and in particular how "interconnected" each participant was with other members. His findings seemed to support the hypothesis that the more "interconnected" the investor (at the centre of many information flows rather than add the edges) the more likely that investor would demonstrate superior return levels to the average. I guess this is a kind of classic transferral of some of the research done in social networking, but very interesting to see it applied pragmatically to financial markets, and I would guess an area where a much greater understanding of investor behaviour could be gleaned. Maybe Johan could do with a little geographic location data to add to his analysis of how information flows.

So overall a good day with some interesting talks - the statistical presentations were challenging to listen to at 4pm on a Friday afternoon but the wine afterwards compensated. I would also recommend taking a read through a paper by Charles S. Tapiero on "The Future of Financial Engineering" for one of the best discussions I have so far read about how big data has the potential to change and improve upon some of the assumptions and models that underpin modern financial theory. Coming back to my starting point in this post on the content of the talks, I liked the description that Charles gives of traditional "statistical" versus "data analytics" approaches, and some of the points he makes about data immediately inferring relationships without the traditional "hypothesize, measure, test and confirm-or-not" were interesting, both in favour of data analytics and in cautioning against unquestioning belief in the findings from data (feels like this post from October 2008 is a timely reminder here). With all of the hype and the hope around the benefits of big data, maybe we would all be wise to remember this quote by a certain well-known physicist: "No amount of experimentation can ever prove me right; a single experiment can prove me wrong."

Posted by Brian Sentance | 7 May 2013 | 1:46 pm

Spreadsheet control and contagion

Just caught saw a reference on LinkedIn to this FT article "Finance groups lack spreadsheet controls". Started to write a quick response and given it is one of my major hobby-horses, I ended up doing a bit of an essay, so I decided to post it here too:

"As many people have pointed out elsewhere, much of the problem with spreadsheet usage is that they are not treated as a corporate and IT asset, and as such things like testing, peer review and general QA are not applied (mind you, maybe more of that should still be applied to many mainstream software systems in financial markets...).

Ralph and the guys at Cluster Seven do a great job in helping institutions to manage and monitor spreadsheet usage (I like Ralph's "we are CCTV for spreadsheets" analogy), but I think a fundamental (and often overlooked) consideration is to ask yourself why did the business users involved decide that they needed spreadsheets to manage trading and risk in the first place? It is a bit like trying to address the symptoms of a illness without ever considering how we got the illness in the first place.

Excel is a great tool, but to quote Spider-Man "with great power comes great responsibility" and I guess we can all see the consequences of not taking the usage of spreadsheets seriously and responsibly. So next time the trader or risk manager says "we've just built this really great model in Excel" ask them why they built it in Excel, and why they didn't build upon the existing corporate IT solutions and tools. In these cost- and risk- conscious times, I think the answers would be interesting..."

Posted by Brian Sentance | 27 March 2013 | 11:09 am

Data Management for Risk at Mediobanca

Very pleased to announce today that Mediobanca, the leading investment bank in Italy, has decided to select TimeScape as its data management system. You can see the press release here.

Posted by Brian Sentance | 25 March 2013 | 12:58 pm

Analytics Strategy from Numerix

Good post from Jim Jockle over at Numerix - main theme is around having an "analytics" strategy in place in addition to (and probably as part of) a "Big Data" strategy. Fits strongly around Xenomorph's ideas on having both data management and analytics management in place (a few posts on this in the past, try this one from a few years back) - analytics generate the most valuable data of all, yet the data generated by analytics and the input data that supports analytics is largely ignored as being too business focussed for many data management vendors to deal with, and too low level for many of the risk management system vendors to deal with. Into this gap in functionality falls the risk manager (supported by many spreadsheets!), who has to spend too much time organizing and validating data, and too little time on risk management itself.

Within risk management, I think it comes down to having the appropriate technical layers in place of data management, analytics/pricing management and risk model management. Ok it is a greatly simplified representation of the architecture needed (apologies to any techies reading this), but the majority of financial institutions do not have these distinct layers in place, with each of these layers providing easy "business user" access to allow risk managers to get to the "detail" of the data when regulators, auditors and clients demand it. Regulators are finally waking up to the data issue (see Basel on data aggregation for instance) but more work is needed to pull analytics into the technical architecture/strategy conversation, and not just confine regulatory discussions of pricing analytics to model risk.

Posted by Brian Sentance | 14 February 2013 | 2:50 pm

Big Data – What is its Value to Risk Management?

A little late on these notes from this PRMIA Event on Big Data in Risk Management that I helped to organize last month at the Harmonie Club in New York. Big thank you to my PRMIA colleagues for taking the notes and for helping me pull this write-up together, plus thanks to Microsoft and all who helped out on the night.

Introduction: Navin Sharma (of Western Asset Management and Co-Regional Director of PRMIA NYC) introduced the event and began by thanking Microsoft for its support in sponsoring the evening. Navin outlined how he thought the advent of “Big Data” technologies was very exciting for risk management, opening up opportunities to address risk and regulatory problems that previously might have been considered out of reach.

Navin defined Big Data as the structured or unstructured in receive at high volumes and requiring very large data storage. Its characteristics include a high velocity of record creation, extreme volumes, a wide variety of data formats, variable latencies, and complexity of data types. Additionally, he noted that relative to other industries, in the past financial services has created perhaps the largest historical sets of data and continually creates enormous amount of data on a daily or moment-by-moment basis. Examples include options data, high frequency trading, and unstructured data such as via social media. Its usage provides potential competitive advantages in a trading and investment management. Also, by using Big Data it is possible to have faster and more accurate recognition of potential risks via seemingly disparate data - leading to timelier and more complete risk management of investments and firms’ assets. Finally, the use of Big Data technologies is in part being driven by regulatory pressures from Dodd-Frank, Basel III, Solvency II, Markets for Financial Instruments Directives (1 & 2) as well as Markets for Financial Instruments Regulation.

Navin also noted that we will seek to answer questions such as:

What is the impact of big data on asset management?
How can Big Data’s impact enhance risk management?
How is big data used to enhance operational risk?

Presentation 1: Big Data: What Is It and Where Did It Come From?: The first presentation was given by Michael Di Stefano (of Blinksis Technologies), and was titled “Big Data. What is it and where did it come from?”. You can find a copy of Michael’s presentation here. In summary Michael started with saying that there are many definitions of Big Data, mainly defined as technology that deals with data problems that are either too large, too fast or too complex for conventional database technology. Michael briefly touched upon the many different technologies within Big Data such as Hadoop, MapReduce and databases such as Cassandra and MongoDB etc. He described some of the origins of Big Data technology in internet search, social networks and other fields. Michael described the “4 V’s” of Big Data: Volume, Velocity, Variety and a key point from Michael was “time to Value” in terms of what you are using Big Data for. Michael concluded his talk with some business examples around use of sentiment analysis in financial markets and the application of Big Data to real-time trading surveillance.

Presentation 2: Big Data Strategies for Risk Management: The second presentation “Big Data Strategies for Risk Management” was introduced by Colleen Healy of Microsoft (presentation here). Colleen started by saying expectations of risk management are rising, and that prior to 2008 not many institutions had a good handle on the risks they were taking. Risk analysis needs to be done across multiple asset types, more frequently and at ever greater granularity. Pressure is coming from everywhere including company boards, regulators, shareholders, customers, counterparties and society in general. Colleen used to head investor relations at Microsoft and put forward a number of points:

A long line of sight of one risk factor does not mean that we have a line of sight on other risks around.
Good risk management should be based on simple questions.
Reliance on 3^rd parties for understanding risk should be minimized.
Understand not just the asset, but also at the correlated asset level.
The world is full of fast markets driving even more need for risk control
Intraday and real-time risk now becoming necessary for line of sight and dealing with the regulators
Now need to look at risk management at a most granular level.

Colleen explained some of the reasons why good risk management remains a work in progress, and that data is a key foundation for better risk management. However data has been hard to access, analyze, visualize and understand, and used this to link to the next part of the presentation by Denny Yu of Numerix.

Denny explained that new regulations involving measures such as Potential Future Exposure (PFE) and Credit Value Adjustment (CVA) were moving the number of calculations needed in risk management to a level well above that required by methodologies such as Value at Risk (VaR). Denny illustrated how the a typical VaR calculation on a reasonable sized portfolio might need 2,500,000 instrument valuations and how PFE might require as many as 2,000,000,000. He then explain more of the architecture he would see as optimal for such a process and illustrated some of the analysis he had done using Excel spreadsheets linked to Microsoft’s high performance computing technology.

Presentation 3: Big Data in Practice: Unintentional Portfolio Risk: Kevin Chen of Opera Solutions gave the third presentation, titled “Unintentional Risk via Large-Scale Risk Clustering”. You can find a copy of the presentation here. In summary, the presentation was quite visual and illustrating how large-scale empirical analysis of portfolio data could produce some interesting insights into portfolio risk and how risks become “clustered”. In many ways the analysis was reminiscent of an empirical form of principal component analysis i.e. where you can see and understand more about your portfolio’s risk without actually being able to relate the main factors directly to any traditional factor analysis.

Panel Discussion: Brian Sentance of Xenomorph and the PRMIA NYC Steering Committee then moderated a panel discussion. The first question was directed at Michael “Is the relational database dead?” – Michael replied that in his view relational databases were not dead and indeed for dealing with problems well-suited to relational representation were still and would continue to be very good. Michael said that NoSQL/Big Data technologies were complimentary to relational databases, dealing with new types of data and new sizes of problem that relational databases are not well designed for. Brian asked Michael whether the advent of these new database technologies would drive the relational database vendors to extend the capabilities and performance of their offerings? Michael replied that he thought this was highly likely but only time would tell whether this approach will be successful given the innovation in the market at the moment. Colleen Healy added that the advent of Big Data did not mean the throwing out of established technology, but rather an integration of established technology with the new such as with Microsoft SQL Server working with the Hadoop framework.

Brian asked the panel whether they thought visualization would make a big impact within Big Data? Ken Akoundi said that the front end applications used to make the data/analysis more useful will evolve very quickly. Brian asked whether this would be reminiscent of the days when VaR first appeared, when a single number arguably became a false proxy for risk measurement and management? Ken replied that the size of the data problem had increased massively from when VaR was first used in 1994, and that visualization and other automated techniques were very much needed if the headache of capturing, cleansing and understanding data was to be addressed.

Brian asked whether Big Data would address the data integration issue of siloed trading systems? Colleen replied that Big Data needs to work across all the silos found in many financial organizations, or it isn’t “Big Data”. There was general consensus from the panel that legacy systems and people politics were also behind some of the issues found in addressing the data silo issue.

Brian asked if the panel thought the skills needed in risk management would change due to Big Data? Colleen replied that effective Big Data solutions require all kinds of people, with skills across a broad range of specific disciplines such as visualization. Generally the panel thought that data and data analysis would play an increasingly important part for risk management. Ken put forward his view all Big Data problems should start with a business problem, with not just a technology focus. For example are there any better ways to predict stock market movements based on the consumption of larger and more diverse sources of information. In terms of risk management skills, Denny said that risk management of 15 years ago was based on relatively simply econometrics. Fast forward to today, and risk calculations such as CVA are statistically and computationally very heavy, and trading is increasingly automated across all asset classes. As a result, Denny suggested that even the PRMIA PRM syllabus should change to focus more on data and data technology given the importance of data to risk management.

Asked how best to should Big Data be applied?, then Denny replied that echoed Ken in saying that understanding the business problem first was vital, but that obviously Big Data opened up the capability to aggregate and work with larger datasets than ever before. Brian then asked what advice would the panel give to risk managers faced with an IT department about to embark upon using Big Data technologies? Assuming that the business problem is well understood, then Michael said that the business needed some familiarity with the broad concepts of Big Data, what it can and cannot do and how it fits with more mainstream technologies. Colleen said that there are some problems that only Big Data can solve, so understanding the technical need is a first checkpoint. Obviously IT people like working with new technologies and this needs to be monitored, but so long as the business problem is defined and valid for Big Data, people should be encouraged to learn new technologies and new skills. Kevin also took a very positive view that IT departments should be encouraged to experiment with these new technologies and understand what is possible, but that projects should have well-defined assessment/cut-off points as with any good project management to decide if the project is progressing well. Ken put forward that many IT staff were new to the scale of the problems being addressed with Big Data, and that his own company Opera Solutions had an advantage in its deep expertise of large-scale data integration to deliver quicker on project timelines.

Audience Questions: There then followed a number of audience questions. The first few related to other ideas/kinds of problems that could be analyzed using the kind of modeling that Opera had demonstrated. Ken said that there were obvious extensions that Opera had not got around to doing just yet. One audience member asked how well could all the Big Data analysis be aggregated/presented to make it understandable and usable to humans? Denny suggested that it was vital that such analysis was made accessible to the user, and there general consensus across the panel that man vs. machine was an interesting issue to develop in considering what is possible with Big Data. The next audience question was around whether all of this data analysis was affordable from a practical point of view. Brian pointed out that there was a lot of waste in current practices in the industry, with wasteful duplication of ticker plants and other data types across many financial institutions, large and small. This duplication is driven primarily by the perceived need to implement each institution’s proprietary analysis techniques, and that this kind of customization was not yet available from the major data vendors, but will become more possible as cloud technology such as Microsoft’s Azure develops further. There was a lot of audience interest in whether Big Data could lead to better understanding of causal relationships in markets rather than simply correlations. The panel responded that causal relationships were harder to understand, particularly in a dynamic market with dynamic relationships, but that insight into correlation was at the very least useful and could lead to better understanding of the drivers as more datasets are analyzed.

Posted by Brian Sentance | 8 February 2013 | 3:14 pm

Rutgers Quantitative Finance Summit

I got my first tour around the NYSE trading floor on Wednesday night, courtesy of an event by Rutgers University on Risk. Good event, mainly around panel discussion moderated by Nicholar Dunbar (Editor of Bloomberg Risk newsletter), and involving David Belmont (Commonfund CRO), Adam Litke (Chief Risk Strategist for Bloomberg), Hilmar Schaumann (Fortress Investment CRO) and Sanjay Sharma (CRO of Global Arbitrage and Trading at RBC).

Nick first asked the panel how do you define and measure risk? Hilmar responded that risk measurement is based around two main activities: 1) understanding how a book/portfolio is positioned (the static view) and 2) understanding sensitivities to risks that impact P&L (the dynamic view). Hilmar mentioned the use of historical data as a guide to current risks that are difficult to measure, but emphasised the need for a qualitative approach when looking at the risks being taken.

David said that he looks at both risk and uncertainty - with risk being defined as those impacts you can measure/estimate. He said that historical analysis was useful but limited given it is based only on what has happened. He thought that scenario analysis was a stronger tool. (I guess with historical analysis you at least get some idea of the impact of things that could not be predicted even it is based on one "simulation" path i.e. reality, whereas you have more flexibility with scenario management to cover all bases, but I guess limited to those bases you can imagine). David said that path-dependent risks such as those in the credit markets in the last crisis were some of the most difficult to deal with.

Adam said that you need to understand why you are measuring risk and understand what risks you are prepared to take. He said that at Wachovia they knew that a 25% house price fall in California would be a near death experience for the bank prior to the 2008 crisis, and in the event the losses were much greater than 25%. His point was really that you must decide what risks you want to survice and at what level. He said that sound common-sense judgement is needed to decide whether a scenario is really-real or not.

Sanjay said that risk managers need to maintain a lot of humility and not to over-trust risk meaurements. He described a little of the risk approach used at RBC where he said they use over 80 different models and employ them as layers/different views on risk to be brought together. He said they start with VaR as a base analysis, but build on this with scenarios, greeks and then on to other more specific reports and analysis. He emphasised that communication is a vital skill for risk managers to get their views and ideas across.

Nicholas then moved on to ask how risk managers should make or reduce risks? - getting away from risk measurement to risk management. Adam said that risks should be delegated out to those that manage them but this needs to be combined with responsibility for the risks too. Keep people and departments within the bounds of what their remit. Be prepared to talk a different business language to different stakeholders dependent upon their understanding and their motivations. David gave some examples of this in his case, where endowment funds what risk premiums over many years and risks are translated/quantified into practical things for example such as a new college building not going ahead etc.

Hilmar said the hedge funds are supposed to take risks, and that the key was not necessarily to avoid losses (although avoid them if you can) but rather to avoid surprises. Like the other speakers, Hilmar emphasised that communication of risks to key stakeholders was vital. He also added the key point that if you don't like a risk you have identified, then try first to take it off rather than hedging it, since hedging could potentially add basis risk and simple more complication.

Nicholas then Sanjay about how risk managers should deal with bringing difficult news to the business? Sanjay suggested that any bad news should be approach in the form of "actionable transparency" i.e. that not only do you say communicate how bad the risk is to all stakeholders but you come along with actionable approaches to dealing with the risk. In all of his experience and despite the crisis, Sanjay's experience is that traders do not want to loose money and if you come with solutions they will listen. He concluded by saying that qualitative analysis should also be used, citing the hypothetical example that you should take notice of dogs (yes, the animal!) buying mortgages, whether or not the mortgages are AAA rated.

Nicholas asked the panel members in turn what risks are they concerned about currently? David said he believed that many risks were not priced into the market currently. He was concerned about policy impacts of action by the ECB and the Fed, and thought the current and forward levels of volatility are low. In Fixed Income markets he thought that Dodd-Frank may have detrimental effects, particular with the current lack of clarity about what is proprietary trading and what is market-making. He thought that should policies and interests rates change, he thought that risk managers should look carefully at what will happen as funds flow out of fixed income and into equities.

Hilmar talked about the postponement of the US debt ceiling limits and that US Government policy battles continue to be an obvious source of risk. In Europe, many countries had elections this year which would be interesting, and that the problems in the Euro-zone are less than they were, but problems in Cyprus could fan the flames of more problems and anxiety. Hilmar said the Japan's new policy of targetting 2% inflation may have effects on the willingness of domestic investors to buy JGBs.

Sanjay said he was worried. In the "Greenspan Years" prior to 2008 a quasi government guarantee on the banks was effectively put in place and that we continue to live with cheap money. When policy eventually changes and interest rates rise, Sanjay wondered whether the world was ready for the wholesale asset revaluation that would then be required.

Adams concerns where mainly around identifying what will be the cause of the next panic in the market. Whilst he said he is in favour of central clearing for OTC derivatives, he thought that the changing market structure combined with implementing central clearing had not been fully thought through and this was a worry to him.

Nicholas asked what do the panelists think to the regulation being implemented? David said that regulators face the same difficulty that risk managers face, in that nobody notices when you took sensible action to protect against a risk that didn't occur. He thinks that regulation of the markets is justified and necessary.

Sanjay said that in the airline and pharmacutical industries regulatory approval was on the whole very robust but that they were dealing with approving designs (aeroplanes and drugs) that are reproduced once approved. He said that such levels of regulation in financial services were not yet possible due to the constant innovation found in the markets, and he wanted regulation to be more dynamic and responsive to market developments. Sanjay also joined those in the industry that are critical of the shear size of Dodd-Frank.

Nicholas said that Adam was obviously keen on operational issues and wondered what plumbing in the industry would he change? Adam said that he is a big fan of automation but operational risk are real and large. He thought that there were too many rules and regulations being applied, and the regulators were not paying attention to the type of markets they want in the future, nor on the effects of current regulation and how people were moving from one part of the industry to another. Adam said that in relation to Knight Capital he was still a strong advocate of standing by the wall socket, ready to pull the plug on the computer. Adam suggested that regulators should look at regulating/approving software releases (I assume here he means for key tasks such as automated trading or risk reporting, not all software).

Given the large number of students present, Nicholas closed the panel by asking what career advice the panelists had for future risk managers? Adam emphasised flexibility in role, taking us through his career background as an equity derivatives and then fixed income trader before coming into risk management. Adam said it was highly unlikely over your career that you would stay with one role or area of expertise.

Hilmar said that having risk managers independent of trading was vitally important for the industry. He thought there were many areas to work with operational risk being potentially the largest, but still with plenty more to do in market risk, compliance and risk modelling. He added that understanding the interdepencies between risks was key and an area for further development.

When asked by Nicholas, David said that risk managers should have a career path right through to CEO of an institution. He wanted to encourage risk management as a necessary level above risk measurement and control. He was excited about the potential of Big Data technologies to help in risk management. David gave some interesting background on his own career initially as an emergining markets debt trader. He said that it is important to know yourself, and that he regarded himself as a sceptic, needing all the information available before making a decision. As such his performance as a trader was consistent but not as high as some, and this became one of the reasons he moved into risk management.

Sanjay said many of the systems used in finance are 20 years old, in complete contrast with the advancies in mobile and internet technologies. As such he thought this was a great opportunity to be involved in the replacement and upgrading of this older infrastructure. Apparently one analyst had estimated that $65B will be spent on risk management over the next 4-5 years.

Adam thought that there was a need for code of ethics for quants (see old post for some ideas). Sanjay added that the industry needed to move away from being involved primarily in attempting to optimise activity around gaming regulation. When asked by Nicholas about Basel III, Adam thought that improved regulation was necessary but Basel III was not the right way to go about it and was way too complex.

Posted by Brian Sentance | 1 February 2013 | 2:41 pm

Chartis Research - Data Management for Risk White Paper

New whitepaper on data management for risk from the analysts Chartis Research, including a section on how Xenomorph's TimeScape solution addresses some of the key issues identified.

Posted by Brian Sentance | 22 January 2013 | 3:14 pm

Not only SQL but too much choice as well?

In relation to the Microsoft/PRMIA event that Brian moderated at last night in New York, I spotted this article recently that tries to map out all the different databases that are now commercially available in some form, from SQL to No SQL and all the various incarnations and flavours in between:

http://gigaom.com/2012/12/20/confused-by-the-glut-of-new-databases-heres-a-map-for-you/

As Brian suggested in his recent post, It's amazing to see how much the landscape has evolved from the domination (mantra?) that there was the relational way, or no way. Obviously times have moved on (er, I guess the Internet happened for one thing...) and people are now far more accepting of the need for different approaches to different types and sizes of business problems. That said, I agree with the article and comments that suggest there do seem to be far too many options available now - there has to be some consolidation coming otherwise it will become increasingly difficult to know where to start. Choice is a wonderful thing, but only in moderation!

Posted by Chris Budgen | 16 January 2013 | 9:30 pm

Numerix and Xenomorph partner in risk management

Just a quick post to highlight Xenomorph's Numerix partnership announcement that went out earlier this week. In summary we have done some great work with Numerix on combining their ability to price and risk manage very complex trades with TimeScape's ability to manage all the data such types of instruments need.

The integration is a great demonstration of the flexibility of TimeScape's data model (see recent post and LinkedIn discussion) and addresses some of the issues discussed and illustrated in an earlier post on data management for risk. Quick thank you to the clients involved in testing and using the integration, to the Numerix team for their assistance on this and to my New York colleagues who led the TimeScape integration work.

Posted by Brian Sentance | 12 December 2012 | 2:36 pm

Big Data - Integration still a key (external) issue

Good breakfast event from SAP and A-Team last Thursday morning. SAP have been getting (and I guess paying for) a lot of good air-time for their SAP Hana in-memory database technology of late. Domenic Iannaccone of SAP started the briefing with an introduction to big data in finance and how their SAP/Sybase offerings knitted together. He started his presentation with a few quotes, one being "Intellectual property is the oil of the 21st century" by Mark Getty (he of Getty images, but also of the Getty oil family) and "Data is the new oil" by both Clive Humby and Gerd Leonhard (not sure why two people quoted saying the same thing but anyway).

For those of you with some familiarity with the Sybase IQ architecture of a year or two back, then in this architecture SAP Hana seems to have replaced the in-memory ASE database that worked in tandem with Sybase IQ for historical storage (I am yet to confirm this, but hope to find out more in the new year). When challenged on how Hana differs from other in-memory database products, Domenic seemed keen to emphasise its analytical capabilities and not just the database aspects. I guess it was the big data angle of bring the "data closer to the calculations" was his main differentiator on this, but with more time I think a little bit more explanation would have been good.

Pete Harris of the A-Team walked us through some of the key findings of what I think is the best survey I have read so far on the usage of big data in financial markets (free sign-up needed I think, but you can get a copy of the report here). Some key findings from a survey of staff at ten major financial institutions included:

Searching for meaning in instructured data was a leading use-case thought of when thinking of big data (Twitter trading etc)
Risk management was seen as a key beneficiary of what the technologies can offer
Aggregation of data for risk was seen as a key application area concerning structured data.
Both news feed but also (surprisingly?) text documents were key unstructured data sources being processed using big data.
In trading news sentiment and time series analysis were key areas for big data.
Creation of a system wide trade database for surveillance and compliance was seen as a key area for enhancement by big data.
Data security remains a big concern with technologists over the use of big data.

There were a few audience questions - Pete clarified that there was a more varied application of big data amongst sell-side firms, and that on the buy-side it was being applied more KYC and related areas. One of the audience made that point that he thought a real challenge beyond the insight gained from big data analysis was how to translate it into value from an operational point of view. There seemed to be a fair amount of recognition that regulators and auditors are wanting a full audit trail of what has gone on across the whole firm, so audit was seen as a key area for big data. Another audience member suggested that the lack of a rigid data model in some big data technologies enabled greater flexibility in the scope of questions/analysis that could be undertaken.

Coming back to the key findings of the survey, then one question I asked Pete was whether or not big data is a silver bullet for data integration. My motivation was that the survey and much of the press you read talks about how big data can pull all the systems, data and calculations together for better risk management, but while I can understand how massively scaleable data and calculation capabilities was extremely useful, I wondered how exactly all the data was pulled together from the current range of siloed systems and databases where it currently resides. Pete suggested that this was stil a problematic area where Enterprise Application Integration (EAI) tools were needed. Another audience member added that politics within different departments was not making data integration any easier, regardless of the technologies used.

Overall a good event, with audience interaction unsurprisingly being the most interesting and useful part.

Posted by Brian Sentance | 3 December 2012 | 2:12 pm

The Missing Data Gap

Getting to the heart of "Data Management for Risk", PRMIA held an event entitled "Missing Data for Risk Management Stress Testing" at Bloomberg's New York HQ last night. For those of you who are unfamiliar with the topic of "Data Management for Risk", then the following diagram may help to further explain how the topic is to do with all the data sets feeding the VaR and scenario engines.

I have a vested interest in saying this (and please forgive the product placement in the diagram above, but hey this is what we do...), but the topic of data management for risk seems to fall into a functionality gap between: i) the risk system vendors who typically seem to assume that the world of data is perfect and that the topic is too low level to concern them and ii) the traditional data management vendors who seem to regard things like correlations, curves, spreads, implied volatilities and model parameters as too business domain focussed (see previous post on this topic) As a result, the risk manager is typically left with ad-hoc tools like spreadsheets and other analytical packages to perform data validation and filling of any missing data found. These ad-hoc tools are fine until the data universe grows larger, leading to the regulators becoming concerned about just how much data is being managed "out of system" (see past post for some previous thoughts on spreadsheets).

The Crisis and Data Issues. Anyway enough background above and on to some of the issues raised at the event. Navin Sharma of Western Asset Management started the evening by saying that pre-crisis people had a false sense of security around Value at Risk, and that crisis showed that data is not reliably smooth in nature. Post-crisis, then questions obviously arise around how much data to use, how far back and whether you include or exclude extreme periods like the crisis. Navin also suggested that the boards of many financial institutions were now much more open to reviewing scenarios put forward by the risk management function, whereas pre-crisis their attention span was much more limited.

Presentation. Don Wesnofske did a great presentation on the main issues around data and data governance in risk (which I am hoping to link to here shortly...)

Issues with Sourcing Data for Risk and Regulation. Adam Litke of Bloomberg asked the panel what new data sourcing challenges were resulting from the current raft of regulation being implemented. Barry Schachter cited a number of Basel-related examples. He said that the costs of rolling up loss data across all operations was prohibitative, and hence there were data truncation issues to be faced when assessing operational risk. Barry mentioned that liquidity calculations were new and presenting data challenges. Non centrally cleared OTC derivatives also presented data challenges, with initial margin calculations based on stressed VaR. Whilst on the subject of stressed VaR, Barry said that there were a number of missing data challenges including the challenge of obtaining past histories and of modelling current instruments that did not exist in past stress periods. He said that it was telling on this subject that the Fed had decided to exclude tier 2 banks from stressed VaR calculations on the basis that they did not think these institutions were in a position to be able to calculate these numbers given the data and systems that they had in place.

Barry also mentioned the challenges of Solvency II for insurers (and their asset managers) and said that this was a huge exercise in data collection. He said that there were obvious difficulties in modelling hedge fund and private equity investments, and that the regulation penalised the use of proxy instruments where there was limited "see-through" to the underlying investments. Moving on to UCITS IV, Barry said that the regulation required VaR calculations to be regularly reviewed on an ongoing basis, and he pointed out one issue with much of the current regulation in that it uses ambiguous terms such as models of "high accuracy" (I guess the point being that accuracy is always arguable/subjective for an illiquid security).

Sandhya Persad of Bloomberg said that there were many practical issues to consider such as exchanges that close at different times and the resultant misalignment of closing data, problems dealing with holiday data across different exchanges and countries, and sourcing of factor data for risk models from analysts. Navin expanded more on his theme of which periods of data to use. Don took a different tack, and emphasised the importance of getting the fundamental data of client-contract-product in place, and suggested that this was a big challenge still at many institutions. Adam closed the question by pointing out the data issues in everyday mortgage insurance as an example of how prevalant data problems are.

What Missing Data Techniques Are There? Sandhya explained a few of the issues her and her team face working at Bloomberg in making decisions about what data to fill. She mentioned the obvious issue of distance between missing data points and the preceding data used to fill it. Sandhya mentioned that one approach to missing data is to reduce factor weights down to zero for factors without data, but this gave rise to a data truncation issue. She said that there were a variety of statistical techniques that could be used, she mentioned adaptive learning techniques and then described some of the work that one of her colleagues had been doing on maximum-likehood estimation, whereby in addition to achieving consistency with the covariance matrix of "near" neighbours, that the estimation also had greater consistency with the historical behaviour of the factor or instrument over time.

Navin commented that fixed income markets were not as easy to deal with as equity markets in terms of data, and that at sub-investment grade there is very little data available. He said that heuristic models where often needed, and suggested that there was a need for "best practice" to be established for fixed income, particularly in light of guidelines from regulators that are at best ambiguous.

I think Barry then made some great comments about data and data quality in saying that risk managers need to understand more about the effects (or lack of) that input data has on the headline reports produced. The reason I say great is that I think there is often a disconnect or lack of knowledge around the effects that input data quality can have on the output numbers produced. Whilst regulators increasingly want data "drill-down" and justfication on any data used to calculate risk, it is still worth understanding more about whether output results are greatly sensitive to the input numbers, or whether maybe related aspects such as data consistency ought to have more emphasis than say absolute price accuracy. For example, data quality was being discussed at a recent market data conference I attended and only about 25% of the audience said that they had ever investigated the quality of the data they use. Barry also suggested that you need to understand to what purpose the numbers are being used and what effect the numbers had on the decisions you take. I think here the distinction was around usage in risk where changes/deltas might be of more important, whereas in calculating valuations or returns then price accuracy might receieve more emphasis.

How Extensive is the Problem? General consensus from the panel was that the issues importance needed to be understood more (I guess my experience is that the regulators can make data quality important for a bank if they say that input data issues are the main reason for blocking approval of an internal model for regulatory capital calculations). Don said that any risk manager needed to be able to justify why particular data points were used and there was further criticism from the panel around regulators asking for high quality without specifying what this means or what needs to be done.

Summary - My main conclusions:

Risk managers should know more of how and in what ways input data quality affects output reports
Be aware of how your approach to data can affect the decisions you take
Be aware of the context of how the data is used
Regulators set the "high quality" agenda for data but don't specify what "high quality" actually is
Risk managers should not simply accept regulatory definitions of data quality and should join in the debate

Great drinks and food afterwards (thanks Bloomberg!) and a good evening was had by all, with a topic that needs further discussion and development.

Posted by Brian Sentance | 16 October 2012 | 3:21 pm

Bankenes Sikringsfond Selects Xenomorph's TimeScape for Faster Data Analysis and High-Quality Decision Support

Just a quick note to say that we have signed a new client, Bankenes Sikringsfond, the Norwegian Banks’ Guarantee Fund. They will be using TimeScape to fulfill requirements for a centralised analytics and data management platform. The press release is available here for those of you who are interested.

Posted by Sara Verri | 11 October 2012 | 10:50 am

Charting, Heatmaps and Reports for TimeScape, plus new Query Explorer

We have a great new software release out today for TimeScape, Xenomorph's analytics and data management solution, more details of which you can find here. For some additional background to this release then please take a read below.

For many users of Xenomorph's TimeScape, our Excel interface to TimeScape has been a great way of extending and expanding the data analysis capabilities of Excel through moving the burden of both the data and the calculation out of each spreadsheet and into TimeScape. As I have mentioned before, spreadsheets are fantastic end-user tools for ad-hoc reporting and analysis, but problems arise when their very usefulness and ease of use cause people to use them as standalone desktop-based databases. The four-hundred or so functions available in TimeScape for Excel, plus Excel access to our TimeScape QL+ Query Language have enabled much simpler and more powerful spreadsheets to be built, simply because Excel is used as a presentation layer with the hard work being done centrally in TimeScape.

Many people like using spreadsheets, however many users equally do not and prefer more application based functionality. Taking this feedback on board has previously driven us to look at innovative ways of extending data management, such as embedding spreadsheet-like calculations inside TimeScape and taking them out of spreadsheets with our SpreadSheet Inside technology. With this latest release of TimeScape, we are providing much of the ease of use, analysis and reporting power of spreadsheets but doing so in a more consistent and centralised manner. Charts can now be set up as default views on data so that you can quickly eyeball different properties and data sources for issues. New Heatmaps allow users to view large colour-coded datasets and zoom in quickly on areas of interest for more analysis. Plus our enhanced Reporting functionality allows greater ease of use and customisation when wanting to share data analysis with other users and departments.

Additionally, the new Query Explorer front really shows off what is possible with TimeScape QL+, in allowing users to build and test queries in the context of easily configurable data rules for things such as data source preferences, missing data and proxy instruments. The new auto-complete feature is also very useful when building queries, and automatically displays all properties and methods available at each point in the query, even including user-defined analytics and calculations. It also displays complex and folded data in an easy manner, enabling faster understanding and analysis of more complex data sets such as historical volatility surfaces.

Posted by Brian Sentance | 17 July 2012 | 3:11 pm

OIS Discounting and Curve Management - Presentation Materials

Just a quick note to say that the video, presentations and supporting documents have now gone up for our recent Wilmott event with Numerix on OIS Curves and Libor in New York. Somewhat topical at the moment given the current bad press for Barclays.

Posted by Brian Sentance | 29 June 2012 | 2:20 pm

Paris Financial Information Summit 2012

I attended the Financial Information Summit event on Tuesday, organized in Paris by Inside Market Data and Inside Reference Data.

Unsurprisingly, most of the topics discussed during the panels focused on reducing data costs, managing the vendor relationship strategically, LEI and building sound data management strategies.

Here is a (very) brief summary of the key points touched which generated a good debate from both panellists and audience:

Lowering data costs and cost containment panels

Make end-users aware of how much they pay for that data so that they will have a different perspective when deciding if the data is really needed or a "nice to have"
Build a strong relationship with the data vendor: you work for the same aim and share the same industry issues
Evaluate niche data providers who are often more flexible and willing to assist while still providing high quality data
Strategic vendor management is needed within financial institutions: this should be an on-going process aimed to improve contract mgmt for data licenses
A centralized data management strategy and consolidation of processes and data feeds allow cost containment (something that Xenomorph have long been advocating)
Accuracy and timeliness of data is essential: make sure your vendor understands your needs
Negotiate redistribution costs to downstream systems

One good point was made by David Berry, IPUG-Cossiom, on the acquisition of data management software vendors by the same data providers (referring to the Markit-Cadis and PolarLake-Bloomberg deals) and stating that it will be tricky to see how the two business units will be managed "separately" (if kept separated...I know what you are thinking!).

There were also interesting case studies and examples supporting the points above. Many panellists pointed out how difficult can be to obtain high quality data from vendors and that only regulation can actually improve the standards. Despite the concerns, I must recognize that many firms are now pro-actively approaching the issue and trying to deal with the problem in a strategic manner. For example, Hand Henrik Hovmand, Market Data Manager, Danske Bank, explained how Danske Bank are in the process of adopting a strategic vendor system made of 4 steps: assessing vendor, classifying vendor, deciding what to do with the vendor and creating a business plan. Vendors are classified as strategic, tactical, legacy or emerging. Based on this classification, then the "bad" vendors are evaluated to verify if they are enhancing data quality. This vendor landscape is used both internally and externally during negotiation and Hovmand was confident it will help Danske Bank to contain costs and get more for the same price.

I also enjoyed the panel on Building a sound management strategy where Alain Robert- Dauton, Sycomore Asset Management, was speaking. He highlighted how asset managers, in particular smaller firms, are now feeling the pressure of regulators but at the same time are less prepared to deal with compliance than larger investment banks. He recognized that asset managers need to invest in a sound risk data management strategy and supporting technology, with regulators demanding more details, reports and high quality data.

For a summary on what was said on LEI, then seems like most financial institutions are still unprepared on how it should be implemented, due to uncertainty around it but I refer you to an article from Nicholas Hamilton in Inside Reference Data for a clear picture of what was discussed during the panel.

Looking forward, the panellists agreed that the main challenge is and will be managing the increasing volume of data. Though, as Tom Dalglish affirmed, the market is still not ready for the cloud, given than not much has been done in terms of legislation. Watch out!

The full agenda of the event is available here.

Posted by Sara Verri | 14 June 2012 | 5:54 pm

OIS Discounting and Curve Management - Wilmott Event - Thursday 31st May

Quick plug for Xenomorph's Wilmott Forum Event on OIS curves tomorrow in downtown Manhattan. The event is done in partnership with Numerix, and will be looking at the issue of OIS vs. Libor discounting from the point of view of a practioner, financial engineer and systems developer. You can register for the event here, and so we hope to see you at 6pm for some great talks and some drinks/socialising afterwards.

Posted by Brian Sentance | 30 May 2012 | 2:07 pm

TabbFORUM Video: Data Is Revenue

Video interview with Paul Rowady of the Tabb Group, primarily about how data management can break out from being just a back office function and become a source of competitive advantage in both the front office and in risk management.

For those of you with a curious mind, the perseverence to watch the video until the end and possibly not such advanced years as me and Paul, then the lead singer of Midnight Oil that he refers to at the close of the video is Peter Garrett, who looks like this:

Whereas I look like this:

See, completely different. Obviously Peter has a great choice in hairstyle though...

Posted by Brian Sentance | 30 May 2012 | 1:22 pm

CVA - a business driver for breaking down asset silos

Xenomorph's analytics partner Numerix sponsored a PRMIA event at New York's Harvard Club this week on Credit Valuation Adjustment (CVA). The event also involved Microsoft, with a surprisingly relevant contribution to the evening on CVA and "Big Data" (I still don't feel comfortable losing the quotes yet, maybe soon...). Credit Valuation Adjustment seems to be the hot topic in risk management and pricing at the moment, with Numerix's competitor Quantifi having held another PRMIA event on CVA only a few months back.

The event started with an introduction to CVA from Aletta Ely of JP Morgan Chase. Aletta started by defining CVA as the market value of counterparty credit risk. I am new to CVA as a topic, and my own experience on any kind of adjustment in valuation for instrument was back at JP Morgan in the mid-90s (those of you under 30 are allowed to start yawning at this point...). We used to maintain separate risk-free curves (what are they now?) and counterparty spread curves, which would be combined to discount the cashflows in the model.

Whilst such an adjustment could be calibrated to come up with an adjusted valuation which would be better than having no counterparty risk modelled at all, it seems one of the key aspects of how CVA differs is that a credit valuation adjustement needs to be done in the context of the whole portfolio of exposures to the counterparty, and not in isolation instrument by instrument. The fact that a trader in equity derivatives was long exposure to a counterparty cannot be looked at in isolation from a short exposure to a portfolio of swaps with the same counterparty on the fixed income desk.

Put another way, CVA only has context if we stand to lose money if our counterparty defaults, and so an aggregated approach is needed to calculate the size of the positive exposures to the counterparty over the lifetime of the portfolio. Also, given this one sided payoff aspect of the CVA calculation, then instrument types such as vanilla interest rate swaps suddenly move from being relatively simple instrument that can be priced off a single curve to instruments that needed optionality to be modelled for the purposes of CVA.

So why has CVA become such a hot topic at the banks? Prior to the 2008/2009 crisis CVA was already around (credit risk has existed for a long time I guess, regardless of whether you regulate or report to it), but given that bank credit spreads were at that time consistently low and stable then CVA had minimal effects on valuations and P&L. Obviously with the advent of Lehmans then this changed, and CVA has been pushed into prominence since it has directly affected P&L in a significant manner for many institutions (for example see these FT articles on Citi and JPMorgan)

A key and I think positive point for the whole industry is the CVA requires a completely multi-asset view, and given regulatory focus on CVA and capital adequacy then as a result it will drive banks away from a siloed approach to data and valuation management. If capital is scarcer and more costly, then banks will invest in understanding both their aggregate CVA and the incremental contribution to CVA of a new trade in the context of all exposures to the counterparty. Looking at incremental CVA, then you can also see that this also drives investment into real or near-realtime CVA calculation, which brings me on to the next talks of the evening by Numerix on CVA calculation methods and a surprisingly good presentation on CVA and "Big Data" from David Cox of Microsoft.

Denny Yu of Numerix did a good job of explaining some of the methods of calculating CVA, and in addition to being cross asset and all the implications that requires for having the ability to price anything, CVA is both data and computationally expensive. It requires both simulation of the scenarios for the default of counterparties through time, but also the valuation of cross-asset portfolios at different points in time. Denny mentioned techniques such as American Monte-Carlo to reduce the computation needed through using the same simulation paths for both default scenarios and valuation.

So on to Microsoft. I have seen some appalling presentations on "Big Data" recently, mainly from the larger software and hardware companies try to jump on the marketing band wagon (main marketing premise: the data problems you have are "Big"...enough said I hope). Surprisingly, David Cox of Microsoft gave a very good presentation around the computation challenges of CVA, and how technologies such as Hadoop take the computational power closer to the data that needs acting on, bringing the analytics and data together. (As an aside, his presentation was notably "Metro" GUI in style, something that seems to work well for PowerPoint where the slide is very visual and it puts more emphasis on the speak to overlay the information). David was obviously keen to talk up some of the cloud technology that Microsoft is currently pushing, but he knew the CVA business topic well and did a good job of telling a good story around CVA, "Big Data" and Cloud technologies. Fundamentally, his pitch was for banks and other institutions to become "Analytic Enterprises" with a common, scaleable and flexible infrastructure for data management and analysis.

In summary it was a great event - the Harvard Club is always worth a visit (bars and grandiose portraits as expected but also barber shop in the basement and squash courts in the loft!), the wine afterwards was tolerably good and the speakers were informative without over-selling their products or company. Quick thank you to Henry Hu of IBM for transportation on the night, and thanks also to Henry for sending through this link to a great introductory paper on CVA and credit risk from King's College London. Whilst the title of the King's paper is a bit long and scary, it takes the form of dialogue between a new employee and a CVA expert, and as such is very readable with lots of background links.

Posted by Brian Sentance | 13 April 2012 | 2:56 pm

NoSQL - the benefit of being specific

NoSQL is an unfortunate name in my view for the loose family of non-relational database technologies associated with "Big Data". NotRelational might be a better description (catchy eh? thought not...) , but either way I don't like the negatives in both of these titles, due to aestetics and in this case because it could be taken to imply that these technologies are critical of SQL and relational technology that we have all been using for years. For those of you who are relatively new to NoSQL (which is most of us), then this link contains a great introduction. Also, if you can put up with a slightly annoying reporter, then the CloudEra CEO is worth a listen to on YouTube.

In my view NoSQL databases are complementary to relational technology, and as many have said relational tech and tabular data are not going away any time soon. Ironically, some of the NoSQL technologies need more standardised query languages to gain wider acceptance, and there will be no guessing which existing query language will be used for ideas in putting these new languages together (at this point as an example I will now say SPARQL, not that should be taken to mean that I know a lot about this, but that has never stopped me before...)

Going back into the distant history of Xenomorph and our XDB database technology, then when we started in 1995 the fact that we then used a proprietary database technology was sometimes a mixed blessing on sales. The XDB database technology we had at the time was based around answering a specific question, which was "give me all of the history for this attribute of this instrument as quickly as possible".

The risk managers and traders loved the performance aspects of our object/time series database - I remember one client with a historical VaR calc that we got running in around 30 minutes on laptop PC that was taking 12 hours in an RDBMS on a (then quite meaty) Sun Sparc box. It was a great example how specific database technology designed for specific problems could offer performance that was not possible from more generic relational technology. The use of database for these problems was never intended as a replacement for relational databases dealing with relational-type "set-based" problems though, it was complementary technology designed for very specific problem sets.

The technologists were much more reserved, some were more accepting and knew of products such as FAME around then, but some were sceptical over the use of non-standard DBMS tech. Looking back, I think this attitude was in part due to either a desire to build their own vector/time series store, but also understandably (but incorrectly) they were concerned that our proprietary database would be require specialist database admin skills. Not that the mainstream RDBMS systems were expensive or specialist to maintain then (Oracle DBA anyone?), but many proprietary database systems with proprietary languages can require expensive and on-going specialist consultant support even today.

The feedback from our clients and sales prospects that our database performance was liked, but the proprietary database admin aspects were sometimes a sales objection caused us to take a look at hosting some of our vector database structures in Microsoft SQL Server. A long time back we had already implemented a layer within our analytics and data management system where we could replace our XDB database with other databases, most notably FAME. You can see a simple overview of the architecture in the diagram below, where other non-XDB databases (and datafeeds) can "plugged in" to our TimeScape system without affecting the APIs or indeed the object data model being used by the client:

Data Unification Layer

Using this layer, we then worked with the Microsoft UK SQL team to implement/host some of our vector database structures inside of Microsoft SQL Server. As a result, we ended up with a database engine that maintained the performance aspects of our proprietary database, but offered clients a standards-based DBMS for maintaining and managing the database. This is going back a few years, but we tested this database at Microsoft with a 12TB database (since this was then the largest disk they had available), but still this contained 500 billion tick data records which even today could be considered "Big" (if indeed I fully understand "Big" these days?). So you can see some of the technical effort we put into getting non-mainstream database technology to be more acceptable to an audience adopting a "SQL is everything" mantra.

Fast forward to 2012, and the explosion of interest in "Big Data" (I guess I should drop the quotes soon?) and in NoSQL databases. It finally seems that due to the usage of these technologies on internet data problems that no relational database could address, the technology community seem to have much more willingness to accept non-RDBMS technology where the problem being addressed warrants it - I guess for me and Xenomorph it has been a long (and mostly enjoyable) journey from 1995 to 2012 and it is great to see a more open-minded approach being taken towards database technology and the recognition of the benefits of specfic databases for (some) specific problems. Hopefully some good news on TimeScape and NoSQL technologies to follow in coming months - this is an exciting time to be involved in analytics and data management in financial markets and this tech couldn't come a moment too soon given the new reporting requirements being requested by regulators.

Posted by Brian Sentance | 4 April 2012 | 4:54 pm

Data Visualisation from the FT

Data visualisation has always been an interesting subject in financial markets, one that seems to always have been talked about about as the next big thing in finance, but one that always seems to fail to meet expectations (of visualisation software vendors mostly...). I went along to an event put on by the FT today about what they term "infographics", set in the Vanderbilt Hall at Grand Central Station New York:

One of my first experiences of data visualisation was showing a partner company, Visual Numerix (VNI), around the Bankers Trust 's London trading floor in 1995. The VNI folks were talking grandly about visualising a "golden corn field of trading oportunities, with the wind of market change forcing the blades of corn to change in size and orientation" - whilst maybe they had been under the influence of illegal substances when dreaming up this description, their disappointment was palpable at trading screen after trading screen full of spreadsheets containing "numbers". Sure there was some charting being used, but mostly and understandably the traders were very focussed on the numbers of the deal that they were about to do (or had just done).

I guess this theme ultimately continues today to a large extent, although given the (media hyped) "explosion of data", visualisation is a useful technique for filtering down a large (er, can I use the word "big"?) data problem to get at the data you really want to work with (quick plug - the next version of our TimeScape product includes graphical heatmaps for looking for data exceptions, statistical anomolies and trading opportunities, which confirms Xenomorph buys into at least this aspect of the "filtering" benefits of visualisation).

Coming back to the presentation, Gillian Tett of the FT said at the event today that "infographics" is cutting edge technology - not sure I would agree although given the location some of the images were very good, like this one representing the stock pile of cash that major corporations have been hoarding (i.e. not spending) over recent years:

There was also some "interactive" aspects to the display where by stepping on part of the hall floor changed the graphic displayed. Biggest problem the FT had with this was persuading anyone to step into the middle of the floor to use it (more of an English reaction to such a request, so the reticience from New Yorker's surprised me):

Videos from the presentation can be found at http://ftgraphicworld.ft.com/ and the journalist involved, David McCandless is worth a listen to for the different ways he looks at data both on the FT site but also in a TED presentation.

Posted by Brian Sentance | 27 March 2012 | 4:54 pm

Risk models and tools at Baruch College

Emanuel Derman gave the last presentation of the day on mathematical models and their role in financial markets. His presentation seemed to build on some of his earlier ideas with Paul Wilmott on the "Modeller's Manifesto".

Emanuel said that there was a "scandal based on models" is wrong; models did (and do) have their faults but they were not a root cause of the crisis. He started his presentation (somewhat "tongue in cheek") by putting forward a "Theory of Deliciousness" to see how one might arrived at the value of something being more or less delicious. This involved discussion of "realised deliciousness" and "expected or implied deliciousness", plus definitions around equally (relatively) delicious things and absolute deliciousness. See post on FT Alphaville for more background, but fundamentally by analogy Emanuel was putting across that there is no "fundamental theory of finance" and that finance is not physics.

He said that economists do not know the difference between theorems and laws. He seemed to be critical of some recent work from Andrew Lo (see recent post) on putting together a "Complete Theory of Human Behaviour" for once again attempting to codify something that it is uncodifiable.

Emanuel described how economists should be more aware of what is and isn't a:

Metaphor - using something physical/tangible to represent a less tangible concept or idea. See this link for his interesting example on sleep/life and debt interest
Model - extending the behaviour of one thing to another. A model aircraft is a very useful model of a full-size aircraft with know inputs and useful outputs of interest. We can try to model the weather but here the inputs are known (temperature, wind etc) but the model is hard to define. In finance it is hard to really see what both the inputs are and what the outputs are too.
Theory - the ultimate non-metaphor. Here he gave the example of Moses asking the burning bush who shall I say sent me to which God replies "I am what I am". Put another way, you can't ask why on a theory, it just is.
Intuition - a premise put forward based neither on logical progression nor on experimentation.

Emanuel said that in Finance there is no absolute value theory, and the majority of models are relative value in nature. From a common sense point of view, the world is not a model. Things change dynamically and in this way effectively all models are wrong to some degree. In summary all financial models are short volatility.

He ended his presentation by saying that nature cares more about principles than regulations (prescriptive regulators beware I guess). His parting quote was by Edward Lucas who said "If you believe that capitalism is a system in which money matters more than freedom, you are doomed when people who don’t believe in freedom attack using money."

Panel Debate

Some highlights:

Bruno Dupire of Bloomberg said that it was important that a financial product was aligned with the needs of the customer, and cited certain complex products (with triggers) as being more in the interests of the vendor not the customer.
Bruno also said that the hedgeability of a product was also key to a more stable financial system (presumably pointing at products like CDO^3 etc). He said that residual risk (that left after hedging with simpler products) should be measured and costed for. Bruno also mention the problems with assessing long term volatility where traders will try to set this input to what best suits their own P&L
Leo Tilman said that risk management needs to be a decision-support discipline and not a policing function. He later suggested that risk managers should have to work as consultants for a while to understand that they get paid for serving the needs of the customer, not just stopping all activity/risks (in fairness to risk managers, I guess they might ask who is my customer? the trader? the CEO? the firm?).
Dilip Madan added to the models debate by saying "what is not in the assumptions will not show up in the conclusions".
Emanuel likes the old GS partner model for banking, and mentioned the example of Brazilian banks where banks/banking staff(?) did not enjoy limited liability. Dilip said he understood the advantage of this but no limited liability would stifle entrepreneurship.
Leon Tatevossian said that post-crisis the relationship between risk managers and traders is better than before, and that there was also greater co-operation between empiricists and modelers. Leo add that risk managers and traders need to speak the same language and understand what each other means by "risk".
Bruno said that models were much less of a problem than leverage.
All seemed to agree that the tools were not invalidated by the crisis, but the framework in which they are used was the important thing.

Posted by Brian Sentance | 11 February 2012 | 8:09 pm

PRMIA - From Risk Measurement to Risk Management by Samuel Won

I attended the PRMIA event last night "Risk Year in Review" at Moody's New York offices. It was a good event, but by far the most interesting topic of the evening for me was from Samuel Won, who gave a talk about some of the best and most innovative risk management techniques being used in the market today. Sam said that he was inspired to do this after reading the book "The Information" by James Gleik about the history of information and its current exponential growth. Below are some of the notes I took on Sam's talk, please accept my apologies in advance for any errors but hopefully the main themes are accurate.

Early '80s ALM - Sam gave some context to risk management as a profession through his own personal experiences. He started work in the early 80's at a supra-regional bank, managing interest rate risk on a long portfolio of mortgages. These were the days before the role of "risk manager" was formally defined, and really revolved around Asset and Liability Management (ALM).

Savings and Loans Crisis - Sam then changed roles and had some first hand experience in sorting out the Savings and Loans crisis of the mid '80s. In this role he become more experienced with products such as mortgage backed securities, and more familiar with some of the more data intensive processes needed to manage such products in order to account for such factors such as prepayment risk, convexity and cashflow mapping.

The Front Office of the '90s - In the '90s he worked in the front office at a couple of tier one investment banks, where the role was more of optimal allocation of available balance sheet rather than "risk management" in the traditional sense. In order to do this better, Sam approached the head of trading for budget to improve and systemise this balance sheet allocation but was questioned as to why he needed budget when the central Risk Control department had a large staff and large budget already.

Eventually, he successfully argued the case that Risk Control were involved in risk measurement and control, whereas what he wanted to implement was active decision support to improve P&L and reduce risk. He was given a total budget of just $5M (small for a big bank) and told to get on with it. These two themes of implementing active decision support (not just risk measurement) and have a profit motive driving better risk management ran through the rest of his talk.

A Datawarehouse for End-Users Too - With a small team and a small budget, Sam made use of postgraduate students to leverage what his team could develop. They had seen that (at the time) getting systems talking to each other was costly and unproductive, and decided as a result to implement a datawarehouse for the front office, implementing data normalisation and data scrubbing, with data dashboard over the top that was easy enough for business users to do data mining. Sam made the point that useability was key in allowing the business people to extract full value from the solution.

Sam said that the techniques used by his team and the developers were not necessarily that new, things like regression and correlation analysis were used at first. These were used to establish key variables/factors, with a view to establish key risk and investment triggers in as near to real-time as possible. The expense of all of this development work was justified through its effects on P&L which given its success resulting in more funding from the business.

Poor Sell-Side Risk Innovation - Sam has seen the most innovative risk techniques being used on the buy-side and was disappointed by the lack of innovation in risk management at the banks. He listed the following sell-side problems for risk innovation:

politically driven requirements, not economically driven
arbitrary increases in capital levels required is not a rigorous approach
no need for decision analysis with risk processes
just passing a test mentality
just do the marginal work needed to meet the new rules
no P&L justification driving risk management

Features of Innovative Approaches - Sam said that he had noted a few key features of some of the initiatives he admired at some of the asset managers:

Based on a sophisticated data warehouse (not usually Oracle or Sybase, but Microsoft and other databases used - maybe driven by ease of use or cost maybe?)
Traders/Portfolio Managers are the people using the system and implementing it, not the technical staff.
Dedicated teams within the trading division to support this, so not relying on central data team.

A Forward-Looking Risk Model Example - The typical output from such decision analysis systems he found was in the form of scenarios for users to consider. A specific example was a portfolio manager involved in event-driven long-short equity strategies around mergers and acquisitions. The manager is interested in the risk that a particular deal breaks, and in this case techniques such as Value at Risk (VaR) do not work, since the arbitrage usually requires going long the company being acquired and short the acquiror (VaR would indicate little risk in this long-short case). The manager implemented a forward looking model that was based on information relevant to the deal in question plus information from similar historic deals. The probabilities used in the model where gathered from a range of sources, and techniques such as triangulation where used to verify the probabilities. Sam views that forward-looking models to assist in decision support are real risk management, as opposed to the backward-looking risk measurement models implemented at banks to support regulatory reporting.

Summary - Sam was a great speaker, and for a change it was refreshing to not have presentation slides backing up what the speaker was saying. His thoughts on forward looking models being true risk management and moving away from risk measurement seem to echo those of Ricardo Rebanato of a few years back at RiskMinds (see post). I think his thoughts on P&L motivation being the only way that risk management advances are correct, although I think there is a lot of risk innovation at the banks but at a trading desk level and not at the firm-wide level which is caught up in regulation - the trading desks know that capital is scarce and are wanting to use it better. I think this siloed risk management flies in the face of much of the firm-wide risk management and indeed firm-wide data management talked about in the industry, and potentially still shows that we have a long way to go in getting innovation and forward looking risk management at a firm level, particularly when it is dominated by regulatory requirements. However, having a truly integrated risk data platform is something of a hobby-horse for me, I think it is the foundation for answering all of the regulatory and risk requirementst to come, whatever their form. Finally, I could not agree more easy analysis for end-users is a vital part of data management for risk, allowing business users to do risk management better. Too many times IT is focussed on systems that require more IT involvement, when the IT investment and focus should be on systems that enable business users (trading, risk, compliance) to do more for themselves. Data management for risk is key area for improvement in the industry, where many risk management sytem vendors assume that the world of data they require is perfect. Ask any risk manager - the world of data is not perfect and manual data validation continues to be a task that takes time away from actually doing risk management.

Posted by Brian Sentance | 14 December 2011 | 11:29 pm

A-Team event – Data Management for Risk, Analytics and Valuations

My colleagues Joanna Tydeman and Matthew Skinner attended the A-Team Group's Data Management for Risk, Analytics and Valuations event today in London. Here are some of Joanna's notes from the day:

Introductory discussion

Andrew Delaney, Amir Halton (Oracle)

Drivers of the data management problem – regulation and performance.

Key challenges that are faced – the complexity of the instruments is growing, managing data across different geographies, increase in M&As because of volatile market, broader distribution of data and analytics required etc. It’s a work in progress but there is appetite for change. A lot of emphasis is now on OTC derivatives (this was echoed at a CityIQ event earlier this month as well).

Having an LEI is becoming standard, but has its problems (e.g. China has already said it wants its own LEI which defeats the object). This was picked up as one of the main topics by a number of people in discussions after the event, seeming to justify some of the journalistic over-exposure to LEI as the "silver bullet" to solve everyone's counterparty risk problems.

Expressed the need for real time data warehousing and integrated analytics (a familiar topic for Xenomorph!) – analytics now need to reflect reality and to be updated as the data is running - coined as ‘analytics at the speed of thought’ by Amir. Hadoop was mentioned quite a lot during the conference, also NoSQL which is unsurprising from Oracle given their recent move into this tech (see post - a very interesting move given Oracle's relational foundations and history)

Impact of regulations on Enterprise Data Management requirements

Virginie O’Shea, Selwyn Blair-Ford (FRS Global), Matthew Cox (BNY Melon), Irving Henry (BBA), Chris Johnson (HSBC SS)

Discussed the new regulations, how there is now a need to change practice as regulators want to see your positions immediately. Pricing accuracy was mentioned as very important so that valuations are accurate.

Again, said how important it is to establish which areas need to be worked on and make the changes. Firms are still working on a micro level, need a macro level. It was discussed that good reasons are required to persuade management to allocate a budget for infrastructure change. This takes preparation and involving the right people.

Items that panellists considered should be on the priority list for next year were:

· Reporting – needs to be reliable and meaningful

· Long term forecasts – organisations should look ahead and anticipate where future problems could crop up.

· Engage more closely with Europe (I guess we all want the sovereign crisis behind us!)

· Commitment of firm to put enough resource into data access and reporting including on an ad hoc basis (the need for ad hoc was mentioned in another session as well).

Technology challenges of building an enterprise management infrastructure

Virginie O’Shea, Colin Gibson (RBS), Sally Hinds (Reuters), Chris Thompson (Mizuho), Victoria Stahley (RBC)

Coverage and reporting were mentioned as the biggest challenges.

Front office used to be more real time, back office used to handle the reference data, now the two must meet. There is a real requirement for consistency, front office and risk need the same data so that they arrive to the same conclusions.

Money needs to be spent in the right way and fims need to build for the future. There is real pressure for cost efficiency and for doing more for less. Discussed that timelines should perhaps be longer so that a good job can be done, but there should be shorter milestones to keep business happy.

Panellists described the next pain points/challenges that firms are likely to face as:

· Consistency of data including transaction data.

· Data coverage.

· Bringing together data silos, knowing where data is from and how to fix it.

· Getting someone to manage the project and uncover problems (which may be a bit scary, but problems are required in order to get funding).

· Don’t underestimate the challenges of using new systems.

Better business agility through data-driven analytics

Stuart Grant, Sybase

Discussed Event Stream Processing, that now analytics need to be carried out whilst data is running, not when it is standing still. This was also mentioned during other sessions, so seems to be a hot topic.

Mentioned that the buy side’s challenge is that their core competency is not IT. Now with cloud computing they are more easily able to outsource. He mentioned that buy side shouldn’t necessarily build in order to come up with a different, original solution.

Data collection, normalisation and orchestration for risk management

Andrew Delaney, Valerie Bannert-Thurner (FTEN), Michael Coleman (Hyper Rig), David Priestley (CubeLogic), Simon Tweddle (Mizuho)

Complexity of the problem is the main hindrance. When problems are small, it is hard for them to get budget so they have to wait for problems to get big – which is obviously not the best place to start from.

There is now a change in behaviour of senior front office management – now they want reports, they want a global view. Front office do in fact care about risk because they don’t want to lose money. Now we need an open dialogue between front office and risk as to what is required.

Integrating data for high compute enterprise analytics

Andrew Delaney, Stuart Grant (Sybase), Paul Johnstone (independent), Colin Rickard (DataFlux)

The need for granularity and transparency are only just being recognised by regulators. The amount of data is an overwhelming problem for regulators, not just financial institutions.

Discussed how OTCs should be treated more like exchange-traded instruments – need to look at them as structured data.

Posted by Brian Sentance | 18 October 2011 | 12:44 am

Internal model approval, risk management and regulatory compliance

Achieving regulatory approval can be challenging if we consider that regulators are concerned about both the risk calculation methodology in place but also the quality, consistency and auditability of the data feeding the risk systems used for regulatory reporting.

The data management project at LBBW (Landesbank Baden-Württemberg), for example, was initiated to support LBBW’s internal model for market risk calculations, combined with the additional aim of enabling risk, back office and accountancy departments to have transparent access to high quality and consistent data.

This required a consolidated approach to the management of data in order to support future business plans and successful growth and we worked with LBBW to provide a centralised analytics and data management platform which could enhance risk management, deliver validated market data based upon consistent validation processes and ensure regulatory compliance.

More information on the joint project at LBBW can be found in the case study, available on our website. Any questions, drop us a line!

Posted by Sara Verri | 22 September 2011 | 7:21 pm

Data Unification - just when you thought it was safe to go back in the water...

Sitting by the sea, you have just finished your MATLAB reading and now are wondering what to read next?

No worries!

We have just published our "TimeScape Data Unification" white paper. Not a pocket edition I am afraid, but some of you may find it interesting.

It describes how - post-crisis - a key business and technical challenge for many large financial institutions is to knit together their many disparate data sources, databases and systems into one consistent framework than can meet the ongoing demands of the business, its clients and regulators. It then analyses the approaches that financial institutions have adopted to respond to this issue, such as implementing a ETL-type infrastructure or a traditional golden copy data management solution.

Taking on from their effectiveness and constraints, it then shows how companies looking to satisfy the need for business-user access to data across multyple systems should consider a "distributed golden copy" approach. This federated approach deals with disparate and distributed sources of data and should also provide easy and end-user interactivity whilst maintaining data quality and auditability.

The white paper is available here if you want to take a look and if you have any feedback or questions, drop us a line!

Posted by Sara Verri | 27 July 2011 | 4:19 pm

MATLAB - The perfect read for the beach...

For those who are wondering what summer reading to take on holiday, we have just published our white paper "TimeScape and MATLAB", a pocket edition which outlines how TimeScape and MATLAB can be combined to provide enhanced data analysis and visualisation tools to financial organisations.

Whilst swimming in the blue ocean, walking in the countryside or enjoying a new country, take a break and find out how TimeScape's best of breed data capture and storage can be combined with the analytical capabilities of MATLAB to produce compelling solutions to real-world problems encountered within financial services.

Ok, ok, kidding here. Just go on holiday and enjoy your time off from complex financial problems!

But when you are back or if you are very interested (or sadly not going on holiday soon), please take a look at our white paper. It details how:

TimeScape data and analytics can be accessed from MATLAB
MATLAB computational and visualization tools can be used to manipulate and analyse TimeScape data
Complex data sets generated in MATLAB can be saved back to TimeScape for persisted storage
MATLAB components can be called from TimeScape to enrich TimeScape hosted functionality

and much more.

Feel also free to suggest this summer reading to your friends (or enemies!).

Posted by Sara Verri | 22 July 2011 | 3:40 pm

PRMIA on Data and Analytics

Final presentation at the PRMIA event yesterday was by Clifford Rossi and was entitled "The Brave New World of Data & Analytics Following the Crisis: A Risk Manager's Perspective".

Clifford got his presentation going with a humorous and self-depricating start by suggesting that his past employment history could in fact be the missing "leading indicator" for predicting orgnisations in crisis, having worked at CitiGroup, WaMu, Countrywide, Freddie Mac and Fannie Mae. One of the other professors present said that he didn't do the same to academia (University of Maryland beware maybe!).

Clifford said that the crisis had laid bare the inadequacy and underinvestment in data and risk technology in the financial services sector. He suggested that the OFR had the potential to be a game changer in correcting this issue and in helping the role of CRO to gain in stature.

He gave an example of a project at one of the GSEs he had worked at called "Project Enterprise" which was to replace 40 year old mainframe based systems (systems that for instance only had 3 digits to identify a transaction). He said that he noted that this project had recently been killed, having cost around $500M. With history like this, it is not surprising that enterpring risk data warehousing capabilities were viewed as black holes without much payoff prior to the crisis. In fact it was only due to Basel that data management projects in risk received any attention from senior management in his view.

During the recent stress test process (SCAP) the regulators found just how woeful these systems were as the banks struggled to produce the scenario results in a timely manner. Clifford said that many banks struggled to produce a consistent view of risk even for one asset type, and that in many cases, corporate acquisitions had exascerbated this lack of consistency in obtaining accurate, timely exposure data. He said that the mortgage processing fiasco showed the inadequacy of these types of systems (echoing something I heard at another event about mortgage tagging information being completely "free-fromat", without even designated fields for "City" and "State" for instance)

Data integrity was another key issue that Clifford discussed, here talking about the lack of historical performance data leading to myopia in dealing with new products and poor defintions of product leading to risk assessments based on the originator rather than on the characteristics of the product. (side note: I remember prior to the crisis the credit derivatives department at one UK bank requisitioning all new server hardware to price new CDO squared deals given it was supposedly so profitable, it was at that point that maybe I should have known something was brewing...) Clifford also outlined some further data challenges, such as the changing statistical relationship between Debt to Income ratio and mortgage defaults once incomes were self-declared on mortgages.

Moving on to consider analytics and models, Clifford outlined a lot of the concerns covered by the Modeller's Manifesto, such as the lack of qualitative judgement and over-reliance on the quantitative, efficiency and automation superceding risk management, limited capability to stress test on a regular basis, regime change, poor model validation, and cognitive biases reinforced by backward-looking statistical analysis. He made the additional point that in relation to the OFR, they should concentrate on getting good data in place before spending resource on building models.

In terms of focus going forward, Clifford said the liquidity, counterparty and credit risk management were not well understood. Possibly echoing Ricardo Rebonato's ideas, he suggested that leading indicators need to be integrated into risk modelling to provide the early warning systems we need. He advocated that the was more to do on integrating risk views across lines of business, counterparties and between the banking and trading book.

Whilst being a proponent of the OFRs potential to mandate better Analytics and data management, he warned (sensibly in my view) that we should not think that the solution to future crises is simply to set up a massive data collection and Modelling entity (see earlier post on the proposed ECB data utility)

Clifford thinks that Dodd-Frank has the potential to do for the CRO role what Sarbanes-Oxley did in elevating the CFO role. He wants risk managers to take the opportunity presented in this post-crisis period to lead the way in promoting good judgement based on sound management of data and Analytics. He warned that senior management buy-in to risk management was essential and could be forced through by regulatory edict.

This last and closing point is where I think where the role of risk management (as opposed to risk reporting) faces it's biggest challenge, in that how can a risk manager be supported in preventing a senior business manager from seeking a overly risky new business opportunity based on what "might" happen in the future - we human beings don't think about uncertainty very clearly and the lack of a resulting negative outcome will be seen by many to invalidate the concerns put forward before a decision was made. Risk management will become known as the "business prevention" department and not regarded as the key role it should be.

Posted by Brian Sentance | 24 June 2011 | 4:26 pm

IKEA and Market Risk Management – Choice is a worrying thing!

Risk management and data control remain at the top of the agenda at many financial institutions. Many have said that the recent crisis highlighted the need for more consistent, transparent, high quality data management, which I totally agree with (but working for Xenomorph, I would I guess!). Although the crisis started in 2007, it would seem that many organizations still do not have the data management infrastructure in place to achieve better risk management.

I moved apartment last week and had to face the terrifying prospect of visiting IKEA to buy some new furniture. On walking through the endless corridors of furniture ideas I wondered whether the people at major financial institutions feel as I did: I knew I needed two wardrobes, I knew the dimensions of the rooms, I knew how many drawers I wanted. Then I got to the wardrobes showroom, sat in front of the “Create your own wardrobe” IKEA software and the nightmare started. How many solutions are there to solve your problems? And how many solutions, once you get to know of their existence, make you aware of a problem you didn’t know you had? That’s how I spent 2 days at IKEA choosing my furniture and still I wonder whether in the end I got the right solution for my needs.

Coming back to risk management, I imagine the same dilemma may be faced by financial institutions looking to implement a data management solution. How many software providers are out there? What data model do they use? Are they flexible enough to satisfy evolving requirements? How can we achieve an integrated data management approach? Will they support all kind of asset classes, even the most complex?

In these times of new regulations where time goes fast and budget is tight, selection processes have become more scrupulous.

As often happens in life, when we need a plumber for example, or a new dentist, we look for positive recommendations, people willing to endorse the efficiency and reliability of the service. So, with this in mind, please take a look at the case study we put together with Rabobank International, who have been using our TimeScape analytics and data management system at their risk department since 2002 for consolidated data management. More client stories are also available on our website here: www.xenomorph.com/casestudies.

I hope that many of you will benefit from reading the case study and for any questions (on IKEA wardrobes too!), please get in touch...

Posted by Sara Verri | 8 June 2011 | 10:07 am

More formal management of instrument valuation needed

Xenomorph has today released its white paper “Instrument Valuation Management: management of derivative and fixed income valuations in a multi-asset, multi-model, multi-datasource and multi-timeframe environment”.

The white paper expands on the “Rates, Curves and Surfaces – Golden Copy Management of Complex Datasets” white paper Xenomorph published recently (see earlier post) and describes how, despite the increasing importance of instrument valuation to investment, trading and risk management decisions, valuation management is not yet formally and fully addressed within data management strategies and remains a big concern for financial institutions.

Too often, says Xenomorph, valuations (and the analytics used to process input and calculate output data) fall between traditional data management providers and pricing model vendors. This leads to the over–use of tactical desktop spreadsheets where data “escapes” the control of the data management system, leading to an increased operational risk.

Whilst instrument valuation is certainly not the primary cause of the recent financial crisis, the lack of high quality, transparent valuations of many complex securities resulted in market uncertainty and in the failure of many risk models fed by untrustworthy valuations.

“A deeper understanding of financial products reduces operational risk and promotes quality, consistency and auditability, ensuring regulatory compliance”, says Brian Sentance, CEO Xenomorph. “Clients’ requirements have evolved and portfolio managers, traders and risk managers recognize that it is no longer sufficient to treat valuation as an external, black-box process offered by pricing service providers”, he adds.

Nowadays, regulators, auditors, clients and investors demand even more drill-down to the underlying details of an instrument’s valuation. It is therefore important to implement an integrated, consistent analytics and data management strategy which cuts across different departments and glues together reference and market data, pricing and analytics models, for transparent, high quality, independent valuation management.

“Our TimeScape solution provides a valuation environment which offers rapid and timely support for even the most complex instruments, allowing our clients to check easily the external valuation numbers, based on their choice of model and data providers”, says Sentance. “Otherwise, what is the point of good data management if the valuations and the analytics used are not based on the same data management infrastructure principles?”

For those who are interested, the white paper is available here.

Posted by Sara Verri | 4 May 2011 | 1:41 pm

Rates, curves and derived data management remains a neglected area following the crisis

Xenomorph has released its white paper 'Rates, Curves and Surfaces – Golden Copy Management of Complex Datasets'. The white paper describes how, despite the increasing interest in risk management and tighter regulations following the crisis, the management of complex datasets – such as prices, rates, curves and surfaces - remains an underrated issue in the industry. One that can undermine the effectiveness of an enterprise-wide data management strategy.

In the wake of the crisis, siloed data management, poor data quality, lack of audit trail and transparency have become some of the most talked about topics in financial markets. People have started looking at new approaches to tackle the data quality issue that found many companies unprepared after Lehman Brothers' collapse. Regulators – both nationally and internationally – strive hard to dictate parameters and guidelines.

In light of this, there seems to be a general consensus on the need for financial institutions to implement data management projects that are able to integrate both market and reference data. However, whilst having a good data management strategy in place is vital, the industry also needs to recognize the importance of model and derived data management.

Rates, curves and derived data management is too often a neglected function within financial institutions. What is the point of having an excellent data management infrastructure for reference and market data if ultimately instrument valuations and risk reports are run off spreadsheets using ad-hoc sources of data?

In this evolving environment, financial institutions are becoming aware of the implications of a poor risk management strategy but are still finding it difficult to overcome the political resistance across departments to implementing centralised standard datasets for valuations and risk.

The principles of data quality, consistency and auditability found in traditional data management functions need to be applied to the management of model and derived data too. If financial institutions do not address this issue, how will they be able to deal with the ever-increasing requests from regulators, auditors and clients to explain how a value or risk report was arrived at?

For those who are interested, the white paper is available here.

Posted by Sara Verri | 24 February 2011 | 5:45 pm

Analytics Management by Sybase and Platform

I went along to a good event at Sybase New York this morning, put on by Sybase and Platform Computing (the grid/cluster/HPC people, see an old article for some background). As much as some of Sybase's ideas in this space are competitive to Xenomorph's, some are very complimentary and I like their overall technical and marketing direction in focussing on the issue of managing of data and analytics within financial markets (given that direction I would, wouldn't I?...). Specifically, I think their marketing pitch based on moving away from batch to intraday risk management is a good one, but one that many financial institutions are unfortunately (?) a long way away from.

The event started with a decent breakfast, a wonderful sunny window view of Manhattan and then proceeded with the expected corporate marketing pitch for Sybase and Platform - this was ok but to be critical (even of some of my own speeches) there is only so much you can say about the financial crisis. The presenters described two reference architectures that combined Platform's grid computing technology with Sybase RAP and the Aleri CEP Engine, and from these two architectures they outlined four usage cases.

The first use case was for strategy back testing. The architecture for this looked fine but some questions were raised from the audience about the need for distributed data cacheing within the proposed architecture to ensure that data did not become the bottleneck. One of the presenters said that distributed cacheing was one option, although data cacheing (involving "binning" of data) can limit the computational flexibility of a grid solution. The audience member also added that when market data changes, this can cause temporary but significant issues of cache consistency across a grid as the change cascades from one node to another.

Apparently a cache could be implemented in the Aleri CEP engine on each grid node, or the Platform guy said that it was also possible to hook in a client's own C/C++ solution into Platform to achieve this, and that their "Data Affinity" offering was designed to assist with this type of issue. In summary their presentation would have looked better with the distributed cacheing illustrated in my view, and it begged the question as to why they did not have an offering or partner in this technical space. To be fair, when asked whether the architecture had any performance issues in this way, they said for the usage case they had then no it didn't - so on that simple and fundamental aspect they were covered.

They had three usage cases for the second architecture, one was intraday market risk, one was counterparty risk exposure and one was intraday option pricing. On the option pricing case, there was some debated about whether the architecture could "share" real-time objects such as zero curves, volatility surfaces etc. Apparently this is possible, but again would have benefitted by being illustrated first as an explicit part of the architecture.

There was one question about the usage of the architecture applied to transactional problems, and as usual for an event full of database specialists there was some confusion as to whether we were talking about database "transactions" or financial transactions. I think it was the latter, but this wasn't answered too clearly but neither was the question asked clearly I guess - maybe they could have explained the counterparty exposure usage case a bit more to see if this met some of the audience member's needs.

The latter question on transactions above got a conversation going on about resilliancy within the architecture, given that the Sybase ASE database engine is held in-memory for real-time updates whilst the historic data resides on shared disk in Sybase IQ, their column-based database offering. Again full resilience is possible across the whole architecture (Sybase ASE, IQ, Aleri and the Symphony Grid from Platform) but this was not illustrated this time round.

Overall good event with some decent questions and interaction.

Posted by Brian Sentance | 20 October 2010 | 8:40 pm

Dodd Frank Regulation - being seen to be doing something?

I went along to a Six Telekurs event "Securities Valuations: Is the Price Right?" last week - good event with some interesting speakers, most notably Paul Atkins of Patomak Partners to talk about the Dodd-Frank Wall Street Reform and Consumer Protection Act 2010. Paul is based out of Washington and was not very complimentary about what has been going on.

He started by saying that the Act was very large in size, with over 2319 pages (compared to SarbOx with only 60) and given this size he suggested that you could guess how many in Congress had actually read it. Background to the Act were:

"Political Tailwinds" such as:
- New Democrat Government with tenuous majority
- Ambitious legislative plans
- Bleak economic back-drop
An angry populace:
- TARP bailouts/Wall St bonuses
- Recession and high unemployment
- Perception that Govt. contributed to crisis
Aggressive case for new regulation based on:
- Lack of confidence in current systems and regulation
- "Too big to fail" demonstrating that regulators lack the toolsets necessary to deal with such events
- High leverage across the financial system and the economy
- Poor risk management by existing participants
- Opaque shadow banking system and opaque derivatives markets

He summarised that Housing and the Credit Rating Agencies were the key fundamentals behind the financial crisis.

Paul said that with the new regulation had the following features:

The Act is a sweeping revision of financial regulation in the US
- few dodged the regulatory changes (notably insurance managed to do this)
The Federal Reserve has emerged pre-eminent amongst all regulatory bodies in the US.
Significant discretion has been yielded to regulators to work out specifics
Sheer size and ambiguous wording of the Act exacerbates the uncertainty in the market and economy and will require further fixes over coming years
The Act does not reform Government Sponsored Enterprises (Fannie Mae, Freddie Mac)
Far from reducing/simplifying the number of agencies involved in regulation the Act eliminated 1 agency and created 13 more
Paul asked the question whether spreads and volatility will rise in the market due to new regulation (such as the Volcker rule) and whether ultimately this will trickle down to hinder or benefit SMEs.
The Act will likely result in regulatory arbitrage opportunities and Paul said this was not a good thing for the United States

Paul said that in his view Congress learned the wrong lessons from the crisis:

No reform of Fannie Mae and Freddie Mac
Government Housing Policy left unaddressed
Transparency still lacking despite efforts from FASB on fair value
International Policy Co-ordination is still an open question as to its extent
No reform of existing regulator structures
The crisis has resulted in payoffs to favoured groups (Unions, Trial Lawyers etc)

Paul talked about how hedge funds and private equity funds were going to experienced increased regulation with them having to register if they have over $100M assets under management and future implications for systemic risk provisions. He mentioned that Venture Capital investments had escaped being required to register if the lock-up period was over 2 years.

He briefly discussed the coming changes in OTC derivatives on centralised clearing, post trade reporting and new liability provisions. Paul was also concerned about certain SEC related issues such as "Whistleblower" provisions which contain a bounty programme of about 10-30% of any fine subsequently awarded against a financial institution. He re-iterated that it was not yet clear what all of the bodies involved in regulation would be doing, and at the same time as this was the case the very same bodies were also being given very strong powers such as that of legal subpoena.

Paul was a very knowledgeable speaker and had some good points to make. Listening to him speak it would seem from my perspective that the Act is a prime example of "being seen to be doing something" to address the crisis rather than something better structured, with all of "law of unintended consequencies" risks that such an initiative entails.

Posted by Brian Sentance | 14 October 2010 | 8:32 pm

« Back to recent posts

Categories

Posts categorized "Analytics Management"