It’s been a while since I wrote my last post, though not all that long since it appeared here. In that time we’ve made a great deal of progress, fought some unexpected dragons, and come out what we think is rather ahead.
Back when I wrote part 1 I had just finished gathering data from a few test environments in preparation for making a decision on a hosting provider. As a major change, it had to go around all the major stakeholders, including those who know how to ask tricky questions. That took a little while since, with support from a major client, we underwent an external audit of our new infrastructure before declaring it ready for testing. This gave us confidence that best practices had been followed and, vitally, gave us the ability to ensure our security-conscious clients of the safety of their data.
After whittling down the options with the twin razors of Practicality and Money, we were left with a choice between two excellent and well-supported platforms: Google Cloud, and Amazon Web Services.
There was a small price difference in Google’s favour and a small breadth of platform difference in Amazon’s favour, but overall our final decision came down to ease of use and documentation. When you’re renting essentially the same thing from the two biggest companies in commodity web hosting, it’s the quality of life differences around the edges that outweigh the differences in the main offerings. For another company, and in particular for another sysadmin with different preferences, this decision might have been reversed – with no ill consequences, I hasten to add.
We eventually settled on using Google Compute Engine for the hosting, monitoring and so on, with a Cloud SQL instance as our database provision and a Cloud Storage bucket for documents and logos. This accomplished the aim of breaking our infrastructure up slightly from the previous situation of running everything on one server, giving us more flexibility to move forward.
Market Dojo is now running on a cluster of smaller instances behind a load balancer, with the shared state (messaging, file storage and database) abstracted to the other services. This means that each portion of Market Dojo – the application, the database and the document storage – can be scaled separately, granting us enormous flexibility. Under heavy load, we get bigger to best serve the requests. Under light load, we shrink, so saving both money and energy.
Further optimisations on the roadmap for next year, with the hope that we can deliver significant performance improvements as time goes by. We have already seen the benefits of being able to scale horizontally and deploy on a rolling basis, and while we will need to be careful to avoid vendor lock-in we have access to some of the most effective technology available.
To read the first part of this blog series click here.
Market Dojo helps procurement professionals negotiate better with our on-demand eSourcing tools. If you’d like to find out more, get in touch or register for free and play around with our software for yourself!Cloud Services, or How We Learned To Stop Worrying and Love Instances
As Market Dojo expands, as all good startups hopefully do, they have begun to need some additional technical staff on hand to support, expand and maintain the eponymous software on which the business depends.
I am the first of these. James, at your service. Well, more at Market Dojo’s service, but since their goal (now, indeed, our goal) is to provide the best possible service to their clients there is a certain amount of inheritance.
One of my first duties in this role has been to take at least partial ownership of the question “how and where do we host Market Dojo?”. Owning and managing our own ‘bare metal’ servers would be uneconomical both in terms of outlay and maintenance hours, so we have for the last few years outsourced that particular problem to $MediumHostingProvider.
For a while now, however, the prevailing view within Market Dojo has been that our contract with them has been ‘coming to an end’. This is in some ways a shame as they were an early partner in our journey and have provided us with excellent service over the last few years. Nevertheless, as our platform expands, with new features each release and a growing number of clients, the demands we place on our infrastructure itself grow accordingly.
Currently the Dojo operates as a single shard on a single VPS (Virtual Private Server); this has been fine and workable for a long time, but our desire for resilience and scalability is pushing us to look further afield for solutions – there’s only so much peace of mind a good backup system can produce.
It was suggested and indeed attempted, that we could improve performance at peak times by increasing our various provisions. Since at Market Dojo we use Ruby on Rails as our app framework the expectation was that we would see increased responsiveness with an increase in memory in particular, as although the number of CPU cores available makes some difference Rails is less parallel than one might like.
On evaluation, this solution struck the team as decidedly suboptimal. In particular the difference in usage between times of high demand, such as the Monday morning rush of auctions from our larger clients getting the head start on the week, and times of low demand such as that experienced late on a Sunday afternoon grows linearly with the number of users; the latter case is negligibly close to zero usage by comparison. This would mean that simply increasing our VPS size was more apt to waste money and electricity than to provide a noticeable all-around benefit to both us and our clients.
Better solutions are out there. There are some very large cloud providers offering both IaaS (Infrastructure as a Service; Virtual Private Servers, for example, managed by the user) and PaaS (Platform as a Service; managed services which take care of the deployment, building and running of the background pieces behind an app, such as databases and web servers, as well as hosting) for various needs, most of which are intended to be scaleable.
After some initial winnowing, we settled on a shortlist of three: IBM Bluemix/Cloud Foundry, AWS Elastic Beanstalk and Google App Engine for PaaS, and the same companies’ offerings for IaaS – SoftLayer, EC2 and Compute Cloud respectively.
We had a number of requirements to consider in order to bring this down to a final choice.
That last point was underpinned by a side concern of backward compatibility. Portions of the Dojo are in the process of being refactored, improved and reconstructed, others have already been seen to, but others still are yet to be addressed in this cycle. It’s an undeniable fact that any piece of software older than one or two years will be at least partly outdated, unless it is particularly small or has a particularly deific development team behind it. For our purposes this ruled out any platform which enforced restrictions on versions of Ruby, Rails, or any of the various Gems upon which we rely.
The decisions became easier from there on.
|Benefits||Full control||Someone else administers|
|Choice of web server||Provided managed web server|
|Billed by spec (predictable)||Billed by usage (auto-flexible)|
|Powerful CLI (Usually)||Easy GUI (Usually)|
|Typically cheaper than PaaS||Low technical knowledge barrier to administration|
|Can be expanded in real time to cope with changes in demand||Can expand themselves in real time to cope with changes in demand|
|Easy to automate setup via images or Puppet||Easy to automate setup via config file and push hooks|
|Drawback||Have to administer manually||No control over software|
|Responsible for software and dependencies||Often hard to install additional dependencies|
|Requires technically competent administrator to make any changes||Narrow access routes|
|Typically more expensive than VPS|
|Can require official technical support to get anything major done|
Tables make everything better. There wasn’t a justifiable reason for a good chart or map in this particular investigation, but there were plenty of data to work with all the same.
There were also a great many emails. An important thing to consider when you’re contemplating a move of this kind, especially where it affects the nature of your underlying infrastructure, is how your larger clients might be disposed towards the changes.
In this case, we consulted with two of our most active large clients, who were helpful in providing both recommendations and requests for the new infrastructure, as both companies have their own security teams. The ability to engage in this sort of collaboration with clients is a boon to all involved, relying heavily on the fact that commercial relationships in general, and security in particular, are inherently positive sum.
Elements of the feedback are confidential, but from our perspective one of the most important points raised was that there is an expectation of external assessment. If you happen to be planning a move of this kind, it would be advisable to budget from the beginning for engaging an independent assessor after the move has been made. We encountered this request before it became relevant, which is another advantage of having this consultation early in the process, so it didn’t constitute a temporal setback despite being effectively a change in scope.
In the ‘known knowns’ column, some of our clients have a strong preference for their data being stored in the EU, subject to the Data Protection Act. This rules out the major US data centres. Fortunately, all the services under consideration had data centres in either the UK, Ireland or Belgium. We also had to consider encryption and colocation of data, such as would occur in a shared database or shared hosting. An encrypted-at-rest VPS is the very minimum expected, however, so those were not significant barriers.
Had we been looking at smaller hosting providers, or at self-hosting, we might also have had to consider redundancy of hardware in addition to the redundancy of data; one of the major advantages of using the large cloud providers is that they have all those concerns in hand; barring natural disasters, enemy action or Outside Context Problems they are highly unlikely to suffer full loss of data.
Mitigation of at least the first risk, and in many cases the second, can be performed relatively simply by having backups in a datacentre in another environment. Our particular use case rules out the optimal configuration of having replicas on every continent and under multiple jurisdictions, which lowers the risk of coordinated attack or localised natural disaster (albeit raising the chance of encountering a subpoena); however, all three of our potential suppliers had multiple EU data centres.
All this in hand, we engaged in a deeper dive.
For testing, we set a procedure:
The nice thing about establishing a procedure beforehand is that there isn’t much wiggle room for preference, not enough coffee, or whatever else might be distracting at a given moment. I’m always mindful of a study demonstrating that the simple act of having and using a checklist correctly reduces the incidence of mistakes and negative outcomes.
The result of all this data collection is still to be determined, but so far we’re quite pleased with the results. Overall, we expect to see both an annual cost saving and an increase in performance, at least initially; as we begin to use the potential of the cloud services to mirror, expand and scale the former of those gains may be sacrificed to the latter. That, however, is another article.
Market Dojo helps procurement professionals negotiate better with our on-demand eSourcing tools. If you’d like to find out more, get in touch or register for free and play around with our software for yourself!5 ways leading CPO’s are looking to embrace procurement technology
Deloitte recently published their Global CPO Survey 2016, where over 300 leading procurement professionals from 33 countries were questioned on key issues. The full report can be found here.
This year there was a particular focus on emerging technologies in the procurement space and the value that they can provide to procurement functions.
We have identified 5 of their key findings.
We are very encouraged that CPO’s have identified the importance of technology in achieving their procurement goals. The thoughts captured above echo the very principles upon which Market Dojo was created; an intuitive cloud-based eSourcing solution with a strong focus on self-service leading to wide adoption and utilisation. Hopefully, this makes us well placed as CPO’s look to implement their vision for digital solutions.
Market Dojo helps procurement professionals negotiate better with our on-demand eSourcing tools. If you’d like to find out more, get in touch or register for free and play around with our software for yourself![NEW RELEASE] Here’s what our developers have been up to…
One of the many beauties of providing SaaS technology is that our clients don’t need to spend time mass updating their systems and reinstalling the latest release of our software. We are a multi-tenanted application, meaning that there’s always one version of Market Dojo available to everyone.
This doesn’t mean that new features and benefits aren’t being released. Any little nuances you find with our software are all logged via our support portal (check out this article on how to log issues and the processes we follow to deal with them).
Our development team have been working very hard for this release to bring you a list of new features which we hope you’ll love…
Hyperlink documents in Questionnaire export report
Within Market Dojo, there are a number of types of question you can ask your suppliers. One of which allows you to reference a document. After the event has ended, the host is then able to download a report of the event.
This update allows the event host (buyer) to click on the specific document that has been referenced and easily see the related one. This can save reviewers a lot of time, as it makes it much easier to check the documentation before giving a score.
Re-invite participants in SIM Dojo
This update allows you as a host to resend the instructions to a supplier on how to onboard to SIM Dojo so as to prompt a Participant who has not answered.
This feature exists in Market Dojo, and will now work in a similar way within our onboarding solution with an option to ‘re-send invite’ to participants. The option will not be visible after participants accept the invite.
For suppliers, this will improve their experience by creating a reminder for them to respond. For hosts, this makes it much easier to chase up late suppliers.
Enter in participant email addresses manually
Market Dojo has a number of ways to add participants to an event. Until now adding them by email address manually could be quite slow, as it needed to be done one by one. Now, you can enter (or copy and paste from Excel) several email addresses in one go. The system will even suggest addresses from your participant database. This is something several customers have requested, and will mean event setup can be completed faster.
Along with that, many other issues that you’ve kindly raised with us have been fixed. And as ever, we love showing you how proud we are of these features. Keep the demo requests coming in so we can hear your feedback and show you how to get the most out of our intuitive eSourcing tools.
Don’t forget to BOOK A DEMO NOW!
We are really pleased to announce that we have joined the HP Alliance ONE partner program…
Market Dojo joined HP Alliance ONE partner program and have realised significant benefits. The AllianceONE program provides software vendors the tools and resources they need to more effectively address client needs. Market Dojo work with HP to deliver major upgrades to their platform, to conduct load testing and for locating suitable hosting providers for their cloud solution.
Description of AllianceONE:
HP AllianceONE partner program offers a solid framework for collaboration by integrating servers, storage, networking, security, power & cooling and services. As a member of the AllianceONE program, Market Dojo can significantly extend market reach and improve selling success. Market Dojo can leverage HP AllianceONE solutions, tools and resources to help clients speed time to application deployment, optimize infrastructure capacity, reduce power consumption and free resources to focus on innovation that drives business growth.
Quotes from Market Dojo:
“The Alliance One program has given Market Dojo access to significant support not normally available to a new UK SaaS business. It is fantastic that they have seen the potential of our innovative online solution to help with effective procurement in the private and public sector.” Alun Rafique, Co-Founder
“We are also very happy that the ISV Alliances team have helped us locate Liberata who can address our hosting needs thus helping our clients wanting to deal with cloud solutions, but who still have concerns over security and data.”Nicholas Martin, Co-Founder
“HP is unusual for a large multinational in that they proactively help SME’s and see the long term benefits. They have been invaluable in supporting our work, for example, by providing workshop rooms for our next innovative product partly funded by grant from the TSB (Technology Strategy Board).” Nick Drewe, Co-Founder
What solution does Market Dojo hope to enhance through AllianceONE:
Market Dojo decided to join the HP Alliance One program due to the unparalleled benefits offered to a new SaaS company seeking to grow in difficult economic conditions. It is also in these times that their innovative tool assisting procurement professionals to reduce their costs becomes of paramount importance and HPs assistance gives Market Dojo solid foundations from which to grow globally.
Last month we surpassed the milestone of our first year in business. Understandably we were delighted, as we read previously that up to four-fifths of start-ups fail in their first 12 months. Yet we feel we are truly on our way to becoming an established player in this market, with a client list that is expanding rapidly. However, when we look back at how we started Market Dojo, there is one aspect that stands out when assessing what helped us reach this goal, and that is the use of cloud technology.
For those who are not familiar with cloud technology, and to admit such a thing is akin to never having watched a Star Wars film or never to have heard a Beatles song, our take on it is the use of software or an application over the web on a ‘pay per use’ or monthly basis which you can use straight away with clear benefits. Think Facebook, LinkedIn, Twitter or YouTube. No installation, no set-up costs, and in many cases no fee. If you like just think of it as the dotcom boom version 2.0, only this time it is here to stay!
When you consider any business, you will find it will most likely have office infrastructure to some extent. Such infrastructure might include a phone system, an e-mail system, a fax system, an accountancy system, a CRM system, a calendar sharing system, a data storage system, a web-hosting system, or even a customer support system. Clearly there can be a lot of systems and many of these are non-core to your business, meaning time and money spent in these areas would be detrimental to other areas of your business.
I can only imagine how much it might have cost us 15 years ago to cater for all these systems. We would probably require a receptionist for the phones, a PA to manage the CRM, an on-site accountant, a very large printed calendar, an even larger filing cabinet and a staffed front-desk for our customer support. We would then need a rather large room to host all this.
This takes me to why we are so grateful for the cloud. At Market Dojo we have embraced the best of today’s technology to turn us into a professional company, as ultimately that is what we wish to extend to our clients.
Our phone system is Skype, where you rent a phone number by the month and only pay for your usage, most of which is free. This includes video conferencing, file sharing and instant messaging for everyone in our company and gives us a global presence by allowing us to rent or cease renting international phone numbers when we like. Whilst Skype is not strictly cloud, as you do install software, you can log in across the world from any computer that has it or that you can download it to, so in many respects it is very similar.
Our CRM is Zoho. We only pay per user per month. It integrates with our other applications, including Skype, which means no repetition. Within our own software development we have the approach of “Don’t Repeat Yourself” (DRY) which any business would be wise to apply to administrative tasks as well. Zoho also caters for the customer support system on the same basis.
Data storage and sharing, calendar sharing and e-mail is all with Google Apps, which is completely free and is as robust as the google.com website, even though everything is still in beta!
Our accountancy software is Clearbooks. Again, pay per month per user. We can even add our accountant to it so that they never even need to set foot in the office, as we can upload all our receipts and invoices to Google Docs.
We don’t have any need for a fax system, but if we did there are plenty of cloud systems out there that charge per use.
And finally our web-hosting is all with an external hosting provider, secure and backed-up daily, paid monthly of course and with zero set-up costs.
The advantages that cloud technology has brought to us are simply enormous. The most obvious one is cost. If I were a sole trader, I would be able to set up and manage all of this for less than £600 a year. I would need no training and no upfront capital expenditure. To ‘house’ all of this technology, all I would require is a computer, in fact no, a smartphone. It would take me all of about 3 hours to set up and I would be happily catered for until I was running a global empire, at which point some apps might start to creak at the edges.
The main risk that companies quote for not examining the cloud as a serious solution can be easily mitigated. Data security is a major and very understandable concern. However, with such a broad choice of hosting providers on the market, you can always source a provider that offers the necessary level of data protection, whether for personal use or for the Ministry of Defence. You can also find cloud-based data back-up providers!
Another major objection is reliability. Today though, the internet is probably less likely to fail than your internal LAN and with an increasing number of companies providing networks that are dual-hosted, these risks are diminishing. In fact, you can actually end up with a more reliable solution than your own internal network. For your servers, do you have back-up generators, CCTV, a fire suppression system, 24/7 security patrols, CESG accreditation, dual-hosting and multiple internet providers? This is certainly what we look for in our hosting provider.
When you compare the old approach to starting a business and the new with all the technology that is now available to us, the two are worlds apart. You don’t need capital expenditure for office infrastructure, nor the staff to manage it, nor the space to house it. You don’t need a hardware refresh, nor have costly upgrades as you grow or as old technology becomes redundant. There is a valid question over whether you need an office at all. And best of all, you only pay for what you use.
Just like you do with us!