For the past year, I have received numerous amount of requests on automated trading platforms, crypto investment strategies, algorithmic and high frequency trading in the crypto market. Most of you were also interested in my technology stack.
Here is a summary of my technology stack and the tools that I have leveraged to give you an idea on what actually is involved, if you intend to build one yourself in the near future.
This platform that I have built, I am using it both as a means of using neural networks for algorithmic trading and also optimising my daily crypto currencies trading by capitalising on market arbitrage. I have improved it over the years with better technology stacks and this is my latest.
TIP: Use a boilerplate to start off and progressively build up
If for some reason you intend your platform to be public, I highly recommend hosting it through an AWS S3 bucket and serving it through AWS Cloudfront as your CDN(Content Delivery Network). Nothing faster than compressing all your assets and serving your frontend using a CDN, its all about the cache :O
Back End API
For my platform, I’m making use of API Gateway as my centralised API endpoint and I filter out to the various exchanges APIs with lambdas (12 crypto exchanges broker apis, used for my Market Arbitrage trading)
I’m also using my own nodejs apis built on docker containers and deployed on a small Kubernetes cluster. Here you have an option to use AWS ECS. I will be moving mine to AWS EKS once it becomes available in Sydney region 🙂
I highly recommend Kubernetes if you are looking for a production grade container orchestration platform for your APIs. Bestt bang for your buck in optimising the throughput of your APIs. And if your infrastructure is on AWS, you can use KOPS to provision while waiting for EKS.
Machine Learning, AI
1st Phase: Neural network trained and deployed on self managed Kubernetes cluster (3 master, 9 minions)
2nd Phase: AWS SageMaker + Inbuilt HPO (Hyperparameter Optimization) (This saved me a tonne of time)
In my experience, almost every data scientists I’ve trained or mentored spend 70% of their time trying to figure out how to deploy/scale their model, platforms and automating builds. And only 30% of their time working on improving and fine tuning their algorithms.
This is where in my opinion AWS SageMaker truly shines. All you need is your Jupyter notebooks and SageMaker takes care of the rest. Build, Train and Deploy. If thats not enough it also provides you with an A/B testing feature to test your models.
Servers – 1st Phase : AWS Spot Instances (GPU Optimized) (I’m using spot instances here for cost saving purposes and also im running a transient cluster, i.e spin up instances->data crunch->terminate instances)
Code: Jupyter Notebook (Codebase for my algos)
Frameworks: Tensorflow, NLP for sentiment analysis
Web crawler: Scrapy
Data: Kaggle, Twitter, CoinDesk, Reddit, Cryptotalk, CoinMarketCap, icoalert
TIP: Start small, get things working with a simple jupyter notebook or test out the simple algorithms that is already prebaked with SageMaker. Play around with the various features of SageMaker and then start building on your fancy algorithms.
AWS S3 – Nuff said
Data Ingestion pipeline
Lambda -> Kinesis->Lambda->S3->Lambda->AWS Elasticsearch
Kinesis makes it easy to ingest and process real time data. For the lambdas, I’m using the provided aws python sdk for my
KPL and KCL
Google Firebase (mainly for real time updates and sync, with Vuejs)
AWS Elasticache (In memory Database, Redis)
As a rule of thumb, automate everything. Every manual step gives a opportunity for failure.
If it can be coded, it can be automated 😉
From my git push to deploy I’m using the AWS suite
Codepipeline, CodeCommit, CodeBuild, CodeDeploy
AWS Cloudwatch, Elasticsearch-Kibana
AWS Cloudwatch helps me monitor most of my AWS resources to a certain level
On top on Cloudwatch I’ve got my logs sending to a simple ELK stack (Elasticsearch, Logstash Kibana)
Terraform, (Infrastructure as Code), Packer (For automating custom AMI builds)
My entire infrastructure on AWS is provisioned and automated through Terraform. For those of you are you are tired of writing 500 lines of Cloudformation yaml templates, I highly recommend Terraform to provision any components of your Infrastructure.
I’m using KMS to encrypt my various exchanges credentials and keys and param store for safe retrieval.
As you can see my tech stack covers a wide range of technologies and each section deserves it own blog. When you put it all together it paints a beautiful picture that words cant describe. This is why to me computer science will always be more art than it is science. I hope this summary of my technology stack gives you a high level overview of the tools and technologies and a better idea of whats involved in the world of automation, machine learning and AI.
If you or your company are struggling with any of the above mentioned technology stack, you know who to call 😉
All data and information provided on this site is for informational purposes only. The Cloudten Blog makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this site & will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis. This information is general in nature and has not taken into account your personal financial position or objectives. Before proceeding please refer to a licensed adviser or tax agent and relevant PDS for product details before proceeding.