I worked in HFT for a while. I find the whole space fascinating. I'm glad OP found similar thrills.
"This aspect, the platform itself, seems to be often overlooked in most discussions. Many conversations revolve around strategies (mean reversion, trend following, linear regression, etc.), and backtesting, without fully addressing the practical mechanics or logistics of strategy implementation, particularly in the context of live, intraday trading."
I'm glad you had fun, OP, but also I think I can shed some light on why most people discuss strategy.
Trading is a perfect storm of ridiculously high tech, ridiculously complicated, ridiculously regulated (Not over-regulated, mind you, this isn't a value judgement. But the amount of regulation is extremely high.), and ridiculously competitive.
But that said, it's the last bit that drives it all. Since it's so competitive, even though building an order entry system, and a risk system, and a position-tracking system, etc is a huge accomplishment (again, congrats OP!), it's table-stakes to even dip your toes in the pool here. Trading shops can attract top talent and robust, bespoke trading systems are basically cost of entry.
So people talk about strategy because everyone already has the table-stakes stuff and are now trying to make money with it.
It doesn't help, too, that lots of market participants aren't even playing the same game. In HFT, we operated on trades with alphas that lasted a few seconds, where races to entry/exit were battled in shaving nanoseconds off FPGAs being able to shoot out orders and microseconds off wireless networks flying market data around new jersey. Meanwhile, banks are more concerned with elections and geopolitics than they are about the weather in Carteret. (Rain = no microwave network for the day). And then there's a million strategies in the middle with alphas that last from hours to weeks.
So it makes it really hard to even speak the same language to each other when talking in common forums.
Not only are the platforms table stakes they are the more straightforward part to build. Even at the bleeding edge of latency you can usually work your way to the limits of your platform budget without having to find anything novel.
The strategies though are where the discovery is. There are a few strategies that are well known and still profitable but those are largely consolidated to the biggest firms. For everything else it’s a discovery process. And done strategies are only profitable for very short regimes.
I miss it sometimes too, but so much has been consolidated it’s largely a big firm world now.
More straightforward, heh, sure. But still damnably complicated. Which just goes to show how much money and how much engineering talent is invested in this world that these things are so taken for granted.
Essentially yes. When I worked in algo trading, it never bothered me that we were extracting profits from the markets, nor that we served little social good. It felt like a step up from where I’d been before (being told that we were making the world a better place, when every engineer knew otherwise.) At least we weren’t making things obviously worse.
What did bother me, and was acknowledged by my coworkers, was how much top talent was being pulled away from productive tasks to essentially wank around. Not just technologists either, but all sorts of mathematicians and scientists were drawn to the flame.
We as a society have managed to allocate so many of the “best and brightest” to either fintech wankery or placing ads in front of eyeballs. It’s enough to make one want to give up on capitalism, except that everything else appears to be worse.
Ads are tremendously useful, just ask any business owner.
The problem with ads is not that they're useless, it's that it is an industry prone to scams and grift. (Because doing advertising right requires all sorts of actual science, and ain't nobody got time for that when there's money to be made.)
Seems like every person I've ever met in M&A has ended up hating life because of their career choice. They seem to burn out in 5 years or so after salting away a million bucks or 3. Was that your feeling doing the HFT thing?
The jobs in algo trading are very interesting for technically - mathéamtically inclined people. It’s really one of those fields where you have a direct impact on the results of your - measurable in additional dollars made.
There is 0 social impact. That’s the downside of course - but hey, how many jobs out there are really having any kind of positive social impact ? Not 0, but close to it.
That impact is not made by HFT - finding the right risk premia for different investments is very valuable but that is a signal measured over days/weeks/months because actual capital investment decisions take that long.
Intraday financial games are zero-sum. What HFTs gain, they leech away from mutual funds and pension funds and retail investors and market makers who operate over a longer horizon.
I work on very interesting technical problems with very smart colleagues for excellent pay. All my career progression is in compensation, so I can remain an IC forever and no one thinks that's a negative. I'm subject to no politics whatsoever, and there's very little politics in the company as a whole. The work I do every day has a direct material impact on the company and I'm rewarded proportionally to my impact. My WLB is so-so, but it's better than it was in grad school so I'll take it.
Regarding social impact, the world does have some demand for liquidity and price discovery. Providing those services is both essential and extremely difficult. It's definitely not the most social good I could be doing with my talents, but I think it's weakly positive.
What would rising levels of being lifted out of poverty suggest? The equality gap doesn’t count for much if you starve or freeze to death, and poverty is declining world wide.
The worst way to measure it is inequality. How the top person is doing relative to the bottom person isn't important. How the bottom person is doing now compared to the bottom person 100 years ago is.
sorry, i don't believe inequality is rising, at least in ways that matter.
in the last 50 years, a substantial number of people were pulled out of poverty.
Obviously this would mean that countries that previously relied on subsidised labour from those third world countries would have to start paying up. I'm not suggesting that it is a zero sum game however.
>We as a society have managed to allocate so many of the “best and brightest” to either fintech wankery or placing ads in front of eyeballs
It's nothing to do with "we as a society". I'm a quant trader and know many others, and the vast majority are in the industry because we care about making money not some leftist save the world crap. Even if socialists managed to completely destroy the financial market, we'd just find another way to make money without trying to save the world (e.g. like the mostly corrupt officials in Russia and China when they were communist). "Society" can't change human nature; even mass brainwashing on the scale attempted by Maoist China failed.
I.e. the best and brightest will always work where they want to work, not where leftists like you want to "allocate" them.
I’m a lot less leftist than you apparently assume. I’m a landlord, among other things. My problem with our system is that the things which “pay” are not things which have much benefit to society. I’ve got zero fault with you for choosing to make money. Didn’t I say I did the same thing and loved it? I just question the system that drives the behavior.
I think you just illustrated the problem. You have complete awareness of what you do, and you somehow manage to see it as a battle against "leftists" rather than selfish greed. I'm cool with that, but it precludes you from having valid opinions on anything related to improving the world for the majority of people. Please accept this and refrain from any social commentary. Your comment above demonstrates that you don't actually operate in society.
> Please accept this and refrain from any social commentary. Your comment above demonstrates that you don't actually operate in society.
Cut the fascist bullshit. Everyone can comment about how society should be run, even people that are very successful financially.
If you try to exclude greed from the design of your societal system, it will immediately fail. In large numbers, economics shows us that altruistic people wash out of the model and everyone operates in their own self-interest (greed).
Self interests have externalities. The pleasure you gain from improving someone's life is in no way comparable to the pleasure you gain from making other people suffer. It's a fallacy to say "everyone operates in their own self-interest". It's usually parroted by anti-social people in order to justify anti-social behaviour.
> Cut the fascist bullshit. Everyone can comment about how society should be run, even people that are very successful financially.
It's not financial success that precludes someone from commenting on how society is run, it's blatant, deliberate anti-social actions. This is the foundation of a social contract. It's the same reason you'll get locked up if you go around punching people in the face.
Also, I don't think you know what "fascist" means.
You’re suggesting that people can’t participate in societal planning because they disagree with your causes. You just dress it up in overly dramatic phrases like “anti-social” and “selfish greed” to make inhumanely oppressing them palatable. This was bog standard behavior of fascism in Italy.
It doesn’t seem like fascism to you because you’ve done some mental gymnastics to pretend you’re not just crushing intellectual opposition, but that’s all it is in the end. Op is not punching anyone in the face. He/she is just blatantly unsupportive of socialism.
Focusing solely on getting rich to the detriment of other people is the antithesis of "societal planning". I'm not sure why you're defending this behavior. Do you really believe that people who strive for profit over the well-being of other people are in any way being "social"? Do you see them as the necessary negative force for positive action to be possible? Do you really believe that humans require adversity to succeed, and that those who choose to be adverse agents are engaging in some self-sacrificial martyrdom so society can benefit from their negative influence?
Please, I want to know; what benefits do the actions of the op have for "societal planning"? The person he was replying to was at least honest in admitting that this behavior "served little social good".
Once again, I think you'd do well to actually read what fascism actually stood for. The propaganda of fascism had a lot more to say about nationalism, "shared history" and the good of corporations than it did to say about things being "anti-social" or even mentioning "selfish greed".
Largely, though I receive 1 or 2 job specs every week for start ups with the keywords 'hft' and 'low latency'. Admittedly there's going to be duplication there if you read them closely.
I think it's a bit of a myth that (ignoring FPGAs) that writing a low-latency software trading system is a time/cost expensive process. Anecdata = I worked at two firms where we did a rewrite from scratch with teams of 5-6 people and traded in the market within 3 months. I'd argue a senior dev that's been around the block a few times could achieve similar when you remove corporate politics, and bikeshedding over design.
The big firm part is paying for multiple quants at $200k++ to come up with strategies and historic market data access for trading models. Small firms are getting backing as long as the co-founders are 'ex-CxO from MegaCorp'.
> I think it's a bit of a myth that (ignoring FPGAs) that writing a low-latency software trading system is a time/cost expensive process.
This depends a lot on the complexity of the trading system and the trading venue specifics. A system to trade single stocks or futures can be built, certified and running in 3 months. A system for options market making will take a lot longer.
Yes, stocks and futures was exactly what I was talking about. Though we also hit the market with options models within another couple of months.
The big costs for small firms are historic data (if you don't have any), colo, distance to exchange, and number of connections.
From the number of job specs I see, it feels like the HFT/low latency market place is healthy enough that there are always new firms appearing. It's competitive, so it's hardly surprising that if someone has new ideas they'll find a backer.
HFT provides a great deal of liquidity and efficient pricing in markets that are adapted to it. It provides a real service, allowing people to transact without using a large bank or broker efficiently on an open market. The fact you can click buy and it buys on almost any stock is likely due to a HFT on the other side. That may not mean much to you directly but it does provide a lot of utility in markets. The biggest gripe people bring about HFT is it during high volatility HFTs usually pull out of the market at a time when people really value the increased liquidity. I think some of the more advanced HFT firms though have moved into longer time range trades which helps provide more liquidity in those markets.
Depends on their strategy but a lot of strategies depend on some sort of price discovery which requires having a probable estimate of what the current price should be and some form of market making around the level. In a very dislocated market the price is unknowable and it becomes gambling, and generally market making strategies are explicitly not about gambling but about facilitating trades around the “true” price.
It may seem so at the first glance, but such trading if carried out by many independent providers actually increases the liquidity if the market for everyone else.
Likewise with crypto. Personally I don't think that if you're calling an API over an Internet it matters if your trading bot is written in go or python (mine was in python). Use the language you're most comfortable in. The network and trade submission/execution at your broker will be 10x slower than your bot anyway. Unless the size of your operation approaches the size where you can get direct market access which seems to be reserved for big forms only.
"increases the liquidity if the market for everyone else"
That's just it. The whole premise is pretty absurd. The market, the actors, everything. It's so far removed from literally anything remotely human. It's the financial equivalent of an infinite sea of AI bots producing CVs and research papers which are only being evaluated and read by other bots.
If you step away from it all for a second, what the hell is the endgame of this whole hustle.
It was always the only end game. AI on quantum computers is the ultimate computing technology — basically the final research technology in our made up little game of Civilization.
Anytime you create technology that sufficiently replicates the creators, you end up with the spirit of the creator embodied in the technology. So, of course digital brains are going to do weird things like crossword puzzles and sodoku at scale, because it's the same kind of useless shit we do to entertain ourselves.
You get into this conversation whenever you dive hard into cyberpunk, which is so many, many things. Even the Internet itself started out that way. The endgame started as a game, and it will end up being a game, played by our creations as odd mirrors of their creators.
I think there's a lot of people who subscribe to doing the same thing to save the planet, of which I have a keen interest. Solarpunk is the name of that movement, and it also has similar crazy ideas.
We're inventing digital brains. It's literally an architecture designed to be removed from being anything remotely human as it's a mimic or replacement technology for intellectual capacity.
If you really take a step back, the endgame is crazier than just bots producing content merely for other bots to consume (which basically describes the vast majority of scientific papers these days too, ironically). When you take that absurdity and multiply it by tens of thousands in terms of efficiency, the whole system we're building looks WILD and almost inconceivably strange to the way we do things now.
I know this is only tangentially related, so I appreciate your understanding that I already understood that and wrote this anyways. :)
At any point in time yes, this is an accurate description. When you integrate over time it’s not. At the end of the line it’s people betting with advanced tools for execution of their bets. Behind the mathematical beauty of risk free and arbitrage, there is a reality of someone, somewhere making a bet at end of a line of contracts. And there is also someone making the opposite bet. Markets are not disseminating information but beliefs as a proxy for information, and as such they are much more human than you would think. The end game is a network of beliefs, livelihoods, politics and petty or grand aspirations, the same mess as in any organisation. The only reason you question the futility of this one specifically is that it’s a clean room abstraction of the futility embedded everywhere. I would call it futility as a service.
Your analogy isn't applicable here. What the OP was trying to get at is that even an individual who doesn't know anything about markets, HFT, liquidity, etc can still benefit from high liquidity from HFT (since it allows for transactions to occur sooner and quicker). In the AI example, the implication is that the final product isn't benefiting consumers.
I mean, efficient allocation of capital is the overarching goal. HFT is either a facilitator of that or a perverse byproduct, depending on who you ask.
It's quite a statement. You're almost saying capitalism and efficient markets are pointless. Maybe they are, but I think it's nothing like crypto.
In the old days before HFT, you weren't sure you'd get the best price. You'd have to rely on a broker to make sure that happens, but as a retail trader you generally got a worse price/out of date price.
Nowadays with HFT you can get pretty much the best price anywhere. Those <1ms HFT traders make that happen. It's the efficient market hypothesis in effect, made possible by HFT.
Now is it pointless to shave off even more ns in the all out war to grab a piece of the order flow? For retail and institutional investors, at a certain point, yes it's completely pointless.
But it's also just pure capitalism at work. HFT firms compete against each other, and the competition is about speed to provide the best price and volume. If you try to regulate with something like enforced delays, then what do you compete on instead?
> You're almost saying capitalism and efficient markets are pointless.
Capitalism necessitates efficient markets (and efficient markets necessitate HFTs) so any criticism of HFTs is a direct criticism of capitalism as well. I mean this isn't really a problem that's specific to HFTs -- there are just a lot of jobs that we can perceive as providing no value or even negative value (jobs that are possible specifically due to capitalism e.g. payday loans, 2008 style trading, certain scams or predatory practices etc.)
I think we can all agree that this is a flaw, and we're not criticizing capitalism to replace it with something else, but rather just recognizing this as a problem. At the end of the day, a useless job is a useless job even if it exists solely because of capitalism.
You are profiting off workers as a middle man in the economy by doing HFT, and trying the justify it by some vague concept of the "correct price". You are producing nothing of value, merely taking away value before someone else notices it is there.
>You're almost saying capitalism and efficient markets are pointless. Maybe they are, but I think it's nothing like crypto.
People are saying this because, HFT sounds similar to 'crypto mining'.
That's people with best infrastructure, the 'big-guys' -- win.
While leaving out the retail investors as broiler chicken, pumped with 'drugs' (by influencers) to spend more on imaginary assets, so that they can be used for 'food' by these 'big-guys'.
There are different influencers for retail investors vs crypto.
In retail investing there are promises of 'retirement paradise', actual tax deductions, the Jim Cramer-like people (at least what I heard in US)
For crypto investing the influencer are different, the geography is wider.
A promise to participate in markets if you do not live the country that has adopted US/UK-based financial services.
- - -
By the way, I think the markets will still have liquidity if there is a rule to wait, for say, 30 min before a stock that was just recently bought -- can be sold (unless by a clear fat finger mistake)
This rule will cause the HFTs to stop existing in the current form.
So are retail investors truly no different from the people gambling their money on the lottery at the 7/11 every night?
Do any regular pedestrian middle class retail investors EVER actually earn enough to retire on? Or do anything meaningful with? If so.. what is THEIR secret sauce, since it's not HFT...
a)Members of political elite that get insider trading stock tips.
(illegal of course). The number of folks in usa congress and senate that become
'very lucky' investors after they join the rank, has to be amazing
b)Lucky
c) everybody else -- that looses.
Overtime, I would say last 30 years, the amount of 'influencing' retail investors to trap them into unreasonable actions had gradually increased.
So the percentage of folks going becoming the victims of the charade, will become higher.
Certainly if you concentrate on ( b ) you create the plausible deniability defense for the manipulators
Nonsense - what could be more beneficial to society than providing imaginary "liquidity" by interposing yourself between legitimate buyers and sellers?
It's a bit like stock brokers - and why wouldn't we want stock brokers to operate at drastically faster-than-human timescales, because we all know the value of a company changes every nanosecond! And "flash crashes" create opportunities for investors to make huge amounts of money!
And just as crypto has poured money into GPU companies (providing opportunities for enterprising secondary market resellers of same) HFT has poured money into networking companies.
Short of creating the great firewall or helping governments slurp up all the traffic on the internet, what could be a more beneficial application of network technology?
HFT (like finance in general) has also gobbled up lots of tech grads, making it easier to find jobs at tech companies.
> Nonsense - what could be more beneficial to society than providing imaginary "liquidity" by interposing yourself between legitimate buyers and sellers?
Now when I say the same thing about index funds people get all huffy
It's similar, though in theory you are paying the index fund to act as your agent, and the fee structure is observable.
Regarding liquidity, I imagine it's possible that the index fund might allow you to withdraw money faster than you could sell on your own, but realistically I kind of doubt it.
The classic theory of longer term investing is that you are providing capital to businesses and rewarded for that. That is a little bit different from being a middle man. My personal opinion though is that lending money/investing capital and capitalism in general is power. Money quantifies the potential of making other humans do things for you, it allows you to excercise power over fellow humans. Inheritance means you can inherit that power from your parents. It's not all that different from feudalism.
YES, 100%! I honestly, cannot agree more. I've read pretty much every book that I could get my hands on and no one talks about the platform. This makes so much sense. Thank you for your perspective. It seems like so much of this stuff is silod off into each company and no one is talking about it.
HFTs are definitely playing a completely different game. I was reading about the exchange architectures and how things are actually wired. I'm getting my data from SIPs while HTFs are directly connected to the exchanges [1]. I'm transacting in seconds and they, like you said, are transacting in microseconds, so there is no comparison. Which, in a way is actually nice in that I'm not really competing with them. Or, maybe I am but I can still make some money. haha.
I'm in the medium frequency trading world. (calculate lots of market conditions, send at most a few orders a minute) I'd like to understand the HFT world a little better.
Is it basically "do what MFT does, but faster", or is there any specific advantage like getting into an order queue with priority?
There’s a very nice book about this subject called “Dark Pools”. I found it very enjoyable, you certainly seem to have the knowledge to enjoy it even more :-)
"This aspect, the platform itself, seems to be often overlooked in most discussions. Many conversations revolve around strategies (mean reversion, trend following, linear regression, etc.), and backtesting, without fully addressing the practical mechanics or logistics of strategy implementation, particularly in the context of live, intraday trading."
I'm glad you had fun, OP, but also I think I can shed some light on why most people discuss strategy.
Trading is a perfect storm of ridiculously high tech, ridiculously complicated, ridiculously regulated (Not over-regulated, mind you, this isn't a value judgement. But the amount of regulation is extremely high.), and ridiculously competitive.
But that said, it's the last bit that drives it all. Since it's so competitive, even though building an order entry system, and a risk system, and a position-tracking system, etc is a huge accomplishment (again, congrats OP!), it's table-stakes to even dip your toes in the pool here. Trading shops can attract top talent and robust, bespoke trading systems are basically cost of entry.
So people talk about strategy because everyone already has the table-stakes stuff and are now trying to make money with it.
It doesn't help, too, that lots of market participants aren't even playing the same game. In HFT, we operated on trades with alphas that lasted a few seconds, where races to entry/exit were battled in shaving nanoseconds off FPGAs being able to shoot out orders and microseconds off wireless networks flying market data around new jersey. Meanwhile, banks are more concerned with elections and geopolitics than they are about the weather in Carteret. (Rain = no microwave network for the day). And then there's a million strategies in the middle with alphas that last from hours to weeks.
So it makes it really hard to even speak the same language to each other when talking in common forums.
It's a fun world. I miss it sometimes.