Reading Log
A reading log for the most interesting articles or academic papers that I’ve recently read to help me remember what it was that I initially found interesting.
Artificial Intelligence
Machine Learning
Economics
Mathematics
Misc
Artificial Intelligence
2019-06-13 - Text-based Editing of Talking-head Video
2019-06-06 - Speech2Face: Learning the Face Behind a Voice
2019-06-02 - How to do Research At the MIT AI Lab
2019-07-31 - A Critique of Pure Learning: What Artifical Neural Networks can Learn from Animal Brains - Anthony M. Zador
“In this view, supervised learning in ANNs should not be viewed as the analog of learning in animals. Instead, since most of the data that contribute an animal’s fitness are encoded by evolution into the genome, it would perhaps be just as accurate (or inaccurate) to call rename it “supervised evolution.” Such a renaming would emphasize that “supervised learning” in ANNs is really recapitulating the extraction of statistical regularities that occurs in animals by both evolution and learning.”
This was really interesting. I don’t know that much about biology, so I’ll have to re-read it sometime soon to fully understand everything.
Text-based Editing of Talking-head Video - Fried, Ohad, et al.
“We acknowledge that bad actors might use such technologies to falsify personal statements and slander prominent individuals. We are concerned about such deception and misuse.”
Speech2Face: Learning the Face Behind a Voice - Oh, Tae-Hyun, et al.
Wow.
How to do Research At the MIT AI Lab - a bunch of people.
Machine Learning
2019-08-09 - A Recipe for Training Neural Networks
2019-06-21 - Searching for Activation Functions
2019-06-16 - An Empirical Exploration of Recurrent Network Architectures
2019-05-01 - Random Search for Hyper-Parameter Optimization
A Recipe for Training Neural Networks - Andrej Karpathy
“The approach I like to take to finding a good model has two stages: first get a model large enough that it can overfit (i.e. focus on training loss) and then regularize it appropriately (give up some training loss to improve the validation loss). The reason I like these two stages is that if we are not able to reach a low error rate with any model at all that may again indicate some issues, bugs, or misconfiguration.”
Interesting. Different from how I approach training neural nets, but I’m going to try this and see how this process could affect the results I’m getting now.
Searching for Activation Functions - Ramachandran, Prajit, Barret Zoph, and Quoc V. Le.
“We then empirically validated the best discovered activation function, which we call Swish and is defined as f(x) = x · sigmoid(βx). Our experiments used models and hyperparameters that were designed for ReLU and just replaced the ReLU activation function with Swish; even this simple, suboptimal procedure resulted in Swish consistently outperforming ReLU and other activation functions … The simplicity of Swish and its similarity to ReLU means that replacing ReLUs in any network is just a simple one line code change.”
An Empirical Exploration of Recurrrent Network Architectures - Jozefowicz Rafal, Wojciech Zaremba, and Ilya Sutskever
“Importantly, adding a bias of size 1 significantly improved the performance of the LSTM on tasks where it fell behind the GRU and MUT1. Thus we recommend adding a bias of 1 to the forget gate of every LSTM in every application; it is easy to do often results in better performance on our tasks. This adjustment is the simple improvement over the LSTM that we set out to discover.”
Random Search for Hyper-Parameter Optimization - Bengio Yoshua, and Bergstra James
“We have shown that random experiments are more efficient than grid experiments for hyper-parameter optimization in the case of several learning algorithms on several data sets. Our analysis of the hyper-parameter response surface (Ψ) suggests that random experiments are more efficient because not all hyperparameters are equally important to tune.”
Economics
2019-01-05 - Forecasting at Scale
2016-10-04 - Thomas Paine: From Pirate to Revolutionary
2016-09-30 - Experimental Methods in Economics
2016-09-27 - The Responsibility of Being Free
2016-09-26 - The Accidental Theorist
Thomas Paine: From Pirate to Revolutionary - Jeff Riggenbach
“These are the times that try men’s souls. The summer soldier and the sunshine patriot will, in this crisis, shrink from the service of their country; but he that stands by it now, deserves the love and thanks of man and woman. Tyranny, like hell, is not easily conquered; yet we have this consolation with us, that the harder the conflict, the more glorious the triumph.”
I really love reading about Thomas Paine.
Experimental Methods in Economics - Vernon L. Smith
“In fact any mathematical optimization problem whose implementation requires measures of incremental cost and value, is a candidate for automation in an integrated person-computed market decision making system.”
This was the paper (part of Dr. Isaac’s Experimental Economics class my senior year of undergrad) that made me want to be an experimental economist.
The Responsibility of Being Free - Tom G. Palmer
“Responsibility is not a burden we must bear to be free; the awareness that “I did that” is what makes freedom a prize worth fighting for. Responsibility is the key to the realization of freedom.”
The Accidental Theorist - Paul Krugman
“Nobody, it seems, warned Greider that he needed to worry about fallacies of composition, that the logic of the economy as a whole is not the same as the logic of a single market.”
Mathematics
2019-09-09 - An Interactive Introduction to Fourier Transforms
An Interactive Introduction to Fourier Transforms - Jez Swanson
Fourier transforms are things that let us take something and split it up into its frequencies
The frequencies tell us about some fundamental properties of the data we have
They can compress data by only storing the important frequencies
And we can also use them to make cool looking animations with a bunch of circles
Misc
2023-11-25 - Snow Crash by Neal Stephenson
2020-11-06 - A Forgotten Election Day Massacre in Florida that Still Haunts Today
2019-10-15 - Write a List of Your Values and Everything Else Will Follow
2019-09-11 - The Communist Plot to Assassinate George Orwell
2019-08-16 - Beating the Averages
2019-08-01 - Inside Tokyo’s audiophile venues
2019-06-17 - Busy Person Patterns
2019-02-25 - How to remember what you read: What to do before, during, and after reading anything
Snow Crash - Neal Stephenson
This was one of my favorite books this year. Very fast-paced, exciting, and the worldbuilding establishes an incredible canvas for the characters’ narratives. I was thrilled with the development of the protagonist of the story, titularly named Hiro Protagonist, by its conclusion. A particularly striking aspect of Snow Crash was the intersection of religion and computer science, likening the spread of ideas to viruses in the human mind. This concept, explored through the metaphor of language as a virus, resonated heavily with me, especially in relation to the ancient Sumerian myth of Enki and the Tower of Babel. The idea that learning a language can reprogram the brain, a concept I remember learning about in Behavioral Economics with Dr. Hammond at FSU, was both informative and thought-provoking.
A Forgotten Election Day Massacre in Florida that Still Haunts Today - Sean Braswell
The story of Ocoee serves as a stark reminder of America’s troubled past, but also as a warning of its far from perfect present, especially in a state that is still battling voter suppression efforts today. Democracy must constantly be fought for, and sometimes against men with guns. And what makes what those like July Perry and Mose Norman did so audacious, says Ortiz, isn’t the actions they took in the face of danger and violence. “It’s the idea that there are things more powerful than guns. There are ideas, there are beliefs in equality. That’s what animat[ed] this movement in Ocoee and that’s what we have to honor today.”
Write a List of Your Values and Everything Else Will Follow - Darius Foroux
Authenticity/Personal Integrity
Truthfulness
Responsibility
Fearlessness
The Communist Plot to Assassinate George Orwell - Duncan White
“The Communists had perhaps mistaken Orwell for another naive volunteer, there to be pushed around, but they had in fact made a powerful enemy, an enemy who now prepared to fight back with his trusted weapons, the typewriter and the pen.”
A true champion of his ideals. I respect Orwell even more after reading about some of the adversity he had to go through.
Beating the Averages - Paul Graham
“You can’t trust the opinions of the others, because of the Blub paradox: they’re satisfied with whatever language they happen to use, because it dictates the way they think about programs.”
Inside Tokyo’s audiphile venues - Aaron Coultate
“At Bridge, Ariizumi says he was taken in by the concept of the “Third Place,” a term used by American sociologist Ray Oldenburg loosely defined as somewhere for people to relax and mingle in between their time at work and home …. I wanted to have a space where people can get away from stress, from tiring days, to take it easy and relax. Those are ordinary, but important, things in our lives.”
Tokyo, in general, seems like such a great place. My next holiday I plan to go there.
Busy Person Patterns - James F. Kile, Donald J. Little, Samir Shah
How to remember what you read: What to do before, during, and after reading anything - Jory MacKay
Apply what you’ve read
Explain it to someone else
Visit and organize your notes