avatarAndrej Karpathy

Free AI web copilot to create summaries, insights and extended knowledge, download it at here

6469

Abstract

market dynamics. This is where feature engineering becomes invaluable. By transforming and synthesizing the raw data using established technical indicators, we can provide the model with enriched insights that could explain underlying market patterns and trends. Such enriched data can significantly improve the model’s ability to anticipate future price movements.</p><p id="07ea">For this forecast, we will utilize a couple of technical indicators for our feature engineering like RSI, MACD, Bollinger Bands, Parabolic SAR, and Stochastic Oscillator. Additionally, we introduce lag features to capture temporal dependencies, ensuring our model benefits from both current and historical contexts.</p><p id="f61d"><b>Let’s calculate the features we will utilize in this forecast</b></p><div id="d4ed"><pre><span class="hljs-comment"># Compute RSI</span> df[<span class="hljs-string">'momentum_rsi'</span>] = RSIIndicator(close=df[<span class="hljs-string">'Close'</span>]).rsi()

<span class="hljs-comment"># Compute MACD</span> macd = MACD(close=df[<span class="hljs-string">'Close'</span>]) df[<span class="hljs-string">'trend_macd'</span>] = macd.macd() df[<span class="hljs-string">'trend_macd_signal'</span>] = macd.macd_signal() df[<span class="hljs-string">'trend_macd_diff'</span>] = macd.macd_diff()

<span class="hljs-comment"># Compute Bollinger Bands</span> bollinger = BollingerBands(close=df[<span class="hljs-string">'Close'</span>]) df[<span class="hljs-string">'volatility_bbm'</span>] = bollinger.bollinger_mavg() df[<span class="hljs-string">'volatility_bbl'</span>] = bollinger.bollinger_lband() df[<span class="hljs-string">'volatility_bbh'</span>] = bollinger.bollinger_hband()

<span class="hljs-comment"># Compute Parabolic SAR</span> psar = PSARIndicator(high=df[<span class="hljs-string">'High'</span>], low=df[<span class="hljs-string">'Low'</span>], close=df[<span class="hljs-string">'Close'</span>]) <span class="hljs-comment"># Assuming you have 'High' and 'Low' columns in your df</span> df[<span class="hljs-string">'trend_psar'</span>] = psar.psar()

<span class="hljs-comment"># Compute Stochastic Oscillator</span> stochastic = StochasticOscillator(high=df[<span class="hljs-string">'High'</span>], low=df[<span class="hljs-string">'Low'</span>], close=df[<span class="hljs-string">'Close'</span>]) <span class="hljs-comment"># Assuming you have 'High' and 'Low' columns</span> df[<span class="hljs-string">'momentum_stoch'</span>] = stochastic.stoch() df[<span class="hljs-string">'momentum_stoch_signal'</span>] = stochastic.stoch_signal()

<span class="hljs-comment"># Create Lag Features</span> df[<span class="hljs-string">'Close_Lag1'</span>] = df[<span class="hljs-string">'Close'</span>].shift(<span class="hljs-number">1</span>)

<span class="hljs-comment"># Drop NaN values introduced due to lag features and indicators</span> df = df.dropna()

<span class="hljs-comment"># Define features and target</span> X = df[[<span class="hljs-string">'momentum_rsi'</span>, <span class="hljs-string">'trend_macd'</span>, <span class="hljs-string">'trend_macd_signal'</span>, <span class="hljs-string">'trend_macd_diff'</span>, <span class="hljs-string">'volatility_bbm'</span>, <span class="hljs-string">'volatility_bbl'</span>, <span class="hljs-string">'volatility_bbh'</span>, <span class="hljs-string">'trend_psar'</span>, <span class="hljs-string">'momentum_stoch'</span>, <span class="hljs-string">'momentum_stoch_signal'</span>, <span class="hljs-string">'Close_Lag1'</span>]] y = df[<span class="hljs-string">'Close'</span>]</pre></div><p id="712f">The above code is organizing the dataset <code>df</code> into input features and a target variable for our model. The input features, captured under <code>X</code>, consist of various the features we calculated on and previously defined. The target variable, denoted by <code>y</code>, is the <code>Close</code> column, representing the daily closing price of EUR/USD, which our model aims to predict based on the provided features.</p><p id="bdfb"><b>Model Initialization and Training</b></p><div id="a7d2"><pre><span class="hljs-comment"># Initialize the model</span> model = xgb.XGBRegressor( learning_rate=<span class="hljs-number">0.75</span>, n_estimators=<span class="hljs-number">200</span>, max_depth=<span class="hljs-number">5</span>, subsample=<span class="hljs-number">0.9</span>, colsample_bytree=<span class="hljs-number">0.8</span>, colsample_bylevel=<span class="hljs-number">0.8</span>, gamma=<span class="hljs-number">0</span>, min_child_weight=<span class="hljs-number">1</span> )

<span class="hljs-comment"># Train the model</span> model.fit(X_train, y_train)</pre></div><p id="7ce9">Continuing from the previously discussed data preparation, this section of code dives into the model initialization and training phases using XGBoost. The <code>xgb.XGBRegressor()</code> initializes a regression model with specified hyperparameters to optimize the forecast. Key parameters include a learning rate of <code>0.75</code>, which determines the step size at each iteration while optimizing, <code>200</code> estimators or trees, and a maximum depth of <code>5</code> for each tree, among others. These hyperparameters play a role in controlling the model’s complexity and fit to the data.</p><p id="4e91">After initializing, the model is trained on the <code>X_train</code> and <code>y_train</code> datasets using the <code>fit</code> method. This step allows the model to learn the underlying patterns from the training data, preparing it to make future predictions on unseen data.</p><p id="95cf"><b>Performance Evaluation and Testing</b></p><div id="a823"><pre><span class="hljs-comment"># Predict on the test set</span> y_pred = model.predict(X_test)

<span class="hljs-comment"># Calculate performance metrics</span> mae = mean_absolute_error(y_test, y_pred) mse = mean_squared_error(y_test, y_pred) rmse = np.sqrt(mse)

<span class="hljs-built_in">print</span>(<span class="hljs-string">f"Mean Absolute Error: <span class="hljs-subst">{mae}</span>"</span>) <span class="hljs-built_in">print</span>(<span class="hljs-string">f"Mean Squared Error: <span class="hljs-subst">{mse}</span>"</span>) <span class="hljs-built_in">print</span>(<span class="hljs-string">f"Root Mean Squared Error: <span class="hljs-subst">{rmse}</span>"</span>)

y_train_pred = model.predict(X_train)</pre></div><p id="8c56">After training the

Options

model on the historical data we evaluate its performance on unseen or test data. Using the <code>predict</code> method of the trained model, predictions (<code>y_pred</code>) are generated for the test dataset <code>X_test</code>. Subsequently, to assess the accuracy and reliability of these predictions, various performance metrics are computed:</p><ul><li><b>The Mean Absolute Error (MAE)</b> provides an average magnitude of errors between predicted and actual values.</li><li><b>The Mean Squared Error (MSE) </b>squares these errors to emphasize larger discrepancies.</li><li><b>Root Mean Squared Error (RMSE) </b>is the square root of MSE, providing error in the same units as the original data.</li></ul><p id="c92b">These metrics are then printed for clear visibility. We concludes by also predicting on the training set (<code>X_train</code>) with <code>y_train_pred</code>, to further analyze and compare the model’s performance on both training and test datasets.</p><p id="44c0">The following output displays the performance metricswhich assess the accuracy of our model’s predictions:</p><div id="6743"><pre><span class="hljs-attribute">Mean</span> Absolute Error: <span class="hljs-number">0</span>.<span class="hljs-number">009141215039947168</span> <span class="hljs-attribute">Mean</span> Squared Error: <span class="hljs-number">0</span>.<span class="hljs-number">000303615460154008</span> <span class="hljs-attribute">Root</span> Mean Squared Error: <span class="hljs-number">0</span>.<span class="hljs-number">017424564848340058</span></pre></div><ul><li><b>Mean Absolute Error (MAE): </b>At 0.0091, it shows the model’s average absolute deviation from the actual values.</li><li><b>Mean Squared Error (MSE):</b> With a value of 0.0003036, it indicates the average squared error, emphasizing larger mistakes.</li><li><b>Root Mean Squared Error (RMSE):</b> At 0.0174, it provides the average error in the original unit, illustrating the typical magnitude of error.</li></ul><p id="7500">The relatively low values across these metrics suggest that the model has a good degree of accuracy in its predictions. The model appears to be reliably forecasting the target variable, depicted with minimal deviations in the forecasted data when compared to the actual data.</p><p id="be3b"><b>Data Visualization</b></p><div id="2b49"><pre><span class="hljs-comment"># Create a new DataFrame for visualization</span> viz_df = pd.DataFrame({<span class="hljs-string">'True'</span>: y_test, <span class="hljs-string">'Predicted'</span>: y_pred})

<span class="hljs-comment"># Concatenate the training data for a complete view</span> viz_df_train = pd.DataFrame({<span class="hljs-string">'True'</span>: y_train, <span class="hljs-string">'Predicted'</span>: y_train_pred}) viz_df = pd.concat([viz_df_train, viz_df])

<span class="hljs-comment"># Plot the results</span> plt.figure(figsize=(<span class="hljs-number">14</span>, <span class="hljs-number">7</span>)) plt.plot(viz_df[<span class="hljs-string">'True'</span>], label=<span class="hljs-string">'True'</span>, color=<span class="hljs-string">'blue'</span>) plt.plot(viz_df[<span class="hljs-string">'Predicted'</span>], label=<span class="hljs-string">'Predicted'</span>, color=<span class="hljs-string">'red'</span>, alpha=<span class="hljs-number">0.7</span>) plt.title(<span class="hljs-string">'EUR/USD Forecast: True vs Predicted'</span>) plt.legend() plt.grid(<span class="hljs-literal">True</span>) plt.show()</pre></div><figure id="ce5c"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*cLlTTCFRMZqUBQIUBeya0g.png"><figcaption></figcaption></figure><p id="62e5">The visual representation of the EUR/USD currency pair’s forecasted versus actual values offers an insightful glimpse into the model’s capabilities. The close alignment between the blue <code>True</code> line and the red <code>Predicted</code> line for most of the chart affirms the model’s strong predictive proficiency, especially given the low Mean Absolute Error (MAE) of 0.0091. The few areas where deviations occur resonate with the Root Mean Squared Error (RMSE) of 0.0174, indicating the average magnitude of error.</p><p id="ba85">Notably, the small segment towards the right end, where predictions seem to diverge slightly, underscores the challenges of exact currency forecasting. Nevertheless, the model, as depicted in the graph and corroborated by the performance metrics, has shown remarkable accuracy in capturing the nuances of the EUR/USD exchange rate’s movements.</p><h1 id="fe27">Conclusion</h1><p id="3036">In conclusion, this exploration into Forex forecasting has underscored the critical interplay between data preprocessing, feature engineering, and model selection. Through this model we found that XGBoost in predicting the EUR/USD currency pair stands out, demonstrating the algorithm’s robustness and adaptability. Finally, the precision showcased by our model reinforces XGBoost’s reputation as an efficient tool to forecast financial data.</p><p id="0141">Read more of my stories here:</p><div id="c60c" class="link-block"> <a href="https://algocraft.xyz/eur-usd-forecasting-simplified-an-lstm-users-guide-337ccdda6158"> <div> <div> <h2>EUR/USD Forecasting Simplified: an LSTM User’s Guide</h2> <div><h3>LSTM, or Long Short-Term Memory, is a specialized type of Recurrent Neural Network (RNN) designed to recognize patterns…</h3></div> <div><p>algocraft.xyz</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*EUIE-cUkt3x2AqBX5nUotA.jpeg)"></div> </div> </div> </a> </div><div id="a6a6" class="link-block"> <a href="https://algocraft.xyz/how-to-get-131-return-with-mean-reversion-trading-strategy-from-stock-selection-to-backtesting-c623870adf31"> <div> <div> <h2>How to Get a 131% Return with Mean Reversion Trading Strategy: From Stock Selection to Backtesting</h2> <div><h3>undefined</h3></div> <div><p>undefined</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/0*o95y-D4ETf1Geqx3)"></div> </div> </div> </a> </div></article></body>

ICLR 2017 vs arxiv-sanity

I thought it would be fun to cross-reference the ICLR 2017 (a popular Deep Learning conference) decisions (which fall into 4 categories: oral, poster, workshop, reject) with the number of times each paper was added to someone’s library on arxiv-sanity. ICLR 2017 decision making involves a number of area chairs and reviewers that decide the fate of each paper over a period of few months, while arxiv-sanity involves one person working 2 hours once a month (me), and a number of people who use it to tame the flood of papers out there. It is a battle between top down and bottom up. Lets see what happens.

Here are the decisions for ICLR 2017. A total of 491 papers were submitted, of which 15 (3%)will be an oral, 183 (37.3%) a poster, 48 (9.8%)were suggested for workshop and 245 (49.9%) were rejected. The accepted papers will be presented at ICLR on April 24–27 in Toulon, which I am really looking forward to. Look how amazing it looks:

Toulon, France. I think.

But I digress.

On the other hand we have arxiv-sanity, which has a library feature. In short, any registered user can add a paper to their library, and arxiv-sanity will train a personalized SVM on bigram tfidf features of the full text of all papers to make content-based recommendations to the user. For example, I have a number of RL/generative models/CV papers in my library and whenever there is a new paper on these topics it will come up on top in my “recommended” tab. The review pool of arxiv-sanity is as of now a total of 3195 users — this is the number of people with an account that have at least one paper in the library. Together, these users have so far included 55,671 papers into their libraries, i.e. an average of 17.4 papers.

An important feature of arxiv-sanity is that users don’t just upvote papers with no repercussions. Adding a paper to your library has some weight, because that paper will influence your recommendations. You have an incentive to only include things that really matter to you in there. It’s clever right? No? Okay fine.

The experiment

Long story short, I loop over all papers in ICLR and try to find them on arxiv using an exact match on the title. Some ICLR papers are not on arxiv, and some won’t get matched because the authors renamed them, or they contain weird characters, etc.

For example, lets look at the papers that got an oral at ICLR 2017. We get:

for oral, found 10/15 papers on arxiv with library counts:
 64 Reinforcement Learning with Unsupervised Auxiliary Tasks
 44 Neural Architecture Search with Reinforcement Learning
 38 Understanding deep learning requires rethinking generalizatio...
 28 Towards Principled Methods for Training Generative Adversaria...
 22 Learning End-to-End Goal-Oriented Dialog
 19 Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy C...
 13 Learning to Act by Predicting the Future
 12 Amortised MAP Inference for Image Super-resolution
  8 Multi-Agent Cooperation and the Emergence of (Natural) Langua...
  8 End-to-end Optimized Image Compression

Here we see that we matched 10 out of 15 oral papers on arxiv, and the number next to each one is the number of people who have added that paper to their library. E.g. “Reinforcement Learning with Unsupervised Auxiliary Tasks” was in a library of 64 arxiv-sanity users. I also had to truncate some paper names because medium.com is improperly conceived and doesn’t let you change the font size.

Now lets look at the posters:

for poster, found 113/183 papers on arxiv with library counts:
149 Adversarial Feature Learning
147 Hierarchical Multiscale Recurrent Neural Networks
140 Recurrent Batch Normalization
 80 HyperNetworks
 79 FractalNet: Ultra-Deep Neural Networks without Residuals
 73 Zoneout: Regularizing RNNs by Randomly Preserving Hidden Acti...
 62 Unrolled Generative Adversarial Networks
 52 Adversarially Learned Inference
 49 Quasi-Recurrent Neural Networks
 48 Do Deep Convolutional Nets Really Need to be Deep and Convolu...
 46 Neural Photo Editing with Introspective Adversarial Networks
 43 An Actor-Critic Algorithm for Sequence Prediction
 41 A Learned Representation For Artistic Style
 37 Structured Attention Networks
 33 Mollifying Networks
 30 DeepCoder: Learning to Write Programs
 28 SGDR: Stochastic Gradient Descent with Warm Restarts
 27 Learning to Navigate in Complex Environments
 27 Generative Multi-Adversarial Networks
 26 Soft Weight-Sharing for Neural Network Compression
 25 Pruning Filters for Efficient ConvNets
 24 Why Deep Neural Networks for Function Approximation?
 24 Mode Regularized Generative Adversarial Networks
 24 Dialogue Learning With Human-in-the-Loop
 24 Designing Neural Network Architectures using Reinforcement Le...
 23 PGQ: Combining policy gradient and Q-learning
 22 Frustratingly Short Attention Spans in Neural Language Modeli...
 21 Tracking the World State with Recurrent Entity Networks
 21 Deep Probabilistic Programming
 20 Density estimation using Real NVP
 20 Adversarial Training Methods for Semi-Supervised Text Classif...
 19 Semi-Supervised Classification with Graph Convolutional Netwo...
 19 PixelVAE: A Latent Variable Model for Natural Images
 19 Learning to Optimize
 19 Learning a Natural Language Interface with Neural Programmer
 19 Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
 19 Dynamic Coattention Networks For Question Answering
 18 PixelCNN++: Improving the PixelCNN with Discretized Logistic ...
 18 Generalizing Skills with Semi-Supervised Reinforcement Learni...
 18 Deep Learning with Dynamic Computation Graphs
 18 Automatic Rule Extraction from Long Short Term Memory Network...
 18 Adversarial Machine Learning at Scale
 17 Learning through Dialogue Interactions by Asking Questions
 16 Learning to Perform Physics Experiments via Deep Reinforcemen...
 16 Categorical Reparameterization with Gumbel-Softmax
 15 Sample Efficient Actor-Critic with Experience Replay
 14 Variational Lossy Autoencoder
 14 Identity Matters in Deep Learning
 14 Bidirectional Attention Flow for Machine Comprehension
 13 Towards a Neural Statistician
 13 Recurrent Mixture Density Network for Spatiotemporal Visual A...
 13 On Detecting Adversarial Perturbations
 12 Trained Ternary Quantization
 12 Improving Policy Gradient by Exploring Under-appreciated Rewa...
 12 Capacity and Trainability in Recurrent Neural Networks
 11 SampleRNN: An Unconditional End-to-End Neural Audio Generatio...
 11 Machine Comprehension Using Match-LSTM and Answer Pointer
 11 Latent Sequence Decompositions
 11 Calibrating Energy-based Generative Adversarial Networks
 10 Unsupervised Cross-Domain Image Generation
 10 Learning to Remember Rare Events
 10 Highway and Residual Networks learn Unrolled Iterative Estima...
  9 TopicRNN: A Recurrent Neural Network with Long-Range Semantic...
  9 Steerable CNNs
  9 Query-Reduction Networks for Question Answering
  9 Lossy Image Compression with Compressive Autoencoders
  9 Learning to Compose Words into Sentences with Reinforcement L...
  8 Stick-Breaking Variational Autoencoders
  8 Deep Variational Information Bottleneck
  8 Batch Policy Gradient Methods for Improving Neural Conversati...
  7 Discrete Variational Autoencoders
  7 Data Noising as Smoothing in Neural Network Language Models
  6 Variable Computation in Recurrent Neural Networks
  6 Sigma Delta Quantized Networks
  6 Dropout with Expectation-linear Regularization
  6 Delving into Transferable Adversarial Examples and Black-box ...
  6 A Compositional Object-Based Approach to Learning Physical Dy...
  5 Towards the Limit of Network Quantization
  5 Tighter bounds lead to improved classifiers
  5 Pointer Sentinel Mixture Models
  5 On the Quantitative Analysis of Decoder-Based Generative Mode...
  5 Neuro-Symbolic Program Synthesis
  5 Lie-Access Neural Turing Machines
  5 Learning to superoptimize programs
  5 Learning Features of Music From Scratch
  5 Improving Neural Language Models with a Continuous Cache
  5 Deep Biaffine Attention for Neural Dependency Parsing
  4 Temporal Ensembling for Semi-Supervised Learning
  4 Diet Networks: Thin Parameters for Fat Genomics
  4 DeepDSL: A Compilation-based Domain-Specific Language for Dee...
  4 DSD: Dense-Sparse-Dense Training for Deep Neural Networks
  4 A recurrent neural network without chaos
  3 Trusting SVM for Piecewise Linear CNNs
  3 The Neural Noisy Channel
  3 Revisiting Classifier Two-Sample Tests
  3 Regularizing CNNs with Locally Constrained Decorrelations
  3 Optimal Binary Autoencoding with Pairwise Correlations
  3 Loss-aware Binarization of Deep Networks
  3 Learning Recurrent Representations for Hierarchical Behavior ...
  3 EPOpt: Learning Robust Neural Network Policies Using Model En...
  3 Deep Information Propagation
  2 Words or Characters? Fine-grained Gating for Reading Comprehe...
  2 Topology and Geometry of Half-Rectified Network Optimization
  2 Maximum Entropy Flow Networks
  2 Incorporating long-range consistency in CNN-based texture gen...
  2 Hadamard Product for Low-rank Bilinear Pooling
  1 Multi-view Recurrent Neural Acoustic Word Embeddings
  1 Inductive Bias of Deep Convolutional Networks through Pooling...
  1 Geometry of Polysemy
  1 Autoencoding Variational Inference For Topic Models
  1 A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING
  0 Deep Multi-task Representation Learning: A Tensor Factorisati...
  0 A Compare-Aggregate Model for Matching Text Sequences

Some got a lot of love (149!), and some very little (0). For workshop suggestions we get:

for workshop, found 23/48 papers on arxiv with library counts:
 60 Adversarial examples in the physical world
 31 Learning in Implicit Generative Models
 16 Surprise-Based Intrinsic Motivation for Deep Reinforcement Le...
 14 Multiplicative LSTM for sequence modelling
 13 Efficient Softmax Approximation for GPUs
 12 RenderGAN: Generating Realistic Labeled Data
 12 Generalizable Features From Unsupervised Learning
 10 Programming With a Differentiable Forth Interpreter
  8 Gated Multimodal Units for Information Fusion
  8 Deep Learning with Sets and Point Clouds
  7 Unsupervised Perceptual Rewards for Imitation Learning
  5 Song From PI: A Musically Plausible Network for Pop Music Gen...
  5 Modular Multitask Reinforcement Learning with Policy Sketches
  5 A Differentiable Physics Engine for Deep Learning in Robotics
  4 Exponential Machines
  4 Dataset Augmentation in Feature Space
  3 Semi-supervised deep learning by metric embedding
  2 Adaptive Feature Abstraction for Translating Video to Languag...
  1 Modularized Morphing of Neural Networks
  1 Learning Continuous Semantic Representations of Symbolic Expr...
  1 Extrapolation and learning equations
  0 Online Structure Learning for Sum-Product Networks with Gauss...
  0 Bit-Pragmatic Deep Neural Network Computing

and I won’t list all 200-something papers that were rejected, but lets look at the few that arxiv-sanity users really liked, but the ICLR ACs and reviewers did not:

for reject, found 58/245 papers on arxiv with library counts:
 46 The Predictron: End-To-End Learning and Planning
 39 RL^2: Fast Reinforcement Learning via Slow Reinforcement Lear...
 35 Understanding intermediate layers using linear classifier pro...
 33 Hierarchical Memory Networks
 31 An Analysis of Deep Neural Network Models for Practical Appli...
 20 Low-rank passthrough neural networks
 19 Higher Order Recurrent Neural Networks
 18 Adding Gradient Noise Improves Learning for Very Deep Network...
 16 Unsupervised Pretraining for Sequence to Sequence Learning
 16 A Joint Many-Task Model: Growing a Neural Network for Multipl...
 15 Adversarial examples for generative models
 14 Gated-Attention Readers for Text Comprehension
 13 Extensions and Limitations of the Neural GPU
 12 Warped Convolutions: Efficient Invariance to Spatial Transfor...
 11 Neural Combinatorial Optimization with Reinforcement Learning
 11 Memory-augmented Attention Modelling for Videos
 10 GRAM: Graph-based Attention Model for Healthcare Representati...
  9 Wav2Letter: an End-to-End ConvNet-based Speech Recognition Sy...
  9 Understanding trained CNNs by indexing neuron selectivity
  9 The Power of Sparsity in Convolutional Neural Networks
  9 Improving Stochastic Gradient Descent with Feedback
  8 Towards Information-Seeking Agents
  8 NEWSQA: A MACHINE COMPREHENSION DATASET
  8 LipNet: End-to-End Sentence-level Lipreading
  7 Generative Adversarial Parallelization
  7 Efficient Summarization with Read-Again and Copy Mechanism
  6 Multi-task learning with deep model based reinforcement learn...
  6 Multi-modal Variational Encoder-Decoders
  6 End-to-End Answer Chunk Extraction and Ranking for Reading Co...
  6 Boosting Image Captioning with Attributes
  6 Beyond Fine Tuning: A Modular Approach to Learning on Small D...
  5 Structured Sequence Modeling with Graph Convolutional Recurre...
  5 Human perception in computer vision
  5 Cooperative Training of Descriptor and Generator Networks

Here is the full version, which was not truncated to fit here. There are a few papers on the top of this list that were possibly unfairly rejected.

Here’s another question — what would ICLR 2017 look like if it were simply voted on by the crowd of arxiv-sanity users (of the papers we can find on arxiv)? Here is an excerpt:

oral:
149 Adversarial Feature Learning
147 Hierarchical Multiscale Recurrent Neural Networks
140 Recurrent Batch Normalization
 80 HyperNetworks
 79 FractalNet: Ultra-Deep Neural Networks without Residuals
 73 Zoneout: Regularizing RNNs by Randomly Preserving Hidden Acti...
 64 Reinforcement Learning with Unsupervised Auxiliary Tasks
 62 Unrolled Generative Adversarial Networks
 60 Adversarial examples in the physical world
 52 Adversarially Learned Inference
-------------------------------------------------
poster:
 49 Quasi-Recurrent Neural Networks
 48 Do Deep Convolutional Nets Really Need to be Deep and Convolu...
 46 The Predictron: End-To-End Learning and Planning
 46 Neural Photo Editing with Introspective Adversarial Networks
 44 Neural Architecture Search with Reinforcement Learning
 43 An Actor-Critic Algorithm for Sequence Prediction
 41 A Learned Representation For Artistic Style
 39 RL^2: Fast Reinforcement Learning via Slow Reinforcement Lear...
 38 Understanding deep learning requires rethinking generalizatio...
 37 Structured Attention Networks
 35 Understanding intermediate layers using linear classifier pro...
 33 Mollifying Networks
 33 Hierarchical Memory Networks
 31 Learning in Implicit Generative Models
 31 An Analysis of Deep Neural Network Models for Practical Appli...
 30 DeepCoder: Learning to Write Programs
...

Again, the full listing can be found here. Note that in particular, some ICLR2017 papers that were rejected would have been almost an oral based on arxiv-sanity users alone, especially the Predictron, RL², “Understanding intermediate layers”, and “Hierarchical Memory Networks”. Conversely, some accepted papers had very little love from arxiv-sanity users. Here is a full confusion matrix:

And here is the confusion matrix in text, for each cell, together with the paper titles. This doesn’t look too bad. The two groups don’t agree on the orals at all, agree on the posters quite a bit, and most importantly there are very few confusions between oral/poster and rejection. Also, congratulations to Max et al. for “Reinforcement Learning with Unsupervised Auxiliary Tasks”, which is the only paper that both groups agree should be an oral :)

Finally, I read the following Medium post a few days ago: “Ten Deserving Deep Learning Papers that were Rejected at ICLR 2017”, by Carlos E. Perez. It seems that arxiv-sanity users agree with this post, and all papers listed there (including LipNet)(that we could also find on arxiv) would have been accepted by arxiv-sanity users.

Discussion

An asterisk. There are several factors that skew these results. For example, the size of arxiv-sanity user base grows over time, so these results likely slightly favor papers that were published on arxiv later than earlier, as these would have come to more user’s attention as new papers on the site. Also, papers are not seen with equal frequencies — for instance if some paper gets tweeted out by someone popular, more people will see it, and more people might add it to their library. And finally, a good argument could be made that on arxiv-sanity “rich get richer”, because arxiv papers are not anonymous and celebrities could get more attention. In this particular case, ICLR 2017 is single-blind so this is not a differentiating factor.

Overall, my own conclusion from this experiment is that there is quite a bit of signal here. And we’re getting it “for free” from a bottom up process on the internet, instead of something that takes a few hundred people several months. And as someone who has had a good amount of long, painful, stressful, rebuttals back and forth on both submitting/reviewing sides that dragged on for multiple weeks/months, I say: Maybe we don’t need it. Or at the very least maybe there is a lot of room for improvement.

EDIT1: someone suggested the fun idea that we add up the number of citations of these papers in ICLR 2018 submitted/accepted papers, and see which ranking “wins” on that metric. Looking forward to that :)

Machine Learning
Artificial Intelligence
Academia
Peer Review
Recommended from ReadMedium