Deep Machine Learning Knowledge Exchange
Hello, this is winnerineast. I believe the better future is Human Being + Machine and I'm working on it in order to make it happen. Here is the inventory for all kinds of knowledges I collected from internet without any sign-in.
Gernal Machine Learning Topics
Machine Learning Blogby Brian McFee
How to become a Data Scientist in 6 months: A hacker’s approach to career planning
Interview Machine Learning Engineer Questions
Blog Post onDeep Learning, NLP, and Representations
Blog Post onNLP Tutorial
Natural Language Processing Blogby Hal Daumé III
Learning machine learning hints and clues
I think the best way for practice-focused methodology is something like'practice — learning — practice', that means where students first come with some existing projects with problems and solutions (practice) to get familiar with traditional methods in the area and perhaps also with their methodology. After practicing with some elementary experiences, they can go into the books and study the underlying theory, which serves to guide their future advanced practice and will enhance their toolbox of solving practical problems. Studying theory also further improves their understanding on the elementary experiences, and will help them acquire advanced experiences more quickly.
Kaggle knowledge competitions
GeneralMachine Learning Topics
Machine learning is fun
Machine learning: an in-depth, non-technical guide
Taught by Geoffrey Hinton, a pioneer in the field of neural networks
Enough Machine Learning to Make Hacker News Readable Again
Machine Learning courses in Universities
Stanford Natural Language ProcessingIntro NLP course with videos. This hasno deep learning. But it is a good primer for traditional nlp.
Michael Collins- one of the best NLP teachers. Check out the material on the courses he is teaching.
Intro to Natural Language Processingon Coursera by U of Michigan
Intro to Artificial Intelligencecourse on Udacity which also covers NLP
Deep Learning for Natural Language Processing (2015 classes)by Richard Socher
Deep Learning for Natural Language Processing (2016 classes)by Richard Socher. Updated to make use of Tensorflow. Note that there are some lectures missing (lecture 9, and lectures 12 onwards).
Natural Language Processing- course on Coursera that was only done in 2013. The videos are not available at the moment. Also Mike Collins is a great professor and his notes and lectures are very good.
Statistical Machine Translation- a Machine Translation course with great assignments and slides.
Udacity Deep LearningDeep Learning course on Udacity (using Tensorflow) which covers a section on using deep learning for NLP tasks (covering Word2Vec, RNN's and LSTMs).
NLTK with Python 3 for Natural Language Processingby Harrison Kinsley(sentdex). Good tutorials with NLTK code implementation.
Mikolovet al. 2013. Performs well on word similarity and analogy task. Expands on famous example: King – Man + Woman = Queen
General Machine Learning Topics
Memory networks are implemented inMemNN. Attempts to solve task of reason attention and memory.
Pre-trained word embeddings for WSJ corpusby Koc AI-Lab
HLBL language modelby Turian
Real-valued vector "embeddings"by Dhillon
Knwl.js- A Natural Language Processor in JS
Retext- Extensible system for analyzing and manipulating natural language
NLP Compromise- Natural Language processing in the browser
Natural- general natural language facilities for node
Pattern- A web mining module for the Python programming language. It has tools for natural language processing, machine learning, among others.
TextBlob- Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of NLTK and Pattern, and plays nicely with both.
YAlign- A sentence aligner, a friendly tool for extracting parallel sentences from comparable corpora.
jieba- Chinese Words Segmentation Utilities.
SnowNLP- A library for processing Chinese text.
KoNLPy- A Python package for Korean natural language processing.
Rosetta- Text processing tools and wrappers (e.g. Vowpal Wabbit)
BLLIP Parser- Python bindings for the BLLIP Natural Language Parser (also known as the Charniak-Johnson parser)
PyNLPl- Python Natural Language Processing Library. General purpose NLP library for Python. Also contains some specific modules for parsing common NLP formats, most notably forFoLiA, but also ARPA language models, Moses phrasetables, GIZA++ alignments.
python-ucto- Python binding to ucto (a unicode-aware rule-based tokenizer for various languages)
python-frog- Python binding to Frog, an NLP suite for Dutch. (pos tagging, lemmatisation, dependency parsing, NER)
colibri-core- Python binding to C++ library for extracting and working with with basic linguistic constructions such as n-grams and skipgrams in a quick and memory-efficient way.
spaCy- Industrial strength NLP with Python and Cython.
PyStanfordDependencies- Python interface for converting Penn Treebank trees to Stanford Dependencies.
MIT Information Extraction Toolkit- C, C++, and Python tools for named entity recognition and relation extraction
CRF++- Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing tasks.
CRFsuite- CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data.
BLLIP Parser- BLLIP Natural Language Parser (also known as the Charniak-Johnson parser)
colibri-core- C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams and skipgrams in a quick and memory-efficient way.
ucto- Unicode-aware regular-expression based tokenizer for various languages. Tool and C++ library. Supports FoLiA format.
frog- Memory-based NLP suite developed for Dutch: PoS tagger, lemmatiser, dependency parser, NER, shallow parser, morphological analyzer.
ReVerbWeb-Scale Open Information Extraction
OpenRegexAn efficient and flexible token-based regular expression language and engine.
CogcompNLP- Core libraries developed in the U of Illinois' Cognitive Computation Group.
Saul- Library for developing NLP systems, including built in modules like SRL, POS, etc.
Clojure-openNLP- Natural Language Processing in Clojure (opennlp)
Infections-clj- Rails-like inflection library for Clojure and ClojureScript
Wit-ai- Natural Language Interface for apps and devices.
Iris- Free text search API over large public document collections.
General Machine Learning Topics
Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning [arXiv]
Overcoming catastrophic forgetting in neural networks [arXiv]
A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs [arXiv]
Importance Sampling with Unequal Support [arXiv]
Quasi-Recurrent Neural Networks [arXiv]
Capacity and Learnability in Recurrent Neural Networks [OpenReview]
Unrolled Generative Adversarial Networks [OpenReview]
Deep Information Propagation [OpenReview]
Structured Attention Networks [OpenReview]
Incremental Sequence Learning [arXiv]
b-GAN: Unified Framework of Generative Adversarial Networks [OpenReview]
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks [OpenReview]
Categorical Reparameterization with Gumbel-Softmax [arXiv]
Image-to-Image Translation with Conditional Adversarial Networks [arXiv]
Lip Reading Sentences in the Wild [arXiv]
Rethinking the Inception Architecture for Computer Vision [arXiv]
Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks [arXiv]
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin [arXiv]
Learning to reinforcement learn [arXiv]
Learning to reinforcement learn [arXiv]
A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models [arXiv]
The Predictron: End-To-End Learning and Planning [OpenReview]
Generalizing Skills with Semi-Supervised Reinforcement Learning [OpenReview]
Sample Efficient Actor-Critic with Experience Replay [OpenReview]
Neural Architecture Search with Reinforcement Learning [OpenReview]
Towards Information-Seeking Agents [OpenReview]
Multi-Agent Cooperation and the Emergence of (Natural) Language [OpenReview]
Improving Policy Gradient by Exploring Under-appreciated Rewards [OpenReview]
Stochastic Neural Networks for Hierarchical Reinforcement Learning [OpenReview]
Tuning Recurrent Neural Networks with Reinforcement Learning [OpenReview]
RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning [arXiv]
Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning [OpenReview]
Learning to Perform Physics Experiments via Deep Reinforcement Learning [OpenReview]
Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU [OpenReview]
Learning to Compose Words into Sentences with Reinforcement Learning[OpenReview]
Deep Reinforcement Learning for Accelerating the Convergence Rate [OpenReview]
Learning to Compose Words into Sentences with Reinforcement Learning [OpenReview]
Learning to Navigate in Complex Environments [arXiv]
Unsupervised Perceptual Rewards for Imitation Learning [OpenReview]
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic [OpenReview]
Reading Text in the Wild with Convolutional Neural Networks [arXiv]
Sequence Level Training with Recurrent Neural Networks [arXiv]
Towards Universal Paraphrastic Sentence Embeddings [arXiv]
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies [arXiv]
Sequence Level Training with Recurrent Neural Networks [arXiv]
Natural Language Understanding with Distributed Representation [arXiv]
sense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word Embeddings [arXiv]
LSTM-based Deep Learning Models for non-factoid answer selection [arXiv]
A Primer on Neural Network Models for Natural Language ProcessingYoav Goldberg. October 2015. No new info, 75 page summary of state of the art.
A neural probabilistic language modelBengio 2003. Seminal paper on word vectors.
Skip Thought Vectors- word representation method
Adaptive skip-gram- similar approach, with adaptive properties
Named Entity Recognition
Recursive Deep Models for Semantic Compositionality Over a Sentiment TreebankSocher et al. 2013. Introduces Recursive Neural Tensor Network and dataset: "sentiment treebank." Includesdemo site. Uses a parse tree.
Neural Machine Translation & Dialog
Sequence to Sequence Learning with Neural Networks(nips presentation). Uses seq2seq to generate translations.
Neural Machine Translation by jointly learning to align and translateBahdanau, Cho 2014. "comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation." Implements attention mechanism.English to French Demo
Iterative Refinement for Machine Translation [OpenReview]
A Convolutional Encoder Model for Neural Machine Translation [arXiv]
Improving Neural Language Models with a Continuous Cache [OpenReview]
Vocabulary Selection Strategies for Neural Machine Translation [OpenReview]
Towards an automatic Turing test: Learning to evaluate dialogue responses [OpenReview]
Dialogue Learning With Human-in-the-Loop [OpenReview]
Batch Policy Gradient Methods for Improving Neural Conversation Models [OpenReview]
Learning through Dialogue Interactions [OpenReview]
Unsupervised Pretraining for Sequence to Sequence Learning [arXiv]
Neural Responding Machine for Short-Text ConversationShang et al. 2015 Uses Neural Responding Machine. Trained on Weibo dataset. Achieves one round conversations with 75% appropriate responses.
A Neural Network Approach to Context-Sensitive Generation of Conversational ResponsesSordoni et al. 2015. Generates responses to tweets. UsesRecurrent Neural Network Language Model (RLM) architecture of (Mikolov et al., 2010).source code:RNNLM Toolkit
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network ModelsSerban, Sordoni, Bengio et al. 2015. Extendshierarchical recurrent encoder-decoderneural network (HRED).
Attention with Intention for a Neural Network Conversation ModelYao et al. 2015 Architecture is three recurrent networks: an encoder, an intention network and a decoder.
A Neural Conversation ModelVinyals,Le2015. Uses LSTM RNNs to generate conversational responses. Usesseq2seq framework. Seq2Seq was originally designed for machine translation and it "translates" a single sentence, up to around 79 words, to a single sentence response, and has no memory of previous dialog exchanges. Used in GoogleSmart Reply feature for Inbox
Incorporating Copying Mechanism in Sequence-to-Sequence LearningGu et al. 2016 Proposes CopyNet, builds on seq2seq.
A Persona-Based Neural Conversation ModelLi et al. 2016 Proposes persona-based models for handling the issue of speaker consistency in neural response generation. Builds on seq2seq.
Deep Reinforcement Learning for Dialogue GenerationLi et al. 2016. Uses reinforcement learing to generate diverse responses. Trains 2 agents to chat with each other. Builds on seq2seq.
Deep learning for chatbotsArticle summary of state of the art, and challenges for chatbots.
Show, Attend and Tell: Neural Image Caption Generation with Visual AttentionXu et al. 2015 Creates captions by feeding image into a CNN which feeds into hidden state of an RNN that generates the caption. At each time step the RNN outputs next word and the next location to pay attention to via a probability over grid locations. Uses 2 types of attention soft and hard. Soft attention uses gradient descent and backprop and is deterministic. Hard attention selects the element with highest probability. Hard attention uses reinforcement learning, rather than backprop and is stochastic.
Memory and Attention Models
End-To-End Memory NetworksSukhbaatar et. al 2015.
Towards AI-Complete Question Answering: A Set of Prerequisite Toy TasksWeston 2015. Classifies QA tasks like single factoid, yes/no etc. Extends memory networks.
Evaluating prerequisite qualities for learning end to end dialog systemsDodge et. al 2015. Tests Memory Networks on 4 tasks including reddit dialog task. SeeJason Weston lecture on MemNN
General NLP topics
Neural autocoder for paragraphs and documents- LSTM representation
Sequence to Sequence Learning- word vectors for machine translation
Teaching Machines to Read and Comprehend- DeepMind paper
Tutorial on Markov Logic Networks (based on this paper)