DL QuickStart
(→Furture) |
(→Build from Git in Linux) |
||
(未显示1个用户的86个中间版本) | |||
第1行: | 第1行: | ||
== Overview == | == Overview == | ||
+ | |||
+ | * Convolutional Neural Networks,CNN,卷积神经网络 | ||
+ | * Recurrent Neural Networks, RNN,循环神经网络 | ||
* Deep Belief Networks 深度信念网络 | * Deep Belief Networks 深度信念网络 | ||
第6行: | 第9行: | ||
− | + | CNN 目前是图像识别(计算机视觉)最有效也是使用最多的 | |
− | + | ||
+ | RNN 则多用于自然语言处理、语音识别、手写体识别等 | ||
+ | |||
https://www.zhihu.com/question/31785984/answer/55063559 | https://www.zhihu.com/question/31785984/answer/55063559 | ||
− | + | '''Gitbook:''' | |
− | + | * [https://xhhjin.gitbooks.io/neural-networks-and-deep-learning-zh/content/ 神经网络与深度学习] Michael Nielsen | |
+ | * [https://www.gitbook.com/book/hit-scir/neural-networks-and-deep-learning-zh_cn/details 神经网络与深度学习2] | ||
+ | |||
+ | |||
+ | * [https://www.pyimagesearch.com/ The Deep Learning for Computer Vision with Python] | ||
+ | |||
+ | |||
+ | '''Blog:''' | ||
+ | |||
+ | * [http://www.pyimagesearch.com/start-here-learn-computer-vision-opencv/ Deep Learning for Computer Vision with Python] | ||
+ | * [http://www.pyimagesearch.com/2016/08/01/lenet-convolutional-neural-network-in-python/ LeNet – Convolutional Neural Network in Python] | ||
+ | |||
+ | |||
+ | * [http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html Image Classification Datasets Results] | ||
+ | |||
+ | <br><br> | ||
+ | |||
+ | == CS231n == | ||
+ | |||
+ | 知乎智能单元: https://zhuanlan.zhihu.com/intelligentunit | ||
+ | |||
+ | |||
+ | [https://zhuanlan.zhihu.com/p/21930884 CS231n官方笔记授权翻译总集篇发布] | ||
+ | |||
+ | |||
+ | * [https://zhuanlan.zhihu.com/p/21441838 简介] | ||
+ | * [https://zhuanlan.zhihu.com/p/21407711 反向传播笔记]] | ||
+ | |||
+ | |||
+ | * [https://zhuanlan.zhihu.com/p/21462488 神经网络笔记1(上)] | ||
+ | * [https://zhuanlan.zhihu.com/p/21513367 神经网络笔记1(下)] | ||
+ | * [https://zhuanlan.zhihu.com/p/21560667 神经网络笔记 2] | ||
+ | * [https://zhuanlan.zhihu.com/p/21741716 神经网络笔记3(上)] | ||
+ | * [https://zhuanlan.zhihu.com/p/21798784 神经网络笔记3(下)] | ||
+ | * [https://zhuanlan.zhihu.com/p/22038289 卷积神经网络笔记] | ||
+ | |||
+ | |||
+ | * [https://zhuanlan.zhihu.com/p/21360434 最优化笔记(上)] | ||
+ | * [https://zhuanlan.zhihu.com/p/21387326 最优化笔记(下)] | ||
+ | |||
+ | * [https://zhuanlan.zhihu.com/p/20918580 线性分类笔记(上)] | ||
+ | * [https://zhuanlan.zhihu.com/p/20945670 线性分类笔记(中)] | ||
+ | * [https://zhuanlan.zhihu.com/p/21102293 线性分类笔记(下)] | ||
+ | |||
+ | * [https://zhuanlan.zhihu.com/p/20894041 图像分类笔记(上)] | ||
+ | * [https://zhuanlan.zhihu.com/p/20900216 图像分类笔记(下)] | ||
+ | |||
+ | * [https://zhuanlan.zhihu.com/p/20878530 Python Numpy教程] | ||
+ | |||
+ | |||
+ | * [https://zhuanlan.zhihu.com/p/22107715 循环神经网络惊人的有效性(上)] | ||
+ | |||
+ | <br><br> | ||
+ | |||
+ | == CNN == | ||
+ | |||
+ | * [https://arxiv.org/pdf/1311.2901.pdf Visualizing and Understanding Convolutional Networks] | ||
<br><br> | <br><br> | ||
第37行: | 第98行: | ||
[[文件:Dl-framwork-compare.png]] | [[文件:Dl-framwork-compare.png]] | ||
− | + | <br> | |
− | == | + | === TensorFlow === |
− | https:// | + | * [https://docs.scipy.org/doc/numpy-dev/user/quickstart.html NumPy Quick Start] |
+ | * [http://www.tensorfly.cn/tfdoc/get_started/introduction.html TensorFlow Quick Start CN] | ||
+ | * [https://www.tensorflow.org/get_started/mnist/pros TensorFlow Quick Start] | ||
+ | * [https://medium.com/@hamedmp/exporting-trained-tensorflow-models-to-c-the-right-way-cf24b609d183 Export TensorFlow Model to C++] | ||
− | |||
− | |||
− | + | ==== Install ==== | |
− | + | ||
− | + | <source lang=bash> | |
− | + | $ sudo apt-get install python-pip python-dev # for Python 2.7 | |
− | + | $ sudo apt-get install python3-pip python3-dev # for Python 3.n | |
− | + | $ pip install tensorflow # Python 2.7; CPU support (no GPU support) | |
− | + | $ pip3 install tensorflow # Python 3.n; CPU support (no GPU support) | |
+ | $ pip install tensorflow-gpu # Python 2.7; GPU support | ||
+ | $ pip3 install tensorflow-gpu # Python 3.n; GPU support | ||
+ | </source> | ||
− | + | <source lang=bash> | |
+ | $ pip install -i https://pypi.tuna.tsinghua.edu.cn/simple tensorflow-gpu==2.5.0 | ||
+ | </source> | ||
<br><br> | <br><br> | ||
− | == | + | ==== Uninstall ==== |
− | + | <source lang=bash> | |
+ | $ sudo pip uninstall tensorflow # for Python 2.7 | ||
+ | $ sudo pip3 uninstall tensorflow # for Python 3.n | ||
+ | </source> | ||
<br><br> | <br><br> | ||
− | == | + | ==== Build from Git in Windows ==== |
− | + | The latest guide: https://tensorflow.google.cn/install/source_windows | |
− | https:// | + | |
+ | <br><br> | ||
+ | |||
+ | ==== Build from Git in Linux ==== | ||
+ | |||
+ | https://tensorflow.google.cn/install/source | ||
+ | |||
+ | <source lang=bash> | ||
+ | $ git clone https://github.com/tensorflow/tensorflow | ||
+ | $ cd tensorflow | ||
+ | $ git checkout r1.0 | ||
+ | </source> | ||
+ | |||
+ | * Install Bazel: https://bazel.build/versions/master/docs/install.html | ||
+ | ** https://docs.bazel.build/versions/main/install-ubuntu.html | ||
+ | ** https://github.com/bazelbuild/bazel/releases?expanded=true&page=2&q=0.25 | ||
+ | |||
+ | <source lang=bash> | ||
+ | $ sudo apt-get install python-numpy python-dev python-pip python-wheel | ||
+ | |||
+ | $ sudo apt-get install python3-numpy python3-dev python3-pip python3-wheel | ||
+ | |||
+ | $ sudo apt-get install libcupti-dev | ||
+ | </source> | ||
+ | |||
+ | |||
+ | <source lang=bash> | ||
+ | $ cd tensorflow # cd to the top-level directory created | ||
+ | $ ./configure | ||
+ | Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python2.7 | ||
+ | Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]: | ||
+ | Do you wish to use jemalloc as the malloc implementation? [Y/n] | ||
+ | jemalloc enabled | ||
+ | Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] | ||
+ | No Google Cloud Platform support will be enabled for TensorFlow | ||
+ | Do you wish to build TensorFlow with Hadoop File System support? [y/N] | ||
+ | No Hadoop File System support will be enabled for TensorFlow | ||
+ | Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N] | ||
+ | No XLA JIT support will be enabled for TensorFlow | ||
+ | Found possible Python library paths: | ||
+ | /usr/local/lib/python2.7/dist-packages | ||
+ | /usr/lib/python2.7/dist-packages | ||
+ | Please input the desired Python library path to use. Default is [/usr/local/lib/python2.7/dist-packages] | ||
+ | Using python library path: /usr/local/lib/python2.7/dist-packages | ||
+ | Do you wish to build TensorFlow with OpenCL support? [y/N] N | ||
+ | No OpenCL support will be enabled for TensorFlow | ||
+ | Do you wish to build TensorFlow with CUDA support? [y/N] Y | ||
+ | CUDA support will be enabled for TensorFlow | ||
+ | Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: | ||
+ | Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0 | ||
+ | Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: | ||
+ | Please specify the cuDNN version you want to use. [Leave empty to use system default]: 5 | ||
+ | Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: | ||
+ | Please specify a list of comma-separated Cuda compute capabilities you want to build with. | ||
+ | You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. | ||
+ | Please note that each additional compute capability significantly increases your build time and binary size. | ||
+ | [Default is: "3.5,5.2"]: 3.0 | ||
+ | Setting up Cuda include | ||
+ | Setting up Cuda lib | ||
+ | Setting up Cuda bin | ||
+ | Setting up Cuda nvvm | ||
+ | Setting up CUPTI include | ||
+ | Setting up CUPTI lib64 | ||
+ | Configuration finished | ||
+ | |||
+ | $ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package | ||
+ | |||
+ | $ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg | ||
+ | |||
+ | $ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.2.0-py2-none-any.whl | ||
+ | </source> | ||
<br><br> | <br><br> | ||
+ | |||
+ | ==== Validate your installation ==== | ||
+ | |||
+ | <source lang=bash> | ||
+ | $ python | ||
+ | # Python | ||
+ | import tensorflow as tf | ||
+ | hello = tf.constant('Hello, TensorFlow!') | ||
+ | sess = tf.Session() | ||
+ | print(sess.run(hello)) | ||
+ | |||
+ | Hello, TensorFlow! | ||
+ | </source> | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | === DarkNet === | ||
+ | |||
+ | * [https://pjreddie.com/darknet/tiny-darknet/ Tiny DarkNet] | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | === Caffe === | ||
+ | |||
+ | * [http://hanzratech.in/2015/07/27/installing-caffe-on-ubuntu.html Caffe Installation on Ubuntu 14.04 (CPU) with PYTHON support] | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | == Hardware == | ||
+ | |||
+ | === GPU Architecture === | ||
+ | |||
+ | * [http://haifux.org/lectures/267/Introduction-to-GPUs.pdf Introduction to GPU architecture] | ||
+ | * [[Nvidia GPU Architecture]] | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | === TPU Architecture === | ||
+ | |||
+ | Tensor Processing Unit(or TPUs) are application-specific integrated circuits (ASICs) developed specifically for machine learning. | ||
+ | |||
+ | * [https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk/view Paper] | ||
+ | * [https://cloudplatform.googleblog.com/2017/04/quantifying-the-performance-of-the-TPU-our-first-machine-learning-chip.html Blog] Quantifying the performance of the TPU, our first machine learning chip 2017-04-05 | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | == Notes == | ||
+ | |||
+ | === Install Nvidia driver for ubuntu 16.04 === | ||
+ | |||
+ | Blacklist the modules. Open the blacklist.conf file. | ||
+ | |||
+ | add the following modules in the file /etc/modprobe.d/blacklist.conf | ||
+ | |||
+ | <pre> | ||
+ | #this might not be required for x86 32 bit users | ||
+ | blacklist amd76x_edac | ||
+ | blacklist vga16fb | ||
+ | blacklist nouveau | ||
+ | blacklist rivafb | ||
+ | blacklist nvidiafb | ||
+ | blacklist rivatv | ||
+ | </pre> | ||
+ | |||
+ | |||
+ | <source lang=bash> | ||
+ | $ sudo update-initramfs -u | ||
+ | </source> | ||
+ | |||
+ | |||
+ | reboot the system, then: | ||
+ | |||
+ | <source lang=bash> | ||
+ | $ sudo apt-get install gcc libc-dev make | ||
+ | $ sudo ./NVIDIA-Linux-x86_64-343.22.run # for GeForce 980 | ||
+ | |||
+ | $ sudo ./NVIDIA-Linux-x86_64-381.22.run # for GeForce 1050M | ||
+ | </source> | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | === Install CUDA Toolkit === | ||
+ | |||
+ | Download: https://developer.nvidia.com/cuda-downloads | ||
+ | |||
+ | <source lang=bash> | ||
+ | $ sudo dpkg -i cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64.deb | ||
+ | $ sudo apt-get update | ||
+ | $ sudo apt-get install cuda | ||
+ | </source> | ||
+ | |||
+ | add the following lines into /etc/bash.bashrc: | ||
+ | |||
+ | <source lang=bash> | ||
+ | export PATH=/usr/local/cuda/bin:$PATH | ||
+ | export LD_LIBRARY_PATH="/usr/local/cuda/lib64 $LD_LIBRARY_PATH" | ||
+ | </source> | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | == OpenAI == | ||
+ | |||
+ | [https://zhuanlan.zhihu.com/p/20924929 从OpenAI看深度学习研究前沿] | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | == Applications == | ||
+ | |||
+ | === Computor Vision === | ||
+ | |||
+ | ==== Face detection ==== | ||
+ | |||
+ | * [http://thecodacus.com/opencv-python-face-detection/ Face Detection Using OpenCV In Python | How To Setup OpenCV Python] | ||
+ | * [http://thecodacus.com/opencv-face-recognition-python-part1/ Face Recognition – OpenCV Python | Dataset Generator] | ||
+ | * [http://thecodacus.com/face-recognition-opencv-train-recognizer/ Face Recognition OpenCV – Training A Face Recognizer] | ||
+ | * [http://thecodacus.com/face-recognition-loading-recognizer/ Face Recognition Using OpenCV | Loading Recognizer] | ||
+ | * https://zhuanlan.zhihu.com/p/25025596 | ||
+ | |||
+ | |||
+ | * [http://www.ebaina.com/bbs/forum.php?mod=viewthread&tid=4858 海思移植opencv+人脸检测] | ||
+ | * http://shervinemami.info/faceRecognition.html | ||
+ | |||
+ | * [http://ahogrammer.com/2016/11/15/deep-learning-enables-you-to-hide-screen-when-your-boss-is-approaching/ Boss Sensor] | ||
+ | * [https://medium.com/@hamedmp/exporting-trained-tensorflow-models-to-c-the-right-way-cf24b609d183 Exporting trained TensorFlow models to C++] | ||
+ | |||
+ | |||
+ | * [https://realpython.com/blog/python/face-detection-in-python-using-a-webcam/ Face Detection in Python Using a Webcam] | ||
+ | * [http://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html Face Recognition with OpenCV] | ||
+ | |||
+ | * [http://eyalarubas.com/face-detection-and-recognition.html Face Detection and Recognition (Theory and Practice)] | ||
+ | |||
+ | |||
+ | * [https://zhuanlan.zhihu.com/p/24816781 基于深度学习的人脸识别技术综述] | ||
+ | * [https://zhuanlan.zhihu.com/p/24837264 谷歌人脸识别系统FaceNet解析] | ||
+ | * [https://zhuanlan.zhihu.com/p/25025596 基于mtcnn和facenet的实时人脸检测与识别系统开发] | ||
+ | * [https://zhuanlan.zhihu.com/p/24316690 使用深度卷积神经网络方法进行3D人脸重建] | ||
+ | |||
+ | |||
+ | * [http://blog.csdn.net/cv_family_z/article/details/51917143 CNN网络二值化--XNOR-Net] | ||
+ | |||
+ | * RAISR(Rapid and Accurate Image Super-Resolution),利用机器学习将低分辨率图像转化为高分辨率图像。这项技术能够在节省带宽75%的情况下分辨率效果达到甚至超过原图,同时速度能够提升大约10到100倍 | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | ==== OpenFace ==== | ||
+ | |||
+ | * http://cmusatyalab.github.io/openface/ | ||
+ | * https://github.com/cmusatyalab/openface | ||
+ | |||
+ | * http://shamangary.logdown.com/posts/800267-openface-installation | ||
+ | |||
+ | |||
+ | * [https://github.com/BrandonJoffe/home_surveillance Home surveillance dashboard that uses OpenFace for face recognition] | ||
+ | * [https://github.com/pyannote/pyannote-video Face detection, tracking, and clustering in videos using OpenFace] --- [http://nbviewer.jupyter.org/github/pyannote/pyannote-video/blob/master/doc/getting_started.ipynb Getting Start] | ||
+ | * [https://github.com/davidsandberg/facenet Face recognition using Tensorflow] | ||
+ | * [https://github.com/aybassiouny/OpenFaceCpp C++ implementation for OpenFace library by CMU] | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | ==== Facial Keypoints Detection ==== | ||
+ | |||
+ | * https://www.kaggle.com/c/facial-keypoints-detection Facial Keypoints Detection - Kaggle project | ||
+ | * [https://github.com/dnouri/kfkd-tutorial dnouri's Kaggle Facial Keypoints Detection tutorial] | ||
+ | * https://github.com/saber1988/facial-keypoints-detection | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | ==== Facial Emotion Recognition ==== | ||
+ | |||
+ | * [https://github.com/JostineHo/mememoji A facial expression classification system that recognizes 6 basic emotions: happy, sad, surprise, fear, anger and neutral] | ||
+ | * [http://cs231n.stanford.edu/reports2016/022_Report.pdf Facial Emotion Recognition in Real Time] | ||
+ | * [https://arxiv.org/pdf/1408.3750.pdf Real-time emotion recognition for gaming using deep convolutional network features] | ||
+ | |||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | ==== Texture Matching ==== | ||
+ | |||
+ | * [http://hanzratech.in/2015/05/30/local-binary-patterns.html Texture Matching using Local Binary Patterns (LBP), OpenCV, scikit-learn and Python] | ||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | == Furture == | ||
+ | |||
+ | Deep Reinforcement Learning 深度增强学习资源 (持续更新)https://zhuanlan.zhihu.com/p/20885568?refer=intelligentunit | ||
+ | |||
+ | 获取人工智能AI前沿信息 https://zhuanlan.zhihu.com/p/21263408?refer=intelligentunit | ||
+ | |||
+ | |||
<br><br> | <br><br> | ||
+ | |||
+ | == Reference == | ||
+ | |||
+ | 问:您对Jeff Hawkins对深度学习的批评有什么看法?Hawkins是On Intelligence一书的作者, 该书2004年出版,内容关于大脑如何工作,以及如何参考大脑来制造智能机器。他声称深度学习没有对时间序列建模。人脑是基于一系列的传感数据进行思考的,人的学习主要在于对序列模式的记忆,比如你看到一个搞怪猫的视频,实际是猫的动作让你发笑,而不是像Google公司所用的静态图片。参见这个链接 | ||
+ | |||
+ | 答:时间相关的神经网络其实有很多工作,递归神经网络模型对时间关系隐性建模,通常应用于语音识别。比如下面这两个工作。 | ||
+ | |||
+ | [1] http://www.cs.toronto.edu/~hinton/absps/RNN13.pdf | ||
+ | |||
+ | [2] http://papers.nips.cc/paper/5166 ... neural-networks.pdf | ||
+ | |||
+ | 还有这篇文章:http://arxiv.org/abs/1312.6026. | ||
+ | |||
+ | 自然语言处理中的序列也有所考虑:http://arxiv.org/abs/1306.2795 | ||
+ | |||
+ | |||
+ | 问:根据我的理解,深度神经网络训练上的成功跟选取正确的超参数有关系,比如网络深度,隐含层的大小,稀疏约束值等等。有些论文基于随机搜索来寻找这些参数。可能跟代码写得好也有关系。有没有一个地方能让研究者找到某些特定任务的合理超参数呢?在这些参数的基础上,可能更容易找到更优化的参数。 | ||
+ | |||
+ | 答:可以看上文关于超参数的部分。James Bergstra 继续了这部分工作。我觉得有这么一个数据库,存储着许多推荐的超参数设置,对于神经网络训练是非常有好处的。Github上面的Hyperopt项目,做了类似的事情。hyperopt项目聚焦于神经网络、卷积网络,给出一些超参数设置的建议。以简单的因子分布的形式给出。比如隐含层的数量应该是1到3,每一层的隐含单元数目应该是50到5000。其实超参数还有很多,以及更好的超参数搜索算法等等。下面是更多的参考论文: | ||
+ | |||
+ | *http://arxiv.org/abs/1306.2795 | ||
+ | *http://arxiv.org/abs/1312.6026 | ||
+ | *http://arxiv.org/abs/1308.0850 | ||
+ | *http://papers.nips.cc/paper/5166-training-and-analysing-deep-recurrent-neural-networks.pdf | ||
+ | |||
+ | |||
+ | 问:Bengio教授,在深度学习中,有那么一类方法,采用比较高级的数学如代数和拓扑集合。John Healy几年前声称通过通过范畴论(Category Theory)改进了神经网络(ART1)。您对于这类尝试有什么看法?是儿戏还是很有前途? | ||
+ | |||
+ | 答:可以看看Morton和Montufar的工作,参考附加材料: | ||
+ | *http://www.ece.unm.edu/~mjhealy/ | ||
+ | *http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.98.6807 | ||
+ | 热带几何以及概率模型中的热带几何 | ||
+ | *http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.242.9890 | ||
+ | |||
+ | |||
+ | * [https://zhuanlan.zhihu.com/p/23255679 寒武纪——一个神经网络指令集架构] | ||
+ | * [https://tech.meituan.com/O2O_Intelligent_distribution.html 即时配送的订单分配策略:从建模和优化] | ||
+ | |||
+ | * [[Financial]] | ||
+ | * [http://www.meng.uno/articles/5b828fc2/ 机器学习&深度学习速查表] | ||
+ | * [https://www.leiphone.com/news/201707/TmZfbVdmupizoVcB.html 深度神经网络可视化工具集锦] | ||
+ | * [https://blog.csdn.net/han2529386161/article/details/102723482 关于 MLPerf 的一些调查] | ||
+ | |||
<br><br> | <br><br> | ||
<br><br> | <br><br> | ||
<br><br> | <br><br> |
2022年4月2日 (六) 11:35的最后版本
目录 |
[编辑] 1 Overview
- Convolutional Neural Networks,CNN,卷积神经网络
- Recurrent Neural Networks, RNN,循环神经网络
- Deep Belief Networks 深度信念网络
- Restricted Boltzmann Machine 受限波尔兹曼机
- Stacked Auto-encoders 堆栈式自动编码器
CNN 目前是图像识别(计算机视觉)最有效也是使用最多的
RNN 则多用于自然语言处理、语音识别、手写体识别等
https://www.zhihu.com/question/31785984/answer/55063559
Gitbook:
- 神经网络与深度学习 Michael Nielsen
- 神经网络与深度学习2
Blog:
[编辑] 2 CS231n
知乎智能单元: https://zhuanlan.zhihu.com/intelligentunit
[编辑] 3 CNN
[编辑] 4 RNN
http://blog.csdn.net/heyongluoyao8/article/details/48636251
http://rayz0620.github.io/2015/05/14/rnn_note_1/
https://www.zhihu.com/question/34681168
[编辑] 5 Framework
推荐斯坦福的 CS231n 课程
在 Lecture 12 中,JJ 对于几个主流的库的应用情况,优缺点逐个做了详细的说明:
[编辑] 5.1 TensorFlow
[编辑] 5.1.1 Install
$ sudo apt-get install python-pip python-dev # for Python 2.7 $ sudo apt-get install python3-pip python3-dev # for Python 3.n $ pip install tensorflow # Python 2.7; CPU support (no GPU support) $ pip3 install tensorflow # Python 3.n; CPU support (no GPU support) $ pip install tensorflow-gpu # Python 2.7; GPU support $ pip3 install tensorflow-gpu # Python 3.n; GPU support
$ pip install -i https://pypi.tuna.tsinghua.edu.cn/simple tensorflow-gpu==2.5.0
[编辑] 5.1.2 Uninstall
$ sudo pip uninstall tensorflow # for Python 2.7 $ sudo pip3 uninstall tensorflow # for Python 3.n
[编辑] 5.1.3 Build from Git in Windows
The latest guide: https://tensorflow.google.cn/install/source_windows
[编辑] 5.1.4 Build from Git in Linux
https://tensorflow.google.cn/install/source
$ git clone https://github.com/tensorflow/tensorflow $ cd tensorflow $ git checkout r1.0
- Install Bazel: https://bazel.build/versions/master/docs/install.html
$ sudo apt-get install python-numpy python-dev python-pip python-wheel $ sudo apt-get install python3-numpy python3-dev python3-pip python3-wheel $ sudo apt-get install libcupti-dev
$ cd tensorflow # cd to the top-level directory created $ ./configure Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python2.7 Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]: Do you wish to use jemalloc as the malloc implementation? [Y/n] jemalloc enabled Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] No Google Cloud Platform support will be enabled for TensorFlow Do you wish to build TensorFlow with Hadoop File System support? [y/N] No Hadoop File System support will be enabled for TensorFlow Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N] No XLA JIT support will be enabled for TensorFlow Found possible Python library paths: /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages Please input the desired Python library path to use. Default is [/usr/local/lib/python2.7/dist-packages] Using python library path: /usr/local/lib/python2.7/dist-packages Do you wish to build TensorFlow with OpenCL support? [y/N] N No OpenCL support will be enabled for TensorFlow Do you wish to build TensorFlow with CUDA support? [y/N] Y CUDA support will be enabled for TensorFlow Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0 Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: Please specify the cuDNN version you want to use. [Leave empty to use system default]: 5 Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: Please specify a list of comma-separated Cuda compute capabilities you want to build with. You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Please note that each additional compute capability significantly increases your build time and binary size. [Default is: "3.5,5.2"]: 3.0 Setting up Cuda include Setting up Cuda lib Setting up Cuda bin Setting up Cuda nvvm Setting up CUPTI include Setting up CUPTI lib64 Configuration finished $ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package $ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg $ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.2.0-py2-none-any.whl
[编辑] 5.1.5 Validate your installation
$ python # Python import tensorflow as tf hello = tf.constant('Hello, TensorFlow!') sess = tf.Session() print(sess.run(hello)) Hello, TensorFlow!
[编辑] 5.2 DarkNet
[编辑] 5.3 Caffe
[编辑] 6 Hardware
[编辑] 6.1 GPU Architecture
[编辑] 6.2 TPU Architecture
Tensor Processing Unit(or TPUs) are application-specific integrated circuits (ASICs) developed specifically for machine learning.
[编辑] 7 Notes
[编辑] 7.1 Install Nvidia driver for ubuntu 16.04
Blacklist the modules. Open the blacklist.conf file.
add the following modules in the file /etc/modprobe.d/blacklist.conf
#this might not be required for x86 32 bit users blacklist amd76x_edac blacklist vga16fb blacklist nouveau blacklist rivafb blacklist nvidiafb blacklist rivatv
$ sudo update-initramfs -u
reboot the system, then:
$ sudo apt-get install gcc libc-dev make $ sudo ./NVIDIA-Linux-x86_64-343.22.run # for GeForce 980 $ sudo ./NVIDIA-Linux-x86_64-381.22.run # for GeForce 1050M
[编辑] 7.2 Install CUDA Toolkit
Download: https://developer.nvidia.com/cuda-downloads
$ sudo dpkg -i cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64.deb $ sudo apt-get update $ sudo apt-get install cuda
add the following lines into /etc/bash.bashrc:
export PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH="/usr/local/cuda/lib64 $LD_LIBRARY_PATH"
[编辑] 8 OpenAI
[编辑] 9 Applications
[编辑] 9.1 Computor Vision
[编辑] 9.1.1 Face detection
- Face Detection Using OpenCV In Python | How To Setup OpenCV Python
- Face Recognition – OpenCV Python | Dataset Generator
- Face Recognition OpenCV – Training A Face Recognizer
- Face Recognition Using OpenCV | Loading Recognizer
- https://zhuanlan.zhihu.com/p/25025596
- RAISR(Rapid and Accurate Image Super-Resolution),利用机器学习将低分辨率图像转化为高分辨率图像。这项技术能够在节省带宽75%的情况下分辨率效果达到甚至超过原图,同时速度能够提升大约10到100倍
[编辑] 9.1.2 OpenFace
- Home surveillance dashboard that uses OpenFace for face recognition
- Face detection, tracking, and clustering in videos using OpenFace --- Getting Start
- Face recognition using Tensorflow
- C++ implementation for OpenFace library by CMU
[编辑] 9.1.3 Facial Keypoints Detection
- https://www.kaggle.com/c/facial-keypoints-detection Facial Keypoints Detection - Kaggle project
- dnouri's Kaggle Facial Keypoints Detection tutorial
- https://github.com/saber1988/facial-keypoints-detection
[编辑] 9.1.4 Facial Emotion Recognition
- A facial expression classification system that recognizes 6 basic emotions: happy, sad, surprise, fear, anger and neutral
- Facial Emotion Recognition in Real Time
- Real-time emotion recognition for gaming using deep convolutional network features
[编辑] 9.1.5 Texture Matching
[编辑] 10 Furture
Deep Reinforcement Learning 深度增强学习资源 (持续更新)https://zhuanlan.zhihu.com/p/20885568?refer=intelligentunit
获取人工智能AI前沿信息 https://zhuanlan.zhihu.com/p/21263408?refer=intelligentunit
[编辑] 11 Reference
问:您对Jeff Hawkins对深度学习的批评有什么看法?Hawkins是On Intelligence一书的作者, 该书2004年出版,内容关于大脑如何工作,以及如何参考大脑来制造智能机器。他声称深度学习没有对时间序列建模。人脑是基于一系列的传感数据进行思考的,人的学习主要在于对序列模式的记忆,比如你看到一个搞怪猫的视频,实际是猫的动作让你发笑,而不是像Google公司所用的静态图片。参见这个链接
答:时间相关的神经网络其实有很多工作,递归神经网络模型对时间关系隐性建模,通常应用于语音识别。比如下面这两个工作。
[1] http://www.cs.toronto.edu/~hinton/absps/RNN13.pdf
[2] http://papers.nips.cc/paper/5166 ... neural-networks.pdf
还有这篇文章:http://arxiv.org/abs/1312.6026.
自然语言处理中的序列也有所考虑:http://arxiv.org/abs/1306.2795
问:根据我的理解,深度神经网络训练上的成功跟选取正确的超参数有关系,比如网络深度,隐含层的大小,稀疏约束值等等。有些论文基于随机搜索来寻找这些参数。可能跟代码写得好也有关系。有没有一个地方能让研究者找到某些特定任务的合理超参数呢?在这些参数的基础上,可能更容易找到更优化的参数。
答:可以看上文关于超参数的部分。James Bergstra 继续了这部分工作。我觉得有这么一个数据库,存储着许多推荐的超参数设置,对于神经网络训练是非常有好处的。Github上面的Hyperopt项目,做了类似的事情。hyperopt项目聚焦于神经网络、卷积网络,给出一些超参数设置的建议。以简单的因子分布的形式给出。比如隐含层的数量应该是1到3,每一层的隐含单元数目应该是50到5000。其实超参数还有很多,以及更好的超参数搜索算法等等。下面是更多的参考论文:
- http://arxiv.org/abs/1306.2795
- http://arxiv.org/abs/1312.6026
- http://arxiv.org/abs/1308.0850
- http://papers.nips.cc/paper/5166-training-and-analysing-deep-recurrent-neural-networks.pdf
问:Bengio教授,在深度学习中,有那么一类方法,采用比较高级的数学如代数和拓扑集合。John Healy几年前声称通过通过范畴论(Category Theory)改进了神经网络(ART1)。您对于这类尝试有什么看法?是儿戏还是很有前途?
答:可以看看Morton和Montufar的工作,参考附加材料:
热带几何以及概率模型中的热带几何