本文介绍了从头开始实现 dropout的处理方法，对大家解决问题具有一定的参考价值
问题描述
此代码尝试使用 dropout 的自定义实现:
%reset -f进口火炬将 torch.nn 导入为 nn# 导入火炬视觉# 导入 torchvision.transforms 作为转换进口火炬将 torch.nn 导入为 nn将 torch.utils.data 导入为 data_utils将 numpy 导入为 np导入 matplotlib.pyplot 作为 plt导入 torch.nn.functional 作为 Fnum_epochs = 1000number_samples = 10从 sklearn.datasets 导入 make_moons从 matplotlib 导入 pyplot从熊猫导入数据帧# 生成二维分类数据集X, y = make_moons(n_samples=number_samples, 噪声=0.1)# 散点图，按类别值着色的点x_data = [a for a in enumerate(X)]x_data_train = x_data[:int(len(x_data) * .5)]x_data_train = [i[1] for i in x_data_train]x_data_trainy_data = [y[i[0]] for i in x_data]y_data_train = y_data[:int(len(y_data) * .5)]y_data_trainx_test = [a[1] for a in x_data[::-1][:int(len(x_data) * .5)]]y_test = [a for a in y_data[::-1][:int(len(y_data) * .5)]]x = torch.tensor(x_data_train).float() # <2>打印(x)y = torch.tensor(y_data_train).long()打印(y)x_test = torch.tensor(x_test).float()打印(x_test)y_test = torch.tensor(y_test).long()打印(y_test)类辍学(nn.Module):def __init__(self, p=0.5, inplace=False):# 打印(p)超级(辍学，自我).__init__()如果 p <0 或 p >1:raise ValueError("辍学概率必须在 0 和 1 之间，"但是得到了 {}".format(p))自我.p = pself.inplace = 就地定义转发(自我，输入):打印(列表(输入.形状))return np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p)))def __repr__(self):inplace_str = ', inplace' if self.inplace else ''返回 self.__class__.__name__ + '(' + 'p=' + str(self.p) + inplace_str + ')'类 MyLinear(nn.Linear):def __init__(self, in_feats, out_feats, drop_p, bias=True):super(MyLinear, self).__init__(in_feats, out_feats,bias=bias)self.custom_dropout = Dropout(p=drop_p)定义转发(自我，输入):dropout_value = self.custom_dropout(self.weight)返回 F.linear(输入，dropout_value，self.bias)my_train = data_utils.TensorDataset(x, y)train_loader = data_utils.DataLoader(my_train,batch_size=2,shuffle=True)my_test = data_utils.TensorDataset(x_test, y_test)test_loader = data_utils.DataLoader(my_train,batch_size=2,shuffle=True)# 设备配置设备 = 'cpu'打印(设备)# 超参数输入大小 = 2隐藏尺寸 = 100num_classes = 2学习率 = 0.0001预测 = []# 一个隐藏层的全连接神经网络类神经网络(nn.Module):def __init__(self, input_size, hidden_size, num_classes, p):super(NeuralNet, self).__init__()# self.drop_layer = nn.Dropout(p=p)# self.drop_layer = MyLinear()# self.fc1 = MyLinear(input_size, hidden_size, p)self.fc1 = MyLinear(input_size, hidden_size , p)self.relu = nn.ReLU()self.fc2 = nn.Linear(hidden_size, num_classes)def forward(self, x):# out = self.drop_layer(x)out = self.fc1(x)out = self.relu(out)out = self.fc2(out)回来模型 = NeuralNet(input_size, hidden_size, num_classes, p=0.9).to(device)# 损失和优化器标准 = nn.CrossEntropyLoss()优化器 = torch.optim.Adam(model.parameters(), lr=learning_rate)# 训练模型total_step = len(train_loader)对于范围内的纪元(num_epochs):对于枚举(train_loader)中的 i，(图像，标签):# 将张量移动到配置的设备图像 = images.reshape(-1, 2).to(device)标签 = 标签.to(设备)# 前传输出 = 模型(图像)损失=标准(输出，标签)# 向后优化optimizer.zero_grad()损失.向后()优化器.step()如果(纪元)% 100 == 0:print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))自定义 dropout 实现为:
class Dropout(nn.Module):def __init__(self, p=0.5, inplace=False):# 打印(p)超级(辍学，自我).__init__()如果 p <0 或 p >1:raise ValueError("辍学概率必须在 0 和 1 之间，"但是得到了 {}".format(p))自我.p = pself.inplace = 就地定义转发(自我，输入):打印(列表(输入.形状))return np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p)))def __repr__(self):inplace_str = ', inplace' if self.inplace else ''返回 self.__class__.__name__ + '(' + 'p=' + str(self.p) + inplace_str + ')'类 MyLinear(nn.Linear):def __init__(self, in_feats, out_feats, drop_p, bias=True):super(MyLinear, self).__init__(in_feats, out_feats,bias=bias)self.custom_dropout = Dropout(p=drop_p)定义转发(自我，输入):dropout_value = self.custom_dropout(self.weight)返回 F.linear(输入，dropout_value，self.bias)我似乎错误地实现了 dropout 功能?:
np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))如何修改才能正确利用dropout?
这些帖子对达到这一点很有用:
Hinton 的 3 行 Python 辍学:).因此:
倒置辍学有点不同.这种方法包括在训练阶段对激活进行缩放，而保持测试阶段不变.比例因子是保持概率 1/1-p = 1/q 的倒数，因此:
反向丢弃有助于定义模型一次，只需更改参数(保持/丢弃概率)即可在同一模型上运行训练和测试.相反，Direct Dropout 会强制您在测试阶段修改网络，因为如果您不乘以 q 输出，神经元将产生比后续神经元预期的值更高的值(因此以下神经元可能会饱和或爆炸):这就是 Inverted Dropout 是更常见实现的原因.
参考文献:
如何实现反向 dropout Pytorch?
class MyDropout(nn.Module):def __init__(self, p: float = 0.5):super(MyDropout, self).__init__()如果 p <0 或 p >1:raise ValueError("dropout 概率必须在 0 到 1 之间，""but got {}".format(p))自我.p = pdef forward(self, X):如果自我训练:二项式 = torch.distributions.binomial.Binomial(probs=1-self.p)返回 X * 二项式.sample(X.size()) * (1.0/(1-self.p))返回 X如何在Numpy中实现?
将 numpy 导入为 nppKeep = 0.8权重 = np.ones([1, 5])binary_value = np.random.rand(weights.shape[0], weights.shape[1]) <保持res = np.multiply(weights, binary_value)res/= pKeep # 这一行被称为反向 dropout 技术打印(资源)如何在 Tensorflow 中实现?
将 tensorflow 导入为 tftf.enable_eager_execution()权重 = tf.ones(shape=[1, 5])keep_prob = 0.8random_tensor = keep_probrandom_tensor += tf.random_uniform(weights.shape)# 0. if [keep_prob, 1.0) 和 1. if [1.0, 1.0 + keep_prob)binary_tensor = tf.floor(random_tensor)ret = tf.div(weights, keep_prob) * binary_tensor打印(ret)This code attempts to utilize a custom implementation of dropout :
%reset -f
import torch
import torch.nn as nn
# import torchvision
# import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
import torch.nn.functional as F
num_epochs = 1000
number_samples = 10
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
# generate 2d classification dataset
X, y = make_moons(n_samples=number_samples, noise=0.1)
# scatter plot, dots colored by class value
x_data = [a for a in enumerate(X)]
x_data_train = x_data[:int(len(x_data) * .5)]
x_data_train = [i[1] for i in x_data_train]
x_data_train
y_data = [y[i[0]] for i in x_data]
y_data_train = y_data[:int(len(y_data) * .5)]
y_data_train
x_test = [a[1] for a in x_data[::-1][:int(len(x_data) * .5)]]
y_test = [a for a in y_data[::-1][:int(len(y_data) * .5)]]
x = torch.tensor(x_data_train).float() # <2>
print(x)
y = torch.tensor(y_data_train).long()
print(y)
x_test = torch.tensor(x_test).float()
print(x_test)
y_test = torch.tensor(y_test).long()
print(y_test)
class Dropout(nn.Module):
def __init__(self, p=0.5, inplace=False):
# print(p)
super(Dropout, self).__init__()
if p < 0 or p > 1:
raise ValueError("dropout probability has to be between 0 and 1, "
"but got {}".format(p))
self.p = p
self.inplace = inplace
def forward(self, input):
print(list(input.shape))
return np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))
def __repr__(self):
inplace_str = ', inplace' if self.inplace else ''
return self.__class__.__name__ + '('
+ 'p=' + str(self.p)
+ inplace_str + ')'
class MyLinear(nn.Linear):
def __init__(self, in_feats, out_feats, drop_p, bias=True):
super(MyLinear, self).__init__(in_feats, out_feats, bias=bias)
self.custom_dropout = Dropout(p=drop_p)
def forward(self, input):
dropout_value = self.custom_dropout(self.weight)
return F.linear(input, dropout_value, self.bias)
my_train = data_utils.TensorDataset(x, y)
train_loader = data_utils.DataLoader(my_train, batch_size=2, shuffle=True)
my_test = data_utils.TensorDataset(x_test, y_test)
test_loader = data_utils.DataLoader(my_train, batch_size=2, shuffle=True)
# Device configuration
device = 'cpu'
print(device)
# Hyper-parameters
input_size = 2
hidden_size = 100
num_classes = 2
learning_rate = 0.0001
pred = []
# Fully connected neural network with one hidden layer
class NeuralNet(nn.Module):
def __init__(self, input_size, hidden_size, num_classes, p):
super(NeuralNet, self).__init__()
# self.drop_layer = nn.Dropout(p=p)
# self.drop_layer = MyLinear()
# self.fc1 = MyLinear(input_size, hidden_size, p)
self.fc1 = MyLinear(input_size, hidden_size , p)
self.relu = nn.ReLU()
self.fc2 = nn.Linear(hidden_size, num_classes)
def forward(self, x):
# out = self.drop_layer(x)
out = self.fc1(x)
out = self.relu(out)
out = self.fc2(out)
return out
model = NeuralNet(input_size, hidden_size, num_classes, p=0.9).to(device)
# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
for i, (images, labels) in enumerate(train_loader):
# Move tensors to the configured device
images = images.reshape(-1, 2).to(device)
labels = labels.to(device)
# Forward pass
outputs = model(images)
loss = criterion(outputs, labels)
# Backward and optimize
optimizer.zero_grad()
loss.backward()
optimizer.step()
if (epoch) % 100 == 0:
print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))
Custom dropout is implemented as :
class Dropout(nn.Module):
def __init__(self, p=0.5, inplace=False):
# print(p)
super(Dropout, self).__init__()
if p < 0 or p > 1:
raise ValueError("dropout probability has to be between 0 and 1, "
"but got {}".format(p))
self.p = p
self.inplace = inplace
def forward(self, input):
print(list(input.shape))
return np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))
def __repr__(self):
inplace_str = ', inplace' if self.inplace else ''
return self.__class__.__name__ + '('
+ 'p=' + str(self.p)
+ inplace_str + ')'
class MyLinear(nn.Linear):
def __init__(self, in_feats, out_feats, drop_p, bias=True):
super(MyLinear, self).__init__(in_feats, out_feats, bias=bias)
self.custom_dropout = Dropout(p=drop_p)
def forward(self, input):
dropout_value = self.custom_dropout(self.weight)
return F.linear(input, dropout_value, self.bias)
It seems I've implemented the dropout function incorrectly ? :
np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1-dropout_percent)[0] * (1.0/(1-self.p))
How to modify in order to correctly utilize dropout ?
These posts were useful in getting to this point :
Hinton's Dropout in 3 Lines of Python : https://iamtrask.github.io/2015/07/28/dropout/
Making a Custom Dropout Function : https://discuss.pytorch.org/t/making-a-custom-dropout-function/14053/2
解决方案
It seems I've implemented the dropout function incorrectly?
np.random.binomial([np.ones((len(input),np.array(list(input.shape))))],1 dropout_percent)[0] * (1.0/(1-self.p))
In fact, the above implementation is known as Inverted Dropout. Inverted Dropout is how Dropout is implemented in practice in the various deep learning frameworks.
What is inverted dropout?
Before jump into the inverted dropout, it can be helpful to see how Dropout works for a single neuron:
Since during train phase a neuron is kept on with probability q (=1-p), during the testing phase we have to emulate the behavior of the ensemble of networks used in the training phase. To this end, the authors suggest scaling the activation function by a factor of q during the test phase in order to use the expected output produced in the training phase as the single output required in the test phase (Section 10, Multiplicative Gaussian Noise). Thus:
Inverted dropout is a bit different. This approach consists in the scaling of the activations during the training phase, leaving the test phase untouched. The scale factor is the inverse of the keep probability 1/1-p = 1/q, thus:
Inverted dropout helps to define the model once and just change a parameter (the keep/drop probability) to run train and test on the same model. Direct Dropout, instead, force you to modify the network during the test phase because if you don’t multiply by q the output the neuron will produce values that are higher respect to the one expected by the successive neurons (thus the following neurons can saturate or explode): that’s why Inverted Dropout is the more common implementation.
References:
How implement inverted dropout Pytorch?
class MyDropout(nn.Module):
def __init__(self, p: float = 0.5):
super(MyDropout, self).__init__()
if p < 0 or p > 1:
raise ValueError("dropout probability has to be between 0 and 1, " "but got {}".format(p))
self.p = p
def forward(self, X):
if self.training:
binomial = torch.distributions.binomial.Binomial(probs=1-self.p)
return X * binomial.sample(X.size()) * (1.0/(1-self.p))
return X
How to implement in Numpy?
import numpy as np
pKeep = 0.8
weights = np.ones([1, 5])
binary_value = np.random.rand(weights.shape[0], weights.shape[1]) < pKeep
res = np.multiply(weights, binary_value)
res /= pKeep # this line is called inverted dropout technique
print(res)
How to implement in Tensorflow?
import tensorflow as tf
tf.enable_eager_execution()
weights = tf.ones(shape=[1, 5])
keep_prob = 0.8
random_tensor = keep_prob
random_tensor += tf.random_uniform(weights.shape)
# 0. if [keep_prob, 1.0) and 1. if [1.0, 1.0 + keep_prob)
binary_tensor = tf.floor(random_tensor)
ret = tf.div(weights, keep_prob) * binary_tensor
print(ret)
这篇关于从头开始实现 dropout的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，WP2