I have trained the following CNN model using pylearn2.
h1
Input space: Conv2DSpace(shape=(25, 150), num_channels=1, axes=('b', 0, 1, 'c'), dtype=float64)
Total input dimension: 3750
h2
Input space: Conv2DSpace(shape=(11, 73), num_channels=8, axes=('b', 'c', 0, 1), dtype=float64)
Total input dimension: 6424
h3
Input space: VectorSpace(dim=1024, dtype=float64)
Total input dimension: 1024
h4
Input space: VectorSpace(dim=1024, dtype=float64)
Total input dimension: 1024
y
Input space: VectorSpace(dim=1024, dtype=float64)
Total input dimension: 1024
You can observe that input examples to this CNN are gray images of size 25 x150. The final number of outputs are 10, that is, the layer 'y' has an output dimension of 10.
My training dataset is created using the CSVDataset in pylearn2, and I'm able to train the model.
However, I have a problem in making predictions using this model, which I'm trying to do using the predict_csv.py file in scripts/mlp folder.
The problem is that predict_csv.py directly loads the test.csv file into a 2d matrix of 1000 x 3750 representing 1000 test examples each having 3750 pixels each. However, while predicting theano expects the input to be of the same format as input of layer 'h1'. The following error occurs:
TypeError: ('Bad input argument to theano function with name "../mlp/predict_csv.py:111" at index 0(0-based)', 'Wrong number of dimensions: expected 4, got 2 with shape (1000, 3750).')
I guess the required format is the ('b', 0, 1, 'c') format of pylearn2.
I would really like to know how do we make this transformation from the 2d array to the above required format. Or any other way this problem could be dealt with?
To solve my problem, I ended up manually converting the 2D set of images (1000 x 3750) to a 4D array with columns as number-of-examples, rows and columns in image, and number-of-channels (1000 x 25 x 150 x 1). It worked fine after this transformation.
I was hoping to find a pylearn2 class or function that directly served my purpose, because while training, pylearn2 is obviously making this change in space itself.
Related
I have a list of Raster Objects, with different extents and dimensions (columns/rows), namely (0, 1400, 0, 2000) and (500,350). The resolution is 4 (dimensions * resolution equals extent). The data values in each raster are either NA or 1.
I would like to index in the object, but using its extent, and not dimensions (so index in the range 0-2000 instead of 0-500). So, I would like to change the raster dimensions to match the extent of the object. Adjusting the resolution does not seem to apply in this case.
I tried to use resample function after creating a new raster with the appropriate dimensions, but after resampling the new raster had correct dimensions but the extent and values (all NA) were messed up. I suppose the solution might be adjusting this approach
ref
a <- raster(nrow = ymax(ref), ncol = xmax(ref))
a <- resample(ref,a)
If the raster was 500x350 pixels before, I would like it to be 2000x1400. So I would guess that where 1 pixel was, there will be 4. And I would like those 4 pixels to all have the same value as the one before.
Any ideas would be appreciated!
I have attached an example raster: https://ucarecdn.com/7cc71657-c829-4a3c-b522-7024b7996efe/
Let's just say I have the following scatterplot:
set.seed(665544)
n <- 100
x <- cbind(
x=runif(10, 0, 5) + rnorm(n, sd=0.4),
y=runif(10, 0, 5) + rnorm(n, sd=0.4)
)
plot(x)
I want to divide this scatterplot into square cells of a specified size and then count how many points fall into each unique cell. This will essentially give me the local density value of that cell. What is the best way of doing this? Is there an R package that can help? Perhaps a 2D histogram method like in Matlab?
Quick clarifications:
1.) I'd like the function/method to take the following 3 arguments: dimensions of total area, dimensions of cell (OR number of cells), and the data. It would then perhaps output a matrix where each value corresponds to a cell's point count.
2.) Q: Why do you want to use this method to determine local density? Isn't this much easier:
library(dbscan)
pointdensity(x, eps = .1, type = "frequency")
A: This method calculates the local density around each point. Though easy, this definition of local density then makes it very difficult (optimization algorithms necessary) to assign new data in a way that it matches the local density distribution of the original data set.
I am using a multibeam echosounder to create a raster stack in R with layers all in the same resolution, which I then convert to a data frame so I can create additive models to describe the distribution of fish around bathymetry variables (depth, aspect, slope, roughness etc.).
The issue I have is that I would like to keep my resonse variable (fish school volume) fine and my predictive variables (bathymetry) coarse, such that I have say 1 x 1m cells representing the distribution of fish schools and 10 x 10m cells representing bathymetry (so the coarse cell is divisible by the fine cell with no remainder).
I can easily create these rasters individually but relating them is the problem. As each coarser cell would contain 10 x 10 = 100 finer cells, I am not sure how to program this into R so that the values are in the right location relative to an x and a y column (for cell addresses). But I realise in this case, I would need each coarse cell value to be repeated 100 times in the data frame.
Any advice would be greatly appreciated! Thanks!
I am trying to fit a line to an image based on the intensity (or color) of the pixels. The figure below shows a typical test image in panel 1 with a line manually drawn in panel 2. The test image (matrix) can be downloaded here: .RData from dropbox .
I would like to use a regression analysis to produce something similar to the manually drawn line in panel 2. However, I can not use a simple linear regression because, as with all images, there are errors in both the x and y axes.
I am open to algorithm descriptions with relevant equations, links, etc... and not necessarily code that I can copy and paste.
METHODS I WANT TO AVOID
Correlating a series synthetic binary images of pixels drawn at various slopes with the actual data image. For example the correlation of the two images below would be quite good, but again, I want to avoid this method.
Using a skeletonization algorithm to reduce the image such that a simple linear regression can be used.
Seismologists, interestingly enough, deal with similar problems where they correct reflection data based on the distance between a seismic source and a receiver with a process known as normal move out (Normal Moveout). I used a similar process.
The general algorithm is:
load in the image
define a series of slopes to investigate
define a window length that is < number of image columns
loop over the series of slopes and...
define index locations (x,y) over the image based on the slope and the size of the window (gray points in row one of image below).
build a matrix from those original matrix indexed at the x,y locations from above (plots in row two of image below).
sum the matrix then normalize the sum by dividing by the length of the summed matrix.
save the each sum (there will be 1 sum for every velocity you loop over)
The velocity vector corresponding to the max (or min) index of the sum vector is the best slope/velocity of the image at that current pixel column (row three in image below).
Perform the above steps along the columns of the image.
The algorithm is visually described in the image below.
The code to perform the above procedure is on one column of the test data given in the question is:
load('test.RData')
## INPUTS ##
img=test
vel.min=1 ## minimum velocity (or slope) to test
vel.max=20 ## max velocity to test
vel.number=100 ## how many velocities to test
win=10 ## size of window to investigate
## define a time index
ti=nrow(img)/2
## set up a vector to hold the velocity correlation values
vel.corrs <- rep(NA,vel.number)
## define the set of velocities to search over
vels <- seq(vel.min,vel.max,length.out=vel.number)
## define a velocity index
vi=1
while(vi<=length(vels)) {
## build a binary matrix with corresponding to the window and velocity
bin.mat <- matrix(0,ncol=ncol(img),nrow=nrow(img))
slope.line <- seq(0,ncol(bin.mat)/vels[vi],length.out=ncol(bin.mat))
bin.mat[(ti-win/2):(ti+win/2),]=1
## define the offeset
offset <- rep(slope.line,each=win+1)
## define the indices of array points according to velocity and window
win.vel.ind <- cbind(which(bin.mat==1,arr.ind=TRUE)[,1]+offset,which(bin.mat==1,arr.ind=TRUE)[,2])
## limit the points to the dimensions of the image
if(any(floor(win.vel.ind[,1]) > nrow(img))){
win.vel.ind[(which(floor(win.vel.ind[,1])>nrow(img))),]=NA
##win.vel.ind <- win.vel.ind[-(which(floor(win.vel.ind[,1])>nrow(img))),]
}
## pluck the values of the image associated with those non-NA indices
slice <- img[win.vel.ind]
## build a matrix of the slice vector with nrow=win+1
slice.mat <- matrix(slice,nrow=win+1,ncol=ncol(img),byrow=FALSE)
## apply a hamming window
##ham.mat <- matrix(hamming(win+1),ncol=ncol(slice.mat),nrow=nrow(slice.mat))
##slice.ham <- slice.mat*ham.mat
## sum this 'slice' and normalize and store
vel.corrs[vi] <- sum(slice,na.rm=TRUE)/length(na.omit(slice))
vi=vi+1
}
I would like to calculate dimensions of fractal written as a n-dimensional array of 0s and 1s. It includes boxing count, hausdorff and packing dimension.
I have only idea how to code boxing count dimensions (just counting 1's in n-dimensional matrix and then use this formula:
boxing_count=-log(v)/log(n);
where n-number of 1's and n-space dimension (R^n)
This approach simulate counting minimal resolution boxes 1 x 1 x ... x 1 so numerical it is like limit eps->0. What do you think about this solution?
Do you have any idea (or maybe code) for calculating hausdorff or packing dimension?
The Hausdorff and packing dimension are purely mathematical tools based in measure theory. They have wonderful properties in that context but are not well suited for experimentation. In short, there is no reason to expect that you can estimate their values based on a single matrix approximation to some set.
Box counting dimension, by contrast, is well suited for numerical investigation. Specifically, let N(e) denote the number of squares of side length e required to cover your fractal set. As you seem to know, the box counting dimension of your set is the limit as e->0 of
log(N(e))/log(1/e)
However, I don't think that just choosing the smallest available value of e is generally a good idea. The standard interpretation in the physics literature, as I understand it, is to presume that the relationship between N(e) and e should be maintained over a broad range of values. A standard way to compute the box-counting dimension is compute N(e) for some choices of e chosen from a sequence that tends geometrically to zero. We then fit a line to the points in a log-log plot of N(e) versus 1/e The box-counting dimension should be approximately the slope of that line.
Example
As a concrete example, the following Python code generates a binary matrix that describes a fractal structure.
import numpy as np
size = 1024
first_row = np.zeros(size, dtype=int)
first_row[int(size/2)-1] = 1
rows = np.zeros((int(size/2),size),dtype=int)
rows[0] = first_row
for i in range(1,int(size/2)):
rows[i] = (np.roll(rows[i-1],-1) + rows[i-1] + np.roll(rows[i-1],1)) % 2
m = int(np.log(size)/np.log(2))
rows = rows[0:2**(m-1),0:2**m]
We can view the fractal structure by simply interpreting each 1 as a black pixel and each zero as white pixel.
import matplotlib.pyplot as plt
plt.matshow(rows, cmap = plt.cm.binary)
This matrix makes a nice test since it can be shown that there is an actual limiting object whose fractal dimension is log(1+sqrt(5))/log(2) or approximately 1.694, yet it's complicated enough to make the box counting estimate a little tricky.
Now, this matrix is 512 rows by 1024 columns; it decomposes naturally into 2 matrices that are 512 by 512. Each of those decomposes naturally into 4 matrices that are 256 by 256, etc. For each such decomposition, we need to count the number of sub matrices that have at least one non-zero element. We can perform this analysis as follows:
cnts = []
for lev in range(m):
block_size = 2**lev
cnt = 0
for j in range(int(size/(2*block_size))):
for i in range(int(size/block_size)):
cnt = cnt + rows[j*block_size:(j+1)*block_size, i*block_size:(i+1)*block_size].any()
cnts.append(cnt)
data = np.array([(2**(m-(k+1)),cnts[k]) for k in range(m)])
data
# Out:
# array([[ 512, 45568],
# [ 256, 22784],
# [ 128, 7040],
# [ 64, 2176],
# [ 32, 672],
# [ 16, 208],
# [ 8, 64],
# [ 4, 20],
# [ 2, 6],
# [ 1, 2]])
Now, your idea is to simply compute log(45568)/log(512) or approximately 1.7195, which is not too bad. I'm recommending that we examine a log-log plot of this data.
xs = np.log(data[:,0])
ys = np.log(data[:,1])
plt.plot(xs,ys, 'o')
This indeed looks close to linear, indicating that we might expect our box-counting technique to work reasonably well. First, though, it might be reasonable to exclude the one point that appears to be an outlier. In fact, that's one of the desirable characteristics of this approach. Here's how to do so:
plt.plot(xs,ys, 'o')
xs = xs[1:]
ys = ys[1:]
A = np.vstack([xs, np.ones(len(xs))]).T
m,b = np.linalg.lstsq(A, ys)[0]
def line(x): return m*x+b
ys = line(xs)
plt.plot(xs,ys)
m
# Out: 1.6902585379630133
Well, the result looks pretty good. In particular, this is a definitive example that this approach can work better than the simple idea of using just one data point. In fairness, though, it's not hard to find examples where the simple approach works better. Also, this set is regular enough that we get some nice results. Generally, one can't really expect box-counting computations to be too reliable.