How to compute efficiently euclidean distances between patches within a neighborhood ?

Hello, I am currently trying to implement a k-nearest neighbor search within a neighborhood.

Lets say the Matrix A contains extracted overlapped patches in vector form of an given image (output of the function im2col(...,'sliding') ). Now assume we take just a couple of blocks and create a set S containing the selected patches and save them in a vector idx. For an exhaustive distance calculation it's easy to use:

 D = bsxfun(@plus,dot(A(:,idx),A(:,idx),1)',dot(A,A,1))-2*(A(:,idx)'*A)

But now I would like to calculate the distance in a given neighborhood. Lets say each column i of the matrix N contains all local neighbors of patch i. At the moment I calculate the distances with the code below:

 D = Inf(length(idx),Asearch); % Asearch: Area of the local neighborhood
 for i = 1:1:length(idx)
        D(i,:) = bsxfun(@plus,dot(A(:,idx(i)),A(:,idx(i)),1)',dot(A(:,N(:,i)),A(:,N(:,i)),1))-2*(A(:,idx(i))'*A(:,N(:,i)));
 end

In practice, the nearest neighbor search in a local neighborhood is over 10x slower than the exhaustive search above. I think the for loop makes the whole calculation slow. Is there any way to speed this up by handy vectorization ?