any hints on how to reduce the size of this thing?
Hi Anton, here's a hint: You're chopping down P then taking the difference between itself and its circshift. This is very close to just a simple diff() command. Circshift is a useful function, but not quite where you've got it ;)
Think of it this way:
A diff() will get vectors from one point to the next. If any neighbouring vectors point in the exact same direction, they are fair game to remove. More specifically, whenever the direction of the line *changes*, you've found a vertex you want to keep. How can you detect that?
Test  Status  Code Input and Output 

1  Pass 
%% Edge case: no vertices
P = zeros(0,2);
P2 = zeros(0,2);
assert(isequal(simplify_polygon(P), P2));

2  Pass 
%% Edge case: one vertex
P = [1 1];
P2 = [1 1];
assert(isequal(simplify_polygon(P), P2));

3  Pass 
%% Edge case: three vertices (a single line segment)
P = [...
1 1
1 2
1 1 ];
P2 = [...
1 1
1 2
1 1];
assert(isequal(simplify_polygon(P), P2));

4  Pass 
%% Single line segment with multiple vertices
P = [ ...
1 1
2 1
3 1
4 1
5 1
4 1
3 1
2 1
1 1];
P2 = [ ...
1 1
5 1
1 1];
assert(isequal(simplify_polygon(P), P2));

5  Pass 
%% Single line segment, different spacing
P = [ ...
1 1
2 1
4 1
5 1
1 1];
P2 = [ ...
1 1
5 1
1 1];
assert(isequal(simplify_polygon(P), P2));

6  Pass 
%% Rectangle
P = [ ...
1 1
2 1
3 1
4 1
4 2
4 3
3 3
2 3
1 3
1 2
1 1];
P2 = [ ...
1 1
4 1
4 3
1 3
1 1];
assert(isequal(simplify_polygon(P), P2));

7  Pass 
%% Two rectangles separated by line segment
P = [ ...
1 2
1 1
2 1
2 2
1 2
1 3
1 4
1 5
2 5
2 4
1 4
1 3
1 2];
P2 = [ ...
1 1
2 1
2 2
1 2
1 5
2 5
2 4
1 4
1 1];
assert(isequal(simplify_polygon(P), P2));

8  Pass 
%% Nonsimple polygon (figure eight)
P = [ ...
1 1
2 2
3 3
1 3
2 2
3 1
1 1];
P2 = [ ...
1 1
3 3
1 3
3 1
1 1];
assert(isequal(simplify_polygon(P), P2));

9  Pass 
%%
P = [ ...
1 1
2 2
3 3
4 4
5 5
5 4
6 3
8 1
7 1
1 1];
P2 = [ ...
1 1
5 5
5 4
8 1
1 1];
assert(isequal(simplify_polygon(P), P2));

10  Pass 
%% Circle; no points should be removed
theta = linspace(0,2*pi,200);
theta(end) = 0;
x = 20*cos(theta);
y = 20*sin(theta);
P = [x', y'];
P2 = P;
assert(isequal(simplify_polygon(P), P2));

11  Pass 
%% Starting vertex can be removed
P = [ ...
2 1
3 1
3 2
3 3
2 3
1 3
1 2
1 1
2 1];
P2 = [ ...
3 1
3 3
1 3
1 1
3 1];
assert(isequal(simplify_polygon(P), P2));

Find the two most distant points
1823 Solvers
Construct a string from letters and counts
143 Solvers
Vectorizing, too easy or too hard?
138 Solvers
337 Solvers
MATCH THE STRINGS (2 CHAR) very easy
255 Solvers