Fastest way to upload streaming points, and removing occasionally

Fastest way to upload streaming points, and removing occasionally

So i have a system (using OpenGL 4.x) where i am receiving a stream of
points (potentially with color and/or normal), from an external source.
And I need to draw these points as GL_POINTS, running custom switchable
shaders for coloring (color could be procedurally generated, or come from
vertex color or normal direction).
The stream consists of receiving a group of points (with or without normal
or color) of an arbitrary count (typical from 1k to 70k points) at a
fairly regular interval (4 to 10 hz), I need to add these points to my
current points and draw all the points so far received points.
I am guaranteed that my vertex type will not change, I am told at the
beginning of the streaming which to expect, so i am either using an
interleaved vertex with: pos+normal+color, pos+normal, pos+color, or just
pos.
My current solution is to allocate interleaved vertex VBOs (with
surrounding VAOs) of the appropriate vertex type at a config file
specified max vertex count (allocated with the DYNAMIC hint).
As new points come in i fill up my current non filled VBO via
glBufferSubData. I keep a count (activePoints) of how many vertices the
current frontier VBO has in it so far, and use glBufferSubData to fill in
a range starting with activePoints, if my current update group has more
vertices than can fit in my frontier buffer (since i limit the vertex
count per VBO), then i allocate a new VBO and fill the range starting at 0
and ending with the number of points left in my update group (not added to
the last buffer), if I still have points left I do this again and again.
It is rare that an update group straddles more than 2 buffers.
When rendering i render all my VBOs (-1) with a
glDrawArrays(m_DrawMode,0,numVertices), where numVertices is equal to max
buffer allowed size, and my frontier buffer with a
glDrawArrays(m_DrawMode,startElem,numElems) to account for it not being
completely filled with valid vertices.
Of course at some point I will have more points than I can draw
interactively, so i have an LRU mechanism that deallocates the oldest
(according to the LRU alg) sets of VBOs as needed.
Is there a more optimal method for doing this? Buffer orphaning? Streaming
hint? Map vs SubData? Something else?
The second issue is that i am now asked to removed points (at irregular
intervals), ranging from 10 to 2000 at a time. But these points are
irregularly spaced within the order I received them initially. I can find
out what offsets in which buffers they currently exit in, but its more of
a scattering than a range. I have been "removing them" by finding their
offsets into the right buffers and one by one calling glBufferSubData with
a range of 1 (its rare that they are beside each other in a buffer), and
changing there pos to be somewhere far off where they will never be seen.
Eventually i guess buffers should be deleted from these remove request
adding up, but I don't currently do that.
What would be a better way to handle that?
Thanks for any input.