Is there an algorithm for calculating an "ongoing" or "incremental" checksum on a stream of data where you get given it in blocks and don't keep previous blocks around?
That is: I get sent block A and its checksum, calculate the checksum from block A and check it, then remember checksum(A) and throw away block A. Then I get sent block B and checksum(A+B), and I use checksum(A) and block B to calculate the new checksum, and they're the same?
CRC might do this? I am confused.
@sil pretty much all message digest algorithms work on a stream. It's often only silly implementations which require the blob as a whole.
Fundamentally a CRC / MD algorithm has n-bits of state which get mutated by a block of data.
Most message digests will perform a finalisation of the internal state to result in the final hash. But IIRC the result of the CRC is just the internal state.
@intrbiz for example Python hashlib objects have a copy() method - you can dupilcate the alogrithm state and hash diverging streams. However I don't know a way you can persist/serialise them. Pickle didn't work.
@sil otherwise just compute a merkle tree.